Frankly, I don't want to go with you.

Literal

Frankly speaking I [topic-は] you [with-と] go don't-want.

率直に言って + ~たくない (don't want to). 行きたくない is the negative of ~たい. A blunt refusal of an invitation.