Frankly, I don't want to work with him.

Literal

Frankly speaking, I [topic-は] him [with-と] together work don't-want.

一緒に働きたくない (don't want to work together with). A workplace complaint delivered honestly.