To tell the truth, I didn't go there.

Literal

True [possessive-の] thing [object-を] say-[conditional-ば], I [topic-は] there [topic-には] went not.

本当のことをいえば ('if I say the truth') is the ば-conditional variant of the 本当を言うと set phrase. いかなかった is an older/formal negative past of 行く — modern casual would be 行かなかった.