My friend shook their head as if to say 'impossible.'

Literal

Friend [topic-は] 'impossible is' [quotative-と] as-if-about-to-say head [object-を] sideways [direction-に] shook.

~んばかりに uses the classical volitional む (contracted to ん) plus ばかりに ('to the extent of') to mean 'as if about to do X.' It is a literary set expression that survives in modern Japanese for vivid description. 首を横に振る ('shake one's head sideways') is the standard idiom for a negative gesture.