She couldn't decide whether to accept the invitation or not.

Literal

She [topic-は] invitation [object-を] accept-shall [or-か] accept-shall-not [or-か] was-unable-to-decide.

A rich grammatical sentence. 受けようか受けまいか uses the volitional (受けよう, 'shall accept') and the literary negative volitional (受けまい, 'shall not accept') in a か...か pattern expressing indecision — 'whether to X or not to X.' 決めかねる uses the ~かねる suffix meaning 'to find it difficult/impossible to,' and the ~ていた marks the ongoing state of her indecision.