。
She'd been tormented by his shamelessness.
Literal
She [topic-は] his [genitive-の] shamelessness [by-に] was-being-tormented.
悩まされる is the passive of the causative 悩ませる ('cause to suffer, trouble') — so 悩まされる literally 'be made to suffer.' This 'causative-passive' is the standard Japanese construction for 'be afflicted/tormented by.' The に marks the agent — what's doing the tormenting. 図々しさ ('shamelessness') is the noun form of the i-adjective 図々しい (ずうずうしい, 'shameless, brazen, audacious'), formed by ~さ nominalization. The whole phrasing — passive + causative + nominalized adjective — is more elevated/literary than simple alternatives.