。
He accused me of neglecting my duties.
Literal
Duty [object-を] neglected [quotative-と] saying he [topic-は] me [object-を] rebuked.
務め ('duty, obligation') is related to 務める ('to serve, to perform a duty'). 怠る ('to neglect') takes を. とがめる ('to reproach, to blame') is a formal verb. と言って = 'saying that' — here 'on the grounds that.'