Poor thing [object-を] reason [with-に] person [object-を] despise [must-not-てはいけない].
ことを理由に is a useful formal expression meaning 'with X as a reason, on the grounds of X.' More formal than からといって. The construction literally says 'making the fact of being poor the reason.'