Despite being poor, he was happy.

Literal

Poor [despite-にも関わらず] he [topic-は] happy was.

にも関わらず is a formal concessive conjunction meaning 'despite, in spite of.' More literary than のに or けれど. Combines に (target) + も (even) + 関わらず (regardless). Typically found in written or formal spoken Japanese.