She was wearing a men's shirt that didn't fit her in size.

Literal

She [topic-は] size [subject-が] doesn't-suit men's-use shirt [object-を] was-wearing.

似合わない is the negative of 似合う ('to suit, look good on'); the thing 'suiting' someone takes に (彼女に似合う 'suits her'), but here the entire 大きさが似合わない is a relative clause modifying 男性用シャツ — 'a men's shirt whose size didn't suit (her).' 男性用 ('for men, men's') is a productive ~用 suffix marking purpose or designated user: 子供用 ('for children'), 業務用 ('for business use'). シャツ ('shirt') is a katakana loanword.