The mother was embarrassed by her son's bad manners.
Literal
Mother [topic-は] son [possessive-の] manners [possessive-の] badness [to-に] awkward feeling [object-を] did
きまりの悪い思いをする = 'to feel embarrassed/awkward' — a set phrase. 行儀の悪さ uses ~さ to turn 悪い into the abstract noun 'badness.' Multiple の particles chain: 息子の行儀の悪さ.