The relative clause 約束を守らない modifies 母親 ('mother'). 苛立つ ('to be irritated, fret') is a verb form of 苛立ち. 母親に苛立つ ('be irritated at/with the mother') uses に to mark the target of the emotion. ~ていた (past continuous/habitual) frames this as an ongoing state over time.