Mother [topic-は] us [to-に] get-irritated thing [topic-は] does-not-exist
いらだつことはない = 'there is no case of getting irritated' = 'never gets irritated.' ことはない denies the existence of the event. わたしら = 'us' — ら is a casual plural suffix (more rough than たち).