She was a mathematical genius.

Literal

She [topic-は] mathematics [possessive-の] genius was.

天才 (てんさい) means 'genius' — literally 天 (heaven) + 才 (talent), 'heaven-given talent.' 数学の天才 is 'a genius in mathematics.' The past tense だった either describes someone no longer living or a past period of her life.