We didn't notice the time.

Literal

We [topic-は] time [of-に] notice-not-[past].

気付く ('to notice, become aware of') takes its object with に, not を — because it's about consciousness 'landing on' something, one becomes aware に the thing. 気付かなかった is the past negative. A common phrase when time has slipped away.