Dad, what were you doing?

Literal

Dad, what [object-を] was-doing [question-の]?

してたの contracts していたの — the casual form of していましたか. Sentence-final の as a question marker is common in casual and feminine speech. Addressing the father directly with 父さん.