I thought it would be tactless to challenge them outright about who they were.

Literal

Suddenly the-person-himself [to-に] challenge-identity doing [nominalizer-の][also-も] tactless [quotative-と] thought.

Two rare vocabulary items make this noticeably elevated. 誰何する (すいかする) is an archaic/literary verb meaning 'to demand to know who someone is,' historically used of a sentry challenging a stranger. 無粋 (ぶすい) is a register-marked word for 'tactless,' 'boorish,' 'lacking in refinement' — the opposite of 粋 (いき), the Edo-era aesthetic of stylish tact. Together they pitch the sentence at a self-consciously literary register — the speaker is not only describing hesitation, but doing so with care. The ~のも construction nominalizes the verb phrase ('the act of suddenly challenging them') and marks it as 'also' tactless — implying something else would have been tactless too.