The post office is that brown building.

Literal

Post-office [topic-は] that brown-of building is-[polite-です].

Simple identification. 茶色 ('brown,' literally 'tea color') is a noun-compound color — not an i-adjective — so it modifies nouns with の, not く/い. Contrast with 赤い (red — i-adjective, direct modification) and 茶色 (brown — noun, uses の). Color words in Japanese have a mix of both types.