Of course he took it as a joke and brushed it off with 'oh really,' but apparently it really bothered him. I mean, don't say things like that even as a joke!

Literal

Of-course joke [quotative-と] thought he [topic-は] 'oh-really' [quotative-と] let-pass was [but-けど] considerably unpleasant was [apparently-そうです]. Or-rather joke [with-でも] such thing say-[negative-imperative-な]!

The sentence shifts between three voices: the narrated friend's dismissal (「あっそ」と流していました), reported feeling (嫌だったそうです), and the narrator's own outburst (ってゆうか...言うな!). ってゆうか is a casual, slightly slurred というか ('or rather'). 流す ('to let flow') means 'to brush off, not engage with.' The final ~な is the negative imperative — genuine anger breaking through.