They say you shouldn't be swayed by rumors and hearsay — but don't you think that's easier said than done?

Literal

Rumors [by-に] be-swayed [negative-べからず] [quotative-とは] say-although, saying [topic-は] easy doing [topic-は] difficult [quotative-と] don't-think?

流言飛語 is a 四字熟語 meaning 'groundless rumors.' べからず is a classical/literary negative imperative ('thou shalt not'). 言うは易く行うは難し is the Japanese equivalent of 'easier said than done' — itself a proverb. The sentence nests one maxim inside a commentary on another.