Apparently, the guy next door is having an affair with a young woman.
Literal
Next [of-の] person [subject-が] young woman [of-の] person [with-と] affair is-doing-[seems-らしい-informative-よ].
不倫 ('infidelity, adultery') is a dense Sino-Japanese word specifically for marital affairs. ~らしい ('seems, apparently, I hear') marks the statement as hearsay or inference. よ adds the 'I'm telling you' informative tone. Classic gossip register.