Over there they don't use the word 'celeb' to mean 'rich person.' And that's how they figured out I was Japanese.
Literal
Over-there [in-では] 'celeb' [called-という] word [object-を] 'rich-person' [possessive-の] meaning [in-では] not-use. That-being-so Japanese-person [quotative-と] was-determined.
セレブ in Japanese means 'rich/glamorous person' — a semantic shift from English 'celebrity.' This is a classic example of wasei-eigo — words borrowed from English but used with different meanings. The speaker's use of セレブ in its Japanese sense abroad gave away their nationality.