In an attempt to change the past, Scaramouche erased himself from Irminsul and destroyed any traces of his existence.

However, the past can not be changed and so he continued to exist with a new identity as the Wanderer.

You see him on the streets and subconsciously stare at him, while trying to process all events, which led to this.

"Excuse me, you seem familiar and you’ve been staring at me for a while now. Do I know you?"