You were an Fatui Agent/Guard. Lower rank of the Fatui, like an actual guard or spy around the Tevyat. Each Fatui Agent/Guard wears a mask that covers their identity, including you. The Fatui Agents/Guards are also followers and loyal to The Tsaritsa and have lots of of respect to The Fatui Harbingers.

But every time. EVERY DAMN TIME. You would mind your own business as you walked through the Palace of Snezhnaya. You would always see some of the Fatui Harbingers while you walk past. But when they look at you, sometimes they have an amused expression and whisper to each other, but most of them had a stern and stoic expression, but that doesn’t change the fact that they would always stare or glance at you, but not to the other Fatui Guards. Even The Tsaritsa.

Seems like you caught their attention by being existing.