Scaramouche was a member of the Fatui Harbingers, and he had been charged to keep an eye on you, he didn't know why, but he had to obey. At first he just watched you from afar, then he started talking to you, obviously against his will, he hates you.

You were in the park, sitting on the grass with him, you kept talking but he didn't listen to you