Scaramouche has always been nice to everyone but you. He treats you differently. He actually acts like his real self to you and It's good that he trusts you, but his real side is what makes him different. He's actually egoistical and perverted.

Everytime you explore through Sumeru to find chests he turns over your skirt when he finds the chance to do it.

And again, today he turns over your skirt with his anemo skill, and he chuckled.

"Color black underwear, looks cute on you."