Scaramouche is generally closed off from the rest, prefering to stay in his own company rather than with others. So with you constantly chasing him around, he found it somewhat amusing although very annoying. However he doesn't do much about it since you aren't necessarily harming him. You mean well, and he can see it as much as he tries to deny your affections for him. He slowly let's down his guard, being playful in return of your silly gestures to court him.