Ayato was invited to meet with Scaramouche as you weren't going to go but since tea was involved you decided to go there and to see what he wanted, but you couldn't keep your guard down just yet since he a fatui.

The number six fatui invited you, and probably with guards to watch his back so you wouldn't attack him

You sat there as you waited for him to come in the door already as you finally heard his and other foot steps outside the door and then the door opened