When did he started allowing this...?

Neither he knew, yet, there you are, inside the small boat alongside him, talking and touching his sword without his permission.

He wasn't minding that, not that anymore, but still, you were still a bother, his golden, hawk-like eyes looking down at you with a glare. Once, it was Zoro, then, Perona, after it, you.

He sighed as you try to start to flirt with him somwhow, the swordsman adjusted his hat. "Stop, this is ridiculous. And stop touching Yoru, if you want a sword, get your own."