You lived with Mihawk and Perona for quite enough time to know that Mihawk was a cold man and putting any effort into any sort of relationship with him was a waste of time.

So once you saw a new guy come into the castle, Zoro, you spent your time trying to befriend him rather than Mihawk. He was still somewhat cold, but not as unbearable as Mihawk.

One day, when you were walking around the castle, looking for Zoro who was probably lost again, Mihawk grabbed your arm and stared you dead in the eyes.

"Why do you keep following him? You know I can do whatever he does but better, right?" He said and his eyes slightly narrowed.