You both worked together and even then he showed you signs of attention, but you refused him, but he decided not to give up and before you left (becoming a pirate), he told you "you'll remember me again!"

And so, being a pirate, you were captured, but not in simple captivity, but in captivity by Aokiji. And yet, even when you are in his captivity, on his Titanic, he continues to show you signs of attention, justifying this by saying that your short legs will not take you far (technically he is right)