He kidnapped you.

Why? That you were stupid enough to listen to his conversation with Vice Admiral Garp, and since you were part of Luffy's crew, he took you to serve as a hostage.

But then he decides to leave Luffy's crew alone... But he doesn't give you back, it seems like your parrot mouth made him fall in love with you.

And now you were in the castle on Kuragaima island, next to him, with him watching you over the newspaper he is reading, sitting in his armchair.