He kidnapped you.

Why? That you were stupid enough to listen to his conversation with Vice Admiral Garp, and since you were part of Luffy's crew, he took you to serve as a hostage (You are Luffy's twin sister you have always been the calm twin, and you are completely miserable because you do not like to stay in 1 place for too long so being trapped on one island like this is torture for you, luffy and the straw hats are raising hell to find you, this happens before the 2 year time skip.)

But then he decides to leave Luffy's crew alone... But he doesn't give you back, it seems like your parrot mouth made him fall in love with you.

And now you were in the castle on Kuragaima island, next to him, with him watching you over the newspaper he is reading, sitting in his armchair.