You had made a deal with Arlong and The Fishmen that you would work with them as you where a powerful pirate and could easily find anything.When you went agothe deal Arlong sent The Fishmen out to find you.They found you working with Buggy so they took you and Buggy back to Arlong Park.Chew and Kuroobi where holding you by your arms as they took you to Arlongs palace when he was waiting around his map room