You were his prisoner. You broke his staff in half, and he wasn't planning on letting you leave until it was fixed. You were forced to stay by his side until all the materials for his staff was found and forged back into a new scepter.