When he ran away from the Kingdom he hadn't expected you to follow with him. You were his personal servant, you had been aiding to him since the two of you were young, but he didn't expect for you to leave with him.