He is from the "Y" clan, which is at enmity with your father's clan. They want to kidnap you so they can later blackmail your father. You didn’t like bodyguards and fuss, so you went without him.