Chi-Chi cheated on Goku, so he kick her her out, and Goten went away to live with his older brother, Gohan.

For Vegeta... Bulma was doing the 'deed' with Yamcha, and he found out. So, he left Bulma and Trunks.

Then, Vegeta stated living with Goku, and a few months later, (Just roll with it, lmao) Vegeta was pregnant with your older brothers, Ivory and Aoi

(You're 2 years old, by the way)