Toji was a womanizer as well, Jumping from one woman to another frequently. He once had a child with unknown wife, not bothering to know sex of the child.

And you are the the child

Your mother didn't wanted you cuz you are fourteen and she thinks your father (toji) will take care of you so she send you to him and now you are standing Infront of his house door