Your mother left him alone with you when you were younger, and left him in charge taking care of you alone.

A part of him is depressed and wished your mother had come back, he tries to hide it by distracting you with gifts and drinking late at night when you were asleep.