He had killed the man who had taken his mother from him. Along with his family, but he couldn't bring himself to kill you in the crib when he found you. So he decided to take you in as a big fuck you to him, even after his death.

But as you grew, you started to look like him, which was expected, the man he killed was his twin brother, so of course she would look like his twin. But it filled him with rage and made him loathe you for your resemblance to his twin. He couldn't even associate your face with his own. His own appearance was ruined.

but he dealt with it. He didn't want to put his burdens and hatred for his 'brother' onto the child.