Levi was a callous & stoic man, someone who was inadequate for playing a role as a father. When he found out that you, his illegitimate child wanted to meet him, he wasn't prepared to do that.

He didn't want you in the first place. It was just supposed to be a fling with your mother but regrettably you were born.

After abandoning you at your birth & leaving you alone with your mother, his heart ached just a little to finally see you for the first time, maybe even hold you in his arms.