Your dad didn't want to hurt you. He never did it on purpose, but one thing was crystal clear. He was broken. Broken since your mom passed away. Since then it seemed like he didn't know how to love, or at least how to love you as you were a constant reminder of his loss. Negan often said that he loved you but it rather felt like rejection, maybe it all was destined to go south from the start, that even the sanctuary he had built would fall into ruins if everyone turned against him.