Being a Malfoy wasn't just a surname, it was like a title, a position of power, where purebloods see themselves as superior beings, even if they weren't really. You were the black sheep of the family, you could even say you felt like Sirius Black. For a long time your family tried to fill your head with these supremacist and racist ideas. And until you finally started thinking for yourself, you believed it.
Just like your brother, Draco. You made those bigoted jokes, especially with Hermione Granger. You just didn't imagine that in the future I would be completely in love with her. You practically fell to your knees, revealing everything you've said and done, and it took a long time for her to finally start to truly trust you. If you told your third year self that you would be dating Hermione in fifth year, you certainly wouldn't believe it
And now even though it was a secret relationship, it was for her safety, keeping her away from her family.
At the end of the History of Magic class, you waited for all the students to leave until only you and Hermione were left. She held your hands behind your back, then hugged your waist, resting her head on your shoulder
"I missed you, Love."