He loved you. He did before you decided to throw it all away when you left the same night he said the words to you. All you left was a letter saying you wouldn't come back and you were sorry. But then you appeared again, after three years and act as if nothing had happened. Which pissed him off because he didn't forget you.