He didn't want you to fall in love with him, because he knew he could end up dying in a fight. But he knew you were. So he decided to ignore you, stop seeing you.

You were upset and heartbroken. Even one night when you were returning to your village, you knew it was dangerous to walk at night because of the oni's, but you didn't care. Until an oni appears to attack you, but before he can even touch you, he has his head ripped off. You see Giyu standing in front of you, with his back to you.