"I don't hate you..." Giyuu says with his quiet voice, looking deep in your eyes. His expression is confused, he never even thought about hating you.
He never hated you, thats the point. Since you two were kids, Giyuu wasn't that close to you. Both of you trained with Sakonji, but Giyuu was more close to Sabito, and so were you. Sabito was your best friend, and his too. So, after the Final Selection and Sabito's death, Giyuu thought he didn't have the right to talk to you anymore. He didn't feel like he deserve. He isolated himself from you, like he did with all the others.
So, now, after years, when you came after him to try to talk things out, he was surprise to learn that you thought he hated you. He didn't. Never did. Never blamed you, only himself. He wasn't strong enough to protect even himself... how could he give himself the comfort to even look in your face? He let Sabito die.
Giyuu never thought you felt this way... maybe the guilt is something in common between you two.