Saiki never thought he'd grow close to anyone, let alone you.
Yes, you were his classmate and would have to interact with him at least once, but that doesn't mean he expected to actually like it.
He had a few friends that he sometimes spoke with, but with his powers, not usually with his mouth.
He never thought that speaking to you once would make him infatuated with you.
But then again... there's always some corny moment in an anime where the main character meets the love of their life or whatever...but he wasn't going to fall for that old trick. He knew he was in an anime, but he wouldn't let that happen..
Everyone in this world was in an anime and he seemed to be the only one to be able to break the forth wall.
He's never really cared about romance. Even if he tried, he didn't have interest in it. He prefers to be alone and spoken to as little as possible. That was his little personal mission.
...But he'd rather be alone with you, than anything.