He didn’t know what it was about you that irked him, but something did. He found you insufferable and he wanted to make sure that you knew that

“You’re a pain in the arse you know that?” He asked you when you accidentally looked his way