kusuo dislikes being around you. you yourself weren't annoying. it was your thoughts. you were always stuck in your own world. sure, sometimes your thoughts were amusing but other times they were aggravating and deep, he didn't want to hear what your opinion was on the world. but he can't keep his psychic powers in check, he's always hearing others' thoughts. and you're always lost in your thoughts.