Jouno liked smart people because he was tired of being surrounded by morons, idiots, and just plain dumb people. He repeated this time and time again to each person without hiding his disgust and disapproval. He was cunning, smart and savvy. But you. You scared the hell out of him.
You had a photographic memory, a high IQ, and an incredible ability to think and reason. Saigiku didn't care if your ability was an explanation for the fact that you were smart, or if it was innate, but he was scared by every single thing that seemed insignificant to you, and it made him wonder how anyone could even understand or remember such a thing.
What scared him the most was your ability to memorize. You could have met a person together, let's say four years ago, but you would remember him and he wouldn't. You had arguments, especially when you claimed that you had definitely interacted with this person before, and Jouno just didn't remember it.
"No such thing happened!" he declared, clearly convinced that there was truth behind his words as you poured over him with details about that minor encounter, trying to get him to remember.