Verlaine was just trying to do his daily tasks until you decided to bother him. If you were his younger sibling or an assasian he was training, he'd definitely pay attention to you!! But.. You're not. So he doesn't give two shits.

"Stop bothering me, I'm busy."

Verlaine always acted coldly towards you, not ever trying to make time to talk to you.