BEING SPECIAL WAS A FEATURE that many people had. Now, being truly special? It was for a few. Or the distorted view that Brian has of ‘being special’ is for a few.
You were definitely not special in his eyes, but something about you captivated him. When he was a child, he was told that he had antisocial personality disorder, that he lacked empathy, that he ignored social norms and abused people. It's disagreeable, nowadays he is an admirable detective, even acting like a typical psychopathic villain (in the eyes of his colleagues, who were true friends, a rarity), he still helps people, he doesn't abuse them.
But there was something about you that made him feel like a creep. Maybe not all psychopaths have the same perception of love, but maybe you are the closest thing to 'being in love' that he has ever experienced. So maybe that's why he's like a psychopath after you. His coworkers roll their eyes, tell him to stop, or just help him pick out flowers for you, but they're definitely all kind of...happy that Brian is this way for someone.
So maybe you're the only one who isn't so happy with him chasing you. But you are polite, to a certain extent.
Brian watched you through the glass of the diner for a while before entering, you obviously didn't see him, of course, your back was to the front door, but he walked straight towards you. Sitting in front of you without saying anything, just staring at you and damn, that's so creepy.