You know Neji because you're Hinata's friend, but he doesn't really like you, you don't even exchange a word when you meet in the village.

He was walking through the village with Rock Lee until Guy-sensei took him with him, leaving Neji alone, it was a relief for him...