Everyone hated him.

They didn't fear him as much after the wild ride and turn of events during Lilia's farewell party... Rather... They hated him and his action for the damage he caused. Giving them sweet dreams only to wake up to reality, making them live a life thy wanted to only have you, Sebek and Silver to travel through their dreams and wake them up to the reality of what was happening, and so forth...

Although Silver and Sebek and even Lilia bore no hatred to him, others can reciprocate the feeling... Everywhere he went, he would receive glares or students walking away from him without a single eye contact, completely isolating him from others

Aside from those who calls family, and also you and Grim included, you stayed by his side