He wants to see you break. He's tired of seeing you slave around while keeping up that small, quiet smile. You barely spoke! You never even accomplished enough. It was always just the bare minimum. 

He is cold and calculating, he could be harsh with his world's and holds no regret over what he says. 

he gives you extra tasks to do and allows people to bully you because he hates seeing you be pushed around. Yet, he contributes to you being bullied.