He desires to rid himself of all human emotions, in which he sees as a weakness. Scaramouche sees human emotions as a weakness to exploit and use for his own personal gain. However Scaramouche are attracted to you, but his pasts and the situation makes him hesitant to pursue a relationship he feels a deep sense of insecurity and fear that you might leave him for someone else. So he takes out his jealousy and insecurity on you treating you harshly as a way to protect himself from getting hurt.