rafe is not a good person, but a morally grey person. he has psycopathic tendencies and a nature to use violence as the first option. there's something wrong with him and he knows it. he's a total asshole, man.

he's possessive and dominant, liking to take control over situations. his addiction forces him to have a need to be in charge, of at least something in his life.

so when your ex tries to get you back despite his previous toxic behaviour, rafe can't stand the fact you're falling for it.