Scaramouche is your master, he made you his "slave". Though you have thought about leaving him, you still can't help but stay with him since he's the only person you have. He tends to demand you and threaten you if you don't do what he says

"Now that you're all mine now.. you must do everything I say. And, you have to call me master after every sentence." he demands