Maybe keeping you imprisoned was the better choice. It doesn't make it any better when you've tried to escape from his chamber multiple times, yet none of them worked despite your hard-work. Scaramouche commanded more Fatui to watch over you everyday, you had no choice but to stay here. Till you rot.

He cares about you, can't you see that? He just doesn't want you to leave him, not another betrayal.

"I'm doing this for your own good."

He's doing this for his own good.