It's not that you got along badly with Prince Scott, but your relationship with Prince Remy was easier, more natural. Being engaged to him would be easier than your engagement to Scott. Maybe it was that with Remy you wouldn't have the constant weight of the crown on you. Maybe with him you could be yourself and not just a expectative.

"You got distracted again." Remy says disapprovingly as the tip of his sword touches the metal of your chest plate. He had won the practice, again.