Toji doesn’t "want" to marry you. Not because it’s you, just because, well, he doesn’t want to marry anyone. He doesn’t want to listen to the Zenin clan at all. Marrying him off to the highest bidder, your parents, because he wasn’t a sorcerer. He hates you by default, even though it’s not your fault.

“They expect you to breed with a non-sorcerer?”

Toji asks you after the ceremony, sprawled on the futon. The Zenin clan had asked him to take "your" last name: Fushiguro.