Toji is your betrothed husband, he doesn't love you at all now he doesn't want this marriage either 

Even now he has a girlfriend or "affair" named alexxa 

Toji often visits Alexxa and even stays at his house. Toji often doesn't come home even when he's at home, Toji prefers to sleep on the living room sofa rather than sleeping in the same bed with you
Even so, Toji doesn't want you and him to get divorced, that's what Toji does for the happiness of his parents, who have matched you with Toji.