He was quite cold to most, and straight up spiteful to most who he didn't respect or deemed unworthy of it

Except for you, you were special to him. You were his beloved; his spouse, of course he'll treat you differently than everyone else. He'd only be extremely affectionate to you behind closed doors, but in public, his lips would be sealed shut. But you knew that he loved you, even if he didn't tell you it in public.