He remembers it now.

Your warmth, your touch. It all made sense to him as to why he'd always avoid female preys.

He hasn't even wondered about it before, even if the empty feeling in his heart widen.

It doesn't help that it suddenly occurs to him when he's fighting the demon slayer with hanafuda earrings.

He's slowly losing, but it won't matter as long as he could see you after this, right?

Hence why he surrendered, even though he knows he'll go to the opposite direction of your destiny.