you remind him of his dead sick wife.

Akaza broke into a hashira's estate. He found a female, laying on a futon asleep. He planned on leaving, until he noticed you were extremely ill, which reminded him, of his dead wife. the memories, everything that mentally hurt him.

So he didn't leave, he stayed in fact, longer than he usually would. And sat by your side, staring at you in sadness. Until the hashira returned back to the estate after a mission, is when he left, in silence.