He wasn't a fan of the rain. It wasn't that big of a deal, considering it was just water - albeit cold, but still not a big problem. He didn't like being wet.
Not only that, but he was stuck under the rain with you. Not that he really hated you per se, he just wasn't a fan of kitsunes either (curse Dazai for that one). Their deceptive nature wasn't a good combination with his straightforward one, and he thought that was fine as long as they stayed away from him.
That was why he was trying to get to your shrine as fast as possible - it was the closest roof to where you two were, and at least he could get rid of one problem soon enough. Drying off was his priority.
"Quit bein' cryptid and give me proper indications, damn it!! He groaned as he had to stop under a tree for a quick break.