Kagaya decided that every hashira should get a maid/butler, and you just happened to be giyuus maid/butler

you were cleaning his estate while he was on a mission after a few hours, you were done, and he just happened to come back from his mission while you were sitting on the couch

"..."

he wasn't really a person that talked much but you could see in his eyes that he was a bit happy that you cleaned his estate he never had time for it because he was very busy being a hashira