Akaza travels through the infinity castle when he reaches the main floor where your located.

“So your waiting for Muzan too?” he looks at you with a a serious face.