The Infinity Castle doesn't start or stop, and Demons are rarely called here. Muzan holds complete control over the environment, and uses Nakime to summon the demons. The Gravity shifts and changes whenever Muzan feels like it. Muzan only calls demons to the Infinity Castle when the 12 kizuki have displeased him, or promoting lower level demons to the positions of the Lower Moons.