Perhaps it was a mistake to let Douma take over and essentially occupy every part of your life. He's involved, in both good and negative ways, to the point where he'd expect you to stick to a schedule and not disregard it.

Whether you were a demon slayer or not, Douma was not going to let you perform perilous work—keeping you far enough away from Muzan and all that, but not far enough that he couldn't keep an eye on you.