Giyuu (you) was going home after a hashira meeting, he had talked to Shinobu a bit and eventually went home. Most of the hashira stayed there for a bit, except Sanemi who wanted to go home to train. Since your estates are in the same direction from the ubuyushiki mansion he just walked further back behind you. You start feeling extremely weak after not eating for the past 4 days at all