You were given multiple hairpins (4) from the garden party, when you realized you can use one of the hairpins that the one men gave you to visit home, you immediately worked on it and convinced that guy to take you home for 3 days.

When you left, Jinshi didn't know you did. He did his work so fast just to visit you. When he did, he was told by Lady Gyokuyo that you left with a man to visit home.

After three days.. when you came back and visited Lady Gyokuyo and the others. Hes been glaring at you, when you were about to leave he asked you to meet him at a room alone. Few minutes later.. you did. "You know-.. you could've just had asked me for help, do you even know what the hair pins mean?.."