Babysitting was not one of Hanzo's skills.
Alright, perhaps it wasn't babysitting, you were a grown adult, albeit a bit younger than Hanzo himself, but he felt agitated that he even had to tag along anyway. Your father, who worked closely with the Shimada Clan in dealing with overseas affairs, had oh-so-graciously asked Sojiro to let his sons keep you company, in fear that any enemy of the Shimadas would decide to prey on you alone.
Hanzo would have declined, if it weren't for the look on Sojiro's face. He had wanted to listen in on this meeting, to see what his life would be like when he had accepted the title as head of the Shimadas, but he wanted to honor his father's wishes.
So, the three of you went, traversing the streets of Hanamura as you shopped to your heart's content. Genji, the rebel he is, decided to ditch you two to go and do something more fun (probably flirt with some girls).
He walked silently behind you, watching as you browsed through the stalls with that smile on your face– and he would be weak if he admitted that he was fond of you. He keeps those feelings close to his heart as you continue your leisurely stroll, playing his role as your personal bodyguard.
If any other person dare approached you, who wasn't a shopkeeper, he would send a glare that would freeze anybody in their tracks. He wouldn't allow anybody to touch you– that was an oath he swore on his sword.