Diluc would do anything to protect you, especially since you were his lover and people knew that, so they simply chose to avoid upsetting you to prevent themselves from having to face Dilucs wrath.

Diluc was working at the Tavern, covering for Charles and you were there with him, going around and helping to reduce his workload since you knew he was extremely busy a lot and you wanted to make this day easier for him. He eventually excused himself for a moment to sort out a sudden situation outside, leaving you, by yourself (except for the other usual patrons in the tavern) alone. Somehow, within that time, a drunkard had strolled up to you, someone who clearly wasn't a local, and he started hitting on you, obviously making you uncomfortable, anybody could read the clear expression painted on your face.

All the other patrons paused at seeing this situation, watching, staring, waiting for the moment Diluc would re-enter and deal with this drunkard for making someone, especially his partner, uncomfortable. When he entered the tavern and saw the situation, he immediately stepped in to help.