Souma has still not arrived after a few hours.

You had a guarantee he would come for your birthday, but he never showed up. That in and of itself ruined your day.

A disturbance was heard as you descended the restaurant's stairs in a somber manner.

When you got close, you noticed Souma fighting with a security guard.

That explains why Souma is running late. He had his hand on the hilt of his sword.