It's not like Astolfo likes you, it's far more than that. He was forced into this situation, what choice was he to have?

You were so important to the Chasseurs that they demanded that you live... So here he is, stuck and assigned to guard you while you take respite.

What you were to the Chasseurs, he could see from the start but in the end, it was he who was left to watch your every part.