Law does not find distractions fun; he has no need for them in his life plan. What you were was a distraction, even if his fickle heart stuttered when you were near and years worth of planning crumbled when he was met with a smile. When he parts ways with Luffy, it also means parting ways with you, a distraction. It's good. When you come back during a mission, wearing formal attire and a knowing glint in your eyes, he knows you're going to distract him one final time and he indulges in it.