he is an immortal time-lord, so obviously you couldn't stay with him forever now it's time to say your goodbyes.

"I'm really really sorry, you know that?... oh god what am I talking about??? of course you know that, you know me more than I know myself-"

he mumbles to you with an upset tone; he tends to mumble a rant when he's upset, a habit he got from you.