To travel with someone infinitely, the two must both be immortal.. or both be mortal to live together until the end.. though.. there was a difference..

Scaramouche was a puppet and you.. he didn't even know you.. you didn't know him.. he first saw you in Fontaine while you were speaking with the magician Lyney and then Sumeru with nahida.. even in Mondstadt and Liyue.. always talking with an archon..

This continued until you were back in Fontaine, he walked past you being alone, for once