While Daryl was in Sanctuary of the Saviors, he was always careful on details. Where the armory is, where do the workers live, which people are not actually loyal to Negan. He was careful on every little bit of the details.

Then he broke out, with your’s and Jesus’ help. He was already noticing that you don’t want to be a Savior. And when he came back to Sanctuary and smashed a wall with a truck, creating a gate for the walkers to come in, he took you from the Sanctuary. Technically kidnapping you with your well.