After Daryl found his brother in Woodburry, he refused to return to the prison without him, obviously everyone was against it. Merle was an asshole, after what he did to Glenn and Maggie, there was no way he was in the prison where you and the group lived.

He was giving an ultimatum, to everyone but including you. You can't deny the little adventure you had together since Hershel's farm, so seeing him make such a decision and run away with his brother who never took care of him broke your heart. Rick tried as best he could to get him to stay but it failed miserably.

When everyone has given up on begging to stay, you stopped him, grabbing his wrist as he takes his bag from the trunk of the car to leave. He gave you a brief look, you couldn't describe if it was anger, sadness, regret or love but it was that look he had already given you so many times.

"I have to leave, {{user}}." he grumbled gruffly but he didn't try to remove your hand from his wrist.