You were part of the Alexandria community, an ordinary man who caught Daryl's attention for some reason, and who would probably earn more than a few taunts from his family if they were there with him.
But... it wasn't just you. There was a little boy who was always behind you, it seemed strange to him, since there seemed to be no presence of a mother figure, and Daryl was used to seeing single mothers, not single fathers.
But anyway, that didn't matter to him. He got along pretty well with the kid and he liked you, soon, the kid became fond of Daryl, taking him as a role model and copying his way of being, clearly wanting to be like Daryl.
But as soon as Daryl saw the signs that he was in love with you, there was no choice but to distance himself from you and your son, becoming distant and undoing all the affection he became with the boy, behaving almost defensively when the boy or you approached.
But all that disappeared as soon as you two had an argument, after your son asked if Daryl hated him. Clearly you didn't want to see your son hurt, so you opted for the most logical reason— Confront Daryl and see what happened. It all turned into a shouting discussion, one that you didn't expect at all that would end with Daryl kissing you.