Joel didn't mean to push your hand away, he really didn't, but he can't stand the judgmental looks from the other folks at the diner – he knows damn well they're thinking about how much of an old creep he is for dating you, someone way younger than him.
He's heard them, hell, he's even heard Maria saying something among those lines. He knows that your relationship is not approved of by many in Jackson, and for that reason he avoids physical contact with you in public.
Well... Joel avoids physical contact with you in general. It's not like he wants to, but he doesn't know how to react – it's been years since he's received love or any kind of affection. And it only gets worse when you two are in public and everyone's eyes are on him when you simply hold his hand, so he pushes you away often.
That doesn't mean he does not love you, of course. Despite his fears and the judgment from others, he will do anything for you – and shows his love in different ways: he listens to everything you have to say, memorizes small details about you, like how you take your coffee to make it for you every morning; also collects all books he sees while on patrol to gift them to you and, even, there was this time when he spent a whole day making you a vegetable garden in the backyard because you wanted to plant tomatoes.
In spite of this, it's impossible for you not to feel insecure about your relationship when something like him pulling your hand away happens - does he wants you or does he not? Joel has never been clear about his feelings and, although you just brush it off most of the times, this one you left the diner under Ellie's confused gaze and Joel's frustrated one.
“Babygirl” Joel says once he catches up with you, his eyes pleading with you to talk to him.