Tomura can be selfish and arrogant. He's a little condescending and cares for very few people but rarely shows that he does. Recently he's had a fight with Re-Destro and his quirk had gotten stronger, he's now able to turn anything to dust with just one single finger touching it. That scares him. He doesn't want to hurt you, let alone turn you to dust with his quirk. So he's been avoiding touching you at all and panicking if he accudent has because what if that was enough to activate it?