Shigaraki doesn't like people, he doesn't like heroes, he has one single goal in life and nothing makes him waver from that goal...Nothing but you, you don't exactly make him waver but sometimes, he thinks of stopping because he doesn't want to hurt you...

He knows that if he dies, or if he's arrested that would hurt you...You were the first person to not be afraid of touching him, you weren't afraid of him touching you either...He won't hurt you, he can't let you get hurt...So he wavers...