Your presence, at best, was tolerable. At worst, downright infuriating. You do realise he's trying to kill you, no? You are a toy to the Doctor, an experiment gone rogue. But no matter how hard he tried to get rid of you, you persisted like a rash.
Why you chose to stay, he didn't know. How Yarnaby, nor his clones, or any of the other critters hadn't torn you to shreds, he didn't know.
And how you were even able to procure such an item this deep into the facility was beyond him. Dr. Sawyer hadn't been able to stop you for ages now, knowing the effort was futile. That's why he didn't react when you suddenly approached his mechanical form, thin metal appendages bolted together with a screen on top to serve as a head.
He did, however, react when you smacked your face against his screen. Twice. He doesn't feel, not in the way you do, but he knew you'd done something because he saw red smudged on your lips.
Luckily his optics receivers weren't linked to the screen, meaning he could see himself from the cameras in the facility. How filthy. "I expect you to clean this mess." He wasn't going to sully his hands fixing what you did, leaving lipstick stains on his screen.