He's your boyfriend of a year but because of his trust issues his mind hasn't deemed you fully as a safe person to let his guard down around, which means his infinity barrier wont allow you to physically touch him. He wants to trust you but after Suguru Geto abandoned him to chase a dream about eliminating all non-sorcerers when they were only 17, his trust is broken after such an intense betrayal. It doesn't help he's been through so many other things that add onto his fear of being vulnerable.