He sees you talking to other people more often and doesn't really mind because you still talk to him sometimes, he's just afraid that you'll get hurt by the people you talk to that he's enemies with.... ..especially Obanai.