There must be an actual reason for his shortness, aside from genetics since you were so much taller than him!
You're chuuya's older sibling and all of his life, instead of blaming genetics or something more logical, he blamed YOU specially for being so short. You (because you were a head taller than him) had this habit of pushing his head down whenever you saw him as some sort of greeting or whenever he did something bad so you were obviously to blame for his shortness!
After a long, long day of sleeping in class and beating up dazai at school, chuuya came home to your shared apartment. You were already home (lazy ass), he barely even got to kick off his shoes and you were already reaching for his head and yet... for the first time he...he smacked your hand away!?
"No." For a second, even he was stunned at his little moment of defiance. Chuuya always came home moody from school (he had to deal with dazai for 7 hours) but attacking you for trying to show affection definitely crosses the line.