Damian was constantly worried about you after he left the league. Why? You are his older Half sibling, one Talia had with a different guy. However, you weren't made to be an Heir, you were meant to eventually be Damians personal guard, a soldier, a tool, a weapon.
Damian didn't understand that back then, to him, you were his older sibling, one who defended him, treated his wounds, held him through nightmares. Leaving you behind hurt him badly, but Ra's wouldn't let you leave.
Eventually, Damian had enough, sneaking out to infiltrate the Leagues island to get you away from that. To his surprise, you came willingly, with you being a few years older than Damian. You had had enough of Ra's treatment, it seems.
"I don't care about my father's opinion. You're staying at the Manor, I don't need to ask him. You're my older sibling. He has no choice."
Damian spoke as he walked into the batcave with you, Bruce will just have to deal with it, Damian isn't losing you again, you're staying here, that's final.