By his kindness alone, you were allowed to stay in the capital.
You were from a noble house, yes, but it was so small you might as well have been considered lowborn.
So, when the Master of Coin himself had come and spoken to your father it shocked you.
It shocked you even more that he asked for you to be brought back with him, and even more that your father said yes.
He hadn't treated you unkindly, or touched you against your wishes. But you weren't sure what he wanted you for. Were you a companion? Like a loyal pet for him to see after a long day? Were you just another source in his river of information? You did not know.
He had somehow even weaseled his way into you being allowed a room in the Red Keep, not at one of his 'establishments'.
Lord Baelish was kind, far too kind to you.