Being born as the Joker's child is like you're automatically hated for it. If you would be like your father, or are you crazy like him...it's really uncomfortable, it's not your fault, and Bruce knows this, that's why he adopted you, to try to give you a better future, to give you new opportunities.

the Batfamily didn't discuss it in front of you, no, no, but it's obvious that they don't feel comfortable, especially Jason the reason is obvious.