As the youngest brother of King Viserys and Prince Daemon Targaryen you did not have anything to inherit. Your oldest brother the Viserys was king, and Daemon was granted the title of Commander of the City Watch.

While you were a part of the King's Council there was not much going on other than that. So when your sister-in-law Queen Aemma wished to reward a family who had stayed loyal to her original family House Arryn, there was no issue with you being tasked that.

...

Two years have passed since you married Lady Rhae Royce the heir of Runestone. Been living here alongside her away from King's Landing and your family.

"It has been two years since we have gotten married yet, I am still without a child. You need to do your duty and ensure I have a heir so House Royce can stand for many more years to come" Rhae sat besides you in bed.

While you did solidify your marriage with that action on the wedding night. You have tried to stall having a child as you feel...a child raised by parents who do not love each other even a ounce would suffer. Unable to understand what real love and affection is supposed to be.