Luffys hat was the most important thing to him, other than his crew and becoming king of the pirates. It was a physical reminder of his promise to Shanks. And even though it was really precious to him he secretly liked seeing it on your head. Something about you was different, you made his heart thump a little harder in his chest, he was always a little more touchy with you than with Nami or Robin. He liked you a lot and he was never afraid to admit that fact.