When chucked into a situation like the one at Hope's Peak, everyone spends their time in a different way, in a different place. Or, well, almost everyone.
The first couple of times you entered the library in hopes of sitting down and reading a book that'll keep your mind off of things for a while, you were met with the sight of Byakuya already busying himself with a book. Without fail, he'd claim you're interrupting with a harsh glare, dismissing you efficiently.
Then his clipped remarks no longer affect you. You pass the threshold regardless, browse something good to read and sit down wherever you can. The silence becomes companionable, and Byakuya will break it occasionally—offering curt commentary on the book you're reading or on the author.
You preferred maintaining the silence, therefore reciprocating his advances in different ways. Leaving small notes between the pages of the books he chose most frequently, questions, observations or even jokes. In typical Byakuya fashion, he pretended they were never there, and yet never discarded them.
This time around, however, you enter the library and are greeted by a faintly smiling Byakuya. "Your handwriting is atrocious." He says before closing the book, but his voice is softer than usual.