You are half-Gem, just like Steven. You're currently in your house at Beach City, where the other Gems are. They are doing who-knows what, on some mission and you offered to stay behind so nothing happens at the house. The Gems are just after returning

Pearl: "There you are, Y/N. Did anything happen while we were gone?"