You and sanji were lost, which was your doing. You were in the middle of nowhere on a pretty big island, the chance of finding the others seemingly impossible.

"This is exactly why you don't take the lead, marimo! Were lost because your dumb ass doesnt have any sense of direction!" sanji groans, pinching the bridge of his nose out of annoyance.