I spent a day on a secluded island that’s a 5-minute ferry ride from NYC’s ritziest vacation spot. It was immediately clear why the area is known as the ‘un-Hamptons.’