We usually have plenty of time to get to Cape May from either the anchorage at Whorton Creek or Still Pond, at 9 knots. You must mind weather and avoid tide against wind for your Delaware Bay trip. The anchorage inside Chesapeake City has been too crowded whenever we went by, at least for us. Once when we got a late start, we stayed at Summit North Marina further down the canal, which was very nice.
On the Delaware, you can anchor behind Reedy Island not far south of the canal mouth; get a good set because there is current coming and going. You can anchor in Cape May Harbor, to either side of the CG station, but it has always been too crowded when we went through, plus we like spending time in the town. We have always used Utsch's there.
I will second the recommendation of the Brigantine anchorage on the north side of Absecon Inlet (Atlantic City). We have stayed there a few times; it is fun to take the dinghy around and explore the area, a weird combination of pretty wetlands and urban grime, plus the casino towers lit up at night. I have not actually set foot on land there, though you can go over to the Farley State Marina (Trump) and dock the dinghy there.
It is an easy an enjoyable run up the coast from AC to Sandy Hook. The anchorage in Sandy Hook proper is a little rolly, and the party, or "head" fishing boats whip right through there and up close to you. Its a short jump over there to Great Kills, where we use either Atlantis or Mansion Marina, since we stay for a few days and go into the city from there via express bus. Very nice restaurants and village; one of our favorite stops. Plenty of shopping close at hand for provisioning. They are discouraging anchoring there and dinghying in. You can also try the Richmond County Yacht Club for a mooring. The state marina is inconvenient to the village and bus lines.
If you are going up the Hudson, we really enjoy anchoring behind Croton Point in Haverstraw Bay. Easy dinghy access to the park or over to the marinas, and trains to the city.
On close-to-NYC LIS, you can grab a town mooring for free (at least a year ago it was) in Manhasset Bay for two days; there is a water taxi service you hail on 9 to get your mooring and to take you to shore, where there are numerous stores and restaurants. We love the anchorage in Oyster Bay, where it is easy to grab a train into the city. I could spend weeks there. Note Bene: the moorings there are $1.50 a foot per day!! Both Manhasset and Oyster have pump out boats too.
That's my view as a dawdling cruiser, I look forward to what the locals have to say. A lot of the decisions depend on whether you are just passing through or want to explore the surrounding cities and towns.