New Castle (formerly New Amstel) was founded by the Dutch in 1651 as Fort Casimir. The fort was placed on a sandy hook because of its commanding view up and down the river.
There are many parallels between New York and New Castle. New Amstel was built on the South River (now the Delaware) by the West India company, headed by Peter Stuyvesant. New York (formerly New Amsterdam) was built on the North River (now the Hudson) by the same company and also governed by Stuyvesant. Both rivers were discovered by Henrik Hudson in 1609. Both became English colonies (and were renamed) when King Charles seized all Dutch possessions in 1664 and granted them to his brother James, Duke of York.
New York is thus named for Charles' brother. According to the Encyclopedia Britannica, New Castle (in many early documents it's one word, Newcastle) was named for William Cavendish, (1592-1676) Duke of Newcastle, King Charles' long time supporter, Royalist leader and major financial contributor. His wife Margaret, Duchess of New Castle was perhaps "one of the first distinguished female writers in England".
Both towns were envisioned by the Dutch as colonies and trans-oceanic trading centers and were originally the sites of local government and commerce.
While New York grew enormously, New Castle was gradually bypassed by trade, transportation routes and a role in government, and thus has managed to retain an unparalleled charm and historic ambiance.