The original village of Walmer is, today, usually referred to as Upper Walmer. It developed around a Manor House and an associated farm and the Old St Mary's Church, both built in the early 12th century.

Cottages for workers and tradesmen soon followed and by Victorian times the small agricultural community had become a thriving village containing the Manor House (Walmer Court), its farm and the church together with a school, shops, inns, a large brewery and a convent with a chapel.

Across Dover Road from the brewery was a large builder's yard owned by the Denne family. Major players in brick-making and building trades for over 400 years, they built the seawall in Ramsgate as well as houses and civic buildings all over East Kent. Wellington House on Dover Road, owned by Mr. William Denne, was sold in 1887 but the adjacent builder's yard was retained and only closed at the end of the 20th century when it was developed as St. Margaret's Close.

Unsurprisingly, many other buildings at Upper Walmer have altered over the centuries. Some of the shops, the schoolhouse and two of the inns are now converted into dwellings.

However, the old St Mary's Church - set in a burial ground in Church Street - remains and communion services are still held there every Thursday morning. The tiny church features rough flint walls with a single aisle, nave and chancel. It was possibly built as a chapel for the d'Auberville family who came with the Norman Conquest. The ruins of their original Manor House can be seen from the churchyard and are now an Ancient Monument. A yew beside the church porch is estimated to be 1,000 years old. The Duke of Wellington regularly attended services in the church and is rumoured to have had a tendency to nod off if the sermon proved too long.

Walmer Court, with its small flint gatehouse and long wooded drive, has been converted into flats. Much of Upper Walmer is included in a designated Conservation Area which offers important safeguards against unsympathetic future development.