The ruins of Old Walmer Court are located off Church Street in Upper Walmer. Close to the Blessed Mary of Walmer church, the ruined remains are believed to be of a semi-fortified manor house built by the d'Auberville family in 1120 during the reign of Henry I. It was surrounded by a wide dry moat, which also enclosed the nearby church. This also dates from around 1120 and was very possibly built by the d'Aubervilles as a chapel to their Walmer Court house. Excavation works on the site have discovered pottery dating from 1150-75.

The original manor building is thought to have been a square-shaped two-storey hall house, flanked by turrets on the corners. On the west side, an external stair led into a forebuilding at first floor level. Some historians believe that the north-east and north-west towers were added later in the 12th century using Caen stone imported from Normandy.

In Victorian times the ruins were used as a tea garden. Today, the remains comprise a roofless rectangular structure, set in a 1/3-acre private site with public access only by agreement with the site's owner. Very little remains of the upper floor of the former hall house but two undercrofts - with their flint walls - and parts of three of the turrets survive. The tallest and most substantial part of the remains forms part of the neighbouring church boundary wall and can be viewed from the churchyard.

Considerable work has been carried out in the past few years by the current owners Bryan Wilding and John Kirkbride. They purchased the site for £44,000 at an auction in 2007 and, since then, much has been done to safeguard the remains and clear and landscape the surroundings. In a local newspaper article published in March 2012, Mr Wilding considered that the ancient remains could officially be described as a "proto-keep" type of castle, as they had been created with a dry moat and no outer wall. As such, it, of course, predates the better-known Walmer Castle, built by Henry VIII in 1539.

English Heritage lists the building as the " Medieval Manor House, Walmer", recording it as a "monument of national importance" under the Ancient Monuments and Archaeological Areas Act 1979.

Also listed

Tucked away behind the Old Walmer Court remains is the present-day Walmer Court - an historically important building, currently sub-divided into a number of private apartments. English Heritage lists the building under the Planning (Listed Buildings and Conservation Areas) Act 1990 for its special architectural or historic interest. It describes the building as 18th century comprising "three storeys stuccoed. Slate roof. Eaves cornice with modillions. two 3-light bow windows on the ground and first floors with a dentilled cornice over them. Doorcase with fluted pilasters and pediment. Lower wing of two storeys and one window to the East end addition with sloping roof to the West. The back door dates from the C17 with a glazed peephole in it."

The neighbouring Church of St Mary The Blessed Virgin is listed by English Heritage as a Grade II* building, c1120 with an Early English chancel. It notes that it was extended C17 and c1826 (but these phases do not survive) and was altered and partly rebuilt in 1898.