The partly ruined Westenhanger Castle is located on what was the site of two medieval manor houses, Westenhanger and Ostenhanger. During the 14th and 15th centuries the houses were owned by the de Criol and Poynings families respectively, and in 1343 John de Criol was granted permission from the Crown to fortify his manor house. The alterations carried out to fortify his house included the construction of curtain walls, seven towers, and the rebuilding of the gatehouse on the western side of the enclosure. In the early 16th century Edward Poynings unified the two manors and in 1540 Westenhangar passed to Henry VIII who offered Thomas Poynings, the owner and Edward's son, land elsewhere in exchange. Henry VIII carried out minimal works to the manor and in 1542 ordered the laying out of a deer park. Just five years later, however, he granted Westenhanger to John Dudley, Earl of Warwick. The manor was again recovered by the Crown in 1566, but in 1585 Elizabeth I transferred the house and estate to Thomas Smith, a farmer. In 1701 the property was sold and most of the buildings were subsequently taken down. The present house on the site was constructed in the 18th century from the remains of a 16th century cross-wing of the main hall.
The site as it exists today consists of the manor house set within a ruined, walled enclosure with the remains of the 14th century towers, surrounded by a partly filled moat. The house is two storeys high with an attic and is constructed from red brick. It is an L-plan with a bastion to the north east corner. The entrance elevation has seven windows with a late 20th century porch. Conservation work began on the castle in 1997 and is open to the public at certain times of the year.