The Hartley Manor is mentioned in Domesday Book as part of the vast possessions of Odo, Bishop of Bayeux. On his disgrace, the Crown seized it. Then it was given to the family Montchensey. It was then passed to the Pembrokes under Edward I and II and then to Richard FitzAlan of Arundel under Richard II. It has then changed hands many times. 

Research shows that men of the Stone Age dwelt here and, after them, the Celts and Romans. Then came the Saxons. Any ‘worship’ by those generations would have been at the highest point of the region and that is where the ancient Parish Church of All Saints still stands. It was not until the 11th century that it was built of stone. Since then, the actual structure of the church has been little altered. A will of 1473 shows that already in pre-Reformation times it was called All Saints. That will left money to ‘the three priests’ for Masses for Thomas Cotyer’s soul. From this it can be assumed that there was a chantry at Hartley. This spiritual life was destroyed during the religious revolution of this land. 

Throughout this turbulent history, a small cottage stood undisturbed within the Hartley Manor, and still stands there, having seen half the life of England. One end and the foundations of Middle Farm are 11th century. To this the sixteenth century added somewhat and the twentieth century even more. No one quite knows the age of its ‘barn’. It probably saw the Reformation and had been thatched by members of the same family, living in the village, for well over a century. In 1876, Hartley was described as “a quiet out of the way place with hop gardens on every hand, three or four comfortable farmhouses, a smith’s forge and a church dedicated to All Saints”. It is likely that Middle Farm was one of these four comfortable farmhouses. It was set in 94 acres.