Stone House was built between 1771 and 1773 by the architect George Gibson for himself. In so doing he supports the contention that architects build quirky homes for their own use.

Gibson was also responsible for St Mary's Church, Lewisham, and John Julian Angerstein's house, Woodlands, in Blackheath.

Stone House was called the Comical House on the tithe map.

While much of its original garden has been built on, the house itself has been recently restored.