The Pagoda was built in 1760 as a summer house for the Earl of Cardigan and was also occupied by other nobility and royalty.

It was enlarged in the mid 19th century and became a London County Council children's home a century later. It is now a private house again.

Its extensive grounds were built on in the 1860s. Though architecturally unusual, its history is similar to other large houses nearby, such as the Knoll or the Orchard.