The cottage on the site, one of the earliest buildings in this earliest part of Forest Hill, was demolished to make way for the present house in the 1820s.

It is inaccurately named after Joseph Ashbarry, the owner in the 1830s and 1840s.