According to Pevsner, Castle House was an early seventeenth-century house which received its name from a castle which is said to have once inhabited the site. It belonged to Lord Shelburne's estate at Bowood, and was the home of Daniel Bull (d.1768), whom Lady Shelburne mentioned in her diary on 25 November 1768:
The intense cold killed in one night our poor orang-outang, or man of the woods, and possibly hastened the death of old Mr Bull which is a serious loss to Lord Shelburne, he being the most faithful, able and zealous agent.
Following Daniel's death it appears that his son David was permitted to remain at Castle House. It is possible that he fulfilled a similar role to his father, as Jeremy Bentham recorded in his 'Remembrances of visits to Bowood, 1781-85' in The works of Jeremy Bentham, 'Mr Bull, who managed, I think, the borough of Calne'. It was Daniel who, in 1770, commissioned Robert Adam to extend the T-shaped house into an H-shape by the addition of a large southern wing. According to King it is unlikely that Adam executed any interior decoration aside from friezes, although there is no evidence to confirm this.

Castle House was purchased by the local council in 1961 who left it to fall into ruins, and it suffered a fire. In the 1970s the seventeenth-century portion was demolished, and the Adam wing was gutted for renovation as sheltered housing for the elderly.

Literature:
J. Bowing, (ed.). The works of Jeremy Bentham, 1843, Volume 10, p. 123; E. Fitzmaurice, Life of William, Earl of Shelburne, afterwards first Marquess of Lansdowne, 1912, Volume I, p. 396; A.T. Bolton, The architecture of Robert and James Adam, 1922, Volume II, Index pp. 6, 64; E. Harris, The furniture of Robert Adam, 1963, p. 48; B. Cherry, and N. Pevsner, The buildings of England: Wiltshire, 1975, pp. 156-57; D. King, The complete works of Robert & James Adam and unbuilt Adam, 2001, Volume I, p. 204

Frances Sands, 2011