Castle House was originally a small seventeenth-century dwelling on the Bowood Estate, but additions were made to create a larger H-shape building in 1770. Adam was responsible for the southern range, a two-storey, nine-bay block, as can be seen in these drawings. Although the building was almost wholly demolished in the 1970s the south front of Adam's block was preserved.

We can see from Adam volume 44/37 that Adam intended to regularise the west court by the addition of small rooms to either side of the entrance, but according to King photographs show that this feature of the design was not executed. Instead the entrance was placed on the east of the hall with a Doric doorway.