Adam's library was located at the east end of the south range at Audley End, and was formed out of a single-storey pavilion, in which Lady Portsmouth had installed a library in the 1750s. As this pavilion was rebuilt it allowed Adam to install higher ceilings than in the majority of the house.

Adam's library was demolished in 1825 along with the study and supper room. His ceiling, the executed design for which is not entirely certain, was destroyed when the space was divided into a bedroom and two dressing rooms. At this time a new library was installed in the storey above where some of Adam's original bookcases survive.