In early records, the garden was referred to as the herb garden of the Elephant Pharmacy. After 1850, it was used for recreational purposes. In 1937 it became a public park.
The pharmacy garden describes the typical developement from monastic to pharmacy garden. The Tea pavillion of the Elephant Pharmacy was rerected by Holbæk Museum in the year 2000 at its old location in the Pharmacy Garden.