The fort dates back to the 16th century and has changed hands many times. The Danish built the first substantial fort in the 1660s, though rebuilding has occurred on many occasions since. It began to serve as the seat of government under British colonial rule and continued to perform that function up to present.
Osu Castle remains the seat of government in Ghana, employing 2,100 workers. The most important functions are carried out in the castle itself, but other buildings are also used. Many international dignitaries have visited the castle while in the region, including U.S. Presidents Richard Nixon, Bill Clinton, and Barack Obama and German Chancellor Gerhard Schröder. Additional rooms were built in order to accommodate Queen Elizabeth II's visit in 1961, one year after Ghana became a republic.
The present castle is made up of various extensions to the original and is thus in an unorthodox shape. It has many facilities for the use of employees, including a clinic, cafe, shopping centre, and post office. It also still accommodates a permanent garrison. The extensive gardens feature a wide variety of plants, both local and imported, and employ 30 people. They are used for the president's outdoor receptions and parties.