The island was first explored by the Portuguese in the 15th century, who named it Ilha de Santo André after one of their sailors, buried there. The first permanent European settlers on the island were Latvians from the Duchy of Courland and Semigallia. In 1651, the Latvian settlers erected  Jacob Fort after Jacob Kettler, the Duke of Courland, and used it as a trade base. The Dutch briefly held the fort from 1659 until the English captured it in 1661; the Dutch formally ceded the fort to the English in 1664.

The English renamed the island James Island and the fort Fort James after James, the Duke of York, later King James II of England. The chartered Royal Adventurers in Africa Company administered the territory, which initially used it for the gold and ivory trade, and later in the slave trade. In 1695, the French captured Fort James after a battle with English sailors. In 1702, Fort James returned to British control. The fort was destroyed and rebuilt several times in this period, both in conflicts between the English and French and by pirates. On 13 June 1750 the Company of Merchants Trading in Africa assumed the administration of the Gambia. Between 1758 - 1779, the Gambia was part of British Senegambia.  The island was abandoned in 1870.