Carter House, the oldest building on St David’s Island, was constructed around 1640. It was built by the descendants of Christopher Carter who was one of two crew members of Sir George Somers’ shipwrecked Sea Venture who remained in Bermuda when the other survivors left for Virginia in 1610.

The Carter House is unusual in that it is built of stone when most other 17th century homes in Bermuda were built of wood. A lower basement supports an upper living area that is reached by an outside staircase. The gable ends are supported by large chimneys. The home's builders wisely set the whole building into a hillside to protect it against the high winds that sweep through Bermuda during hurricane season and the months of January and February.

During the time when this part of Bermuda was occupied by US military personnel, Carter House was used as a beauty parlour. After the military left the island in the 1960s, the house was restored and now accommodates the St David’s Island Historical Society Museum.