The house at 23 Hayward Avenue in Colchester was built in 1765 for Joseph Isham, Jr. after his marriage to Sarah, daughter of Dr. Oliver Bulkeley (who provided the land for building the house). Isham operated a store and served in the Commissary Department during the Revolutionary War. After his death in 1810, Sarah lived in the house until 1834. The gambrel-roofed building originally had a large center chimney, which was taken down around 1820 by Isham’s son, Ralph Isham, who replaced it with two smaller chimneys and used the extra stone to build the foundation of his new house, next door at 11 Hayward Avenue. From 1834 until his death in 1852, Benjamin Swan, Jr., lived in the house. Originally from Woodstock, Vermont, he had married Ralph’s daughter Ann and worked for the Hayward Rubber Company. Later owners substantially altered the colonial house, adding a tall central wall dormer projecting from the gambrel roof, a large cupola and a porch across the front of the house.