The house at 1146 Boston Post Road in Old Saybrook was built c. 1800-1803 for William Chalker. It originally stood on the opposite side of the street but was moved and an addition built when the road was straightened later on in the nineteenth century. Around that time the house was acquired by Daniel C. Spencer.
A wealthy merchant, Daniel Chapman Spencer (1823-1906) started his business career as a store clerk and then was a traveling salesman with a stock of goods carried in a peddler’s wagon. He then worked for Moulton, Plympton, Williams & Co., one of the leading wholesale dry goods firms of New York. After that company went out of business he moved on to Claflin, Mellen & Co. in New York, at the time the second largest dry goods store in the United States and soon to become the largest. He ran the company‘s notion department for thirteen years, until he broke down from the strain and decided to retire on January 1, 1868. He chose to retire to his hometown of Old Saybrook. As described in the History of Middlesex County, Connecticut with Biographical Sketches of Its Prominent Men (1884):
Mr. Spencer had previously purchased a number of acres contiguous to the old homestead property in Saybrook, known as the Chalker farm. Here he retired to spend his days. The old place was enlarged and improved and soon made to “blossom like a rose.” The meadows were turned into cranberry patches on which he spent several thousand dollars in working and improving. He surrounded his residence with trees and flowers until it now has the appearance of fairy land. Amid these surroundings he soon recovered his health and then devoted his energies to making such public improvements in the town as should tend to attract others to this beautiful spot selected by Col. Fenwick as the “garden spot of the earth,” more than two hundred years ago.
Spencer owned greenhouses across the street from the Chalker House. He was also involved in the development of the Borough of Fenwick, including the building of the hotel known as Fenwick Hall.
The Chalker-Spencer House was altered around 1880 when the original roof was replaced by a Mansard roof. It was later used as a boarding house.