In Historic Litchfield, 1721-1907 (1907), Alice T. Bulkeley writes:
The house now occupied by Mr. Elbert P. Roberts, one of Litchfield‘s real estate dealers, on the corner of North and East streets, was built in 1792 by Charles Butler, cashier of the Litchfield Bank. It was originally a story and a half gable-roofed house. In the early part of the nineteenth century [1813] it was bought by Frederick Deming, father of the present Mr. Frederick Deming of North street, who enlarged it and built on the east wing. When Mr. Deming moved to New York he sold the place to Oliver S. Weller, and the latter built the small building where the school now is, for a small store, where he sold dry and wet goods, chiefly the latter. After his death Mrs. Weller continued the business as long as she lived, when the house went to two nieces in Woodbury who are its present owners. On the death of these ladies the house will be the property of St. Michael’s Parish Church.