Durham‘s most elegant Greek Revival style house was built by Benjamin Hutchins Coe on Main Street in 1830. The son of a wealthy Middlefield farmer, Benjamin H. Coe married Lydia Curtis of Durham in 1823. He had ambitions to become an artist and in 1833 moved to New York, where he became an art-dealer and author of books on drawing. He was described by H.W. French, in Art and Artists in Connecticut (1879), as follows:
The drawing-teacher of wide fame, Benjamin H. Coe, was born in Hartford, Conn., Oct. 8, 1799. His pictures are quiet, pleasant views; but his merit lay in his teaching. F. E. Church and E. S. Bartholomew, beside many others, came to him for their first information. He lived as a farmer, till, as he said of himself, he was too old to learn more than the rudiments. Always having possessed a great fondness for art, he mastered these rudiments with wonderful activity and success. He possessed a remarkable faculty for imparting truths in a way to fasten them in memory; and a long life of teaching both private students and large classes in nearly all of the important cities of New England, New York, and New Jersey, sustains this reputation. He had a very large private school in the University Building, in New-York City; moving from there in 1854 to his present home in New Haven. He opened his last school there, which he carried on successfully for ten years; then gave it up to one of his pupils, and entered into the temperance-work, writing and distributing tracts, and working in ale-houses, with the vigor of a young convert. Within a year, failing health has somewhat interrupted this work.
The house was sold to Samuel Parsons, a successful dry goods merchant in New York City. Suffering from tuberculosis, Parsons (1788-1848) had retired to his home town at the age of 45. His widow lived in the house until 1887. John R. Smith, a painter and decorator, occupied the house starting in 1902 with his wife, Hester Eliza Coe, a cousin of Benjamin H. Coe. The house three additions and a porch, all dating to the nineteenth century.