Reverend Joseph Ayer (1793-1875) was pastor of the Congregational Church in North Stonington from 1825 to 1837. Sometime during that period his house, located at 94 Main Street, was built. He then moved to Sprague where he became minister of the Hanover Congregational Church. North Stonington village was once known as Milltown and as related in his obituary in The Congregational Quarterly, Vol. XIX, No. 2 (April, 1877):
At the time he commenced his residence in Milltown, a village within the bounds of his parish, there were in that small village ten places in which intoxicating liquors were sold in larger or smaller quantities, — eight stores, and two taverns. Within a short time he was permitted to see them all closed, or cleansed of the fumes of alcohol, — an achievement hardly to be paralleled in the annals of the temperance reform.