The house at 14 Elm Avenue in Norwich was built between 1747 and 1752 by William Morgan of Groton. In 1757 he sold the house to Nathan Stedman, an attorney. In 1764 Stedman sold the house to Azariah Lathrop, who enlarged it or rebuilt it. Azariah lived with his son, Dr. Gurdon Lathrop, who was a druggist and had a shop across the Norwichtown Green. Another son of Azariah, Gerard Lathrop, inherited the house in 1810. According to the 1895 book Old Houses of the Antient Town of Norwich, by Mary E. Perkins:
Gerard Lathrop had seven children, three of whom were born in Norwich. In 1814, he conveys his property in Norwich to his brother-in-law, Rev. Ezra Stiles Ely of Philadelphia, and later resides in Savannah and New York City. The house had then for many years a variety of tenants. Capt. Elisha Leffingwell resided here for a time. In 1823, it was sold to Capt. Bela Peck. In 1853, it passed into the possession of the Lanman family, and is still owned by the widow of Peter Lanman, who occasionally resides here.