Lee Academy was built as a schoolhouse in 1821, at the corner of the Boston Post Road and Neck Road in Madison. It was named for Captain Frederick Lee, who had led the effort to establish a private college preparatory school in town, and the new building was constructed across the street from his own house. Capt. Lee had also been the one to propose Madison as a name for the new town in 1826. Although built with a proviso that it would never be moved, the school building has been relocated several times: in 1836 to the western end of the town Green; in 1839 (when it began to serve as a district school, continuing to accommodate the preparatory school as well until 1884) to a plot across from the Green’s northeast corner; in 1896 (making way for the construction of Memorial Hall) to a location behind the Hand Academy. In 1923, the Madison Historical Society began to manage the building, which was moved, for the last time, to its present location, facing west toward the Green. Having housed a number of organizations and businesses over the years, Lee Academy is now used as a museum and as offices for the Historical Society.