A school was established by the town of Norfolk as early as 1768. Initially, students were taught in the church parsonage, until the the School Society built a small structure, used as a school and church conference room, in 1819. According to an 1899 speech by librarian Henry H. Eddy (quoted in the 1900 History of Norfolk): When John F. Norton was the teacher at the school, it
was so successful that by 1838 there were upwards of seventy pupils under his charge. The next year, the need of still greater accommodations being felt, an Academy Corporation was formed for the purpose of building an academy, and in 1840 such a building was erected on the east side of the Green, for the sum of $2,000. As the career of Mr. Norton had been so successful he was appointed first principal, and continued as such until duties outside of the town took him away.
As Frederic S. Dennis relates, in his 1917 book, The Norfolk Village Green:
The Town Hall, originally the academy, was built in 1840 and from that time on was used as the place for the transactions of town business, including voting. In 1846 a committee was appointed to confer with the proprietors of the academy with a view to the use of this building for town meetings. The lower floor is used for town meetings; the upper floor is the property of Mr. and Mrs. Carl Stoeckel; it was not unusual in early days to have one building owned by two or more parties. In addition to the school room above and the town hall below, there was constructed in the basement a lock-up, which has been built on the first floor by partitioning off a room.
Today, the building serves a different purpose, as the Norfolk Historical Society Museum.