Frederic S. Dennis, in The Norfolk Village Green (1917), writes that the earliest library in Norfolk dates to 1761:
A library company was then formed, and about 150 volumes were collected; and this library remained in activity about thirty-five years, when it was dissolved, the books to be distributed among the original donors. In 1824 a second library was formed and incorporated with 142 volumes, besides periodicals. Like its predecessor it was short lived and dissolved in 1866. The books passed into the hands of Mrs. Charlotte Mills, and Miss Louise Stevens, who subsequently founded a third library, which was in the hands of a committee. This new Library was placed on a business basis and a yearly fee of one dollar was charged for membership. It continued for a year and its books formed the nucleus of a fourth Library. In 1881 Miss Isabella Eldridge opened a reading room in the Scoville house on the Green, and the books of the third Library were placed there.
Isabella Eldridge’s reading room was so successful, that in 1888 she decided to endow a library in memory of her parents, the Rev. Joseph Eldridge and Sarah Battell Eldridge. She hired architect George Keller of Hartford to design the Norfolk Library, which was constructed in 1888 and opened to the public in 1889. The library has a first floor built of red freestone, quarried at Longmeadow, Massachusetts. The upper floors feature fish scale shingles and the original roof had fluted Spanish tile, since replaced. In 1911, Keller designed a reading room, added to the rear of the Library. A later addition is the children’s wing of 1985, designed by Alec Frost and also constructed of Longmeadow red freestone.