Located across from where Pearl Street splits from South Main Street, on a high bluff above the Naugatuck River, is a structure that was perhaps built as early as 1740. It was acquired around 1778 by E. Turel Whittemore and served as a tavern. At that point, the building was only one story high. The second story was added in 1867 by Martin Castle, who dismantled the building’s old chimney and used the stones to constructed the terraced wall in front of the property. On the northwest corner of the old tavern was a barroom, where in 1780 a group of Torries planned the robbery of the home of the Patriot, Capt. Ebenezer Dayton, which was located in Bethany. This infamous incident led to the dramatic kidnapping, in nearby Oxford, of the Patriot boy, Chauncey Judd, a 16-year-old member of the Oxford militia, who ran into the fleeing robbers. They were later captured and sent to Newgate Prison and Judd was freed. The Whittemore Tavern has housed various businesses over the years.