BPL and Harvard IDI Release 83M Segmented Articles from 135 Years of U.S. Newspapers

Found first: a primary source the press has not covered yet.

The Boston Public Library and Harvard Law School's Institutional Data Initiative have released an open dataset built from 1.47 million scanned newspaper pages spanning 135 years, from 1795 to 1930. The release includes 83 million individually segmented and classified article crops, along with the full pipeline code, on Hugging Face and GitHub.

What the source says

The BPL and Harvard IDI applied a machine learning pipeline to Massachusetts newspaper collections covering Boston neighborhoods, Worcester, Springfield, Salem, New Bedford, and others. Rather than running OCR across whole pages, the pipeline first segments each page into individual articles and then applies OCR per segment. Downstream steps classify content type (news articles, advertisements, birth notices, literary works, illustrations), extract named entities, detect language, and produce pre-computed embeddings. The collection spans material in English, Yiddish, German, Swedish, and French. A full technical report is available on arXiv (2608.18972).

Why it matters

Public libraries hold large volumes of historical text that has been inaccessible to ML researchers in structured form. The per-article OCR approach is the core technical decision: it yields cleaner text and meaningful provenance (article type, named entities, reading order) that page-level digitization discards. The BPL alone holds tens of millions of pages on microfilm, with roughly 3 million currently digitized and searchable through DigitalCommonwealth.org. The pipeline code is the durable contribution here, because other institutions with similar microfilm collections can now apply the same approach to their own archives without rebuilding from scratch.