FineBooks scores 14 open OCR models on 2,165 old book pages

FineBooks, a collaboration between Hugging Face and EleutherAI, has published a public leaderboard that scores 14 open OCR models against 2,165 expert-transcribed pages from the Biodiversity Heritage Library, drawn from 18th- and 19th-century printed books in English, French, German and Latin. The top model, rednote-hilab's dots.mocr at 3B parameters, hit 97.6% reading accuracy. The two runners-up are both sub-2B: OvisOCR2 at 0.9B scored 96.9%, and PaddleOCR-VL-1.6 at 1B scored 96.1%.

The write-up frames this as a training-data problem, not a library problem. "Public domain books are one of the largest sources of long-form text that can be freely used for training open AI models — but only if the text quality is good enough," Sebastian Majstorovic and Daniel van Strien write in the accompanying post. Full runs across the ground truth cost between $0.50 and $3.00 on Hugging Face Jobs; the cheapest model, PaddleOCR-VL-1.6, comes in at $0.34 per thousand pages. Van Strien is one of two researchers we track who circulated the leaderboard on launch.

The leaderboard reports two CER variants: a "diplomatic" score that counts silent modernization of archaic letterforms like the long s (ſ) as an error, and a "reading" score that tolerates it. That split is why the same page can look like a solved problem for AI training and an open one for scholarly transcription. The ground truth itself covers only Antiqua typefaces in four languages. Fraktur, non-Latin scripts, handwriting and multi-column layouts sit outside scope.

"In most cases, however, these new OCR models have been primarily trained on and optimized for modern documents," the authors note.