Underwood lab releases 866-question 1831-1930 LLM benchmark

A team led by Ted Underwood at the University of Illinois has released Chronologic-EN-1.0, a benchmark of 866 questions drawn from 187 period sources that measures whether language models can answer in the voice of an English-language writer situated between 1831 and 1930. Details and a 100-question public sample are on GitHub; the accompanying paper is on arXiv.

The scene the benchmark imagines is a modern user asking a model to reply as, say, "an American fashion writer in the 1870s" or an Indian historian of the 1920s, a plausibly situated persona rather than a named individual. Every question ships with metadata naming date, nationality, often genre and authorial background, and, in the authors' words, "ground-truth answers are taken from period texts rather than written by us." The full 866-question set is gated behind controlled HuggingFace access to prevent contamination.

Scoring runs on two tracks. A likelihood pass compares per-token log-probabilities across candidates. A free-generation pass grades responses on substance, using LLM judges with Bradley-Terry calibration, and on style, using fine-tuned DeBERTa classifiers that predict composition date and flag AI-generated text. Thirteen models were tested.

The paper reports that generative tasks are harder than discriminative ones, and that reasoning models can typically identify weaknesses in their own generated answers. Models pretrained exclusively on historical text lead when scored by answer likelihood but "cannot compete with commercial models in free generation." The authors' summing-up: "None of the models...represent historical contexts in a fully persuasive way yet, but progress toward that goal is evident."