GPT-5 Knows More Finance Facts Than It Admits, FinDendro Finds

GPT-5 correctly answered 76% of controlled U.S. financial facts going back to 1999 while claiming it knew only 38% of them, and still got 61% of questions right when it said outright that it did not know the answer.

That is the headline result from FinDendro, a new benchmark of 2,226 dated U.S. facts covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions from 1999 to 2026. Controls for persistence and dominant outcomes keep chance accuracy near 0.27, the authors report.

Knowledge claims can persist beyond the measured knowledge boundary by up to seven quarters, the paper finds. Adding a historical declared date to the prompt, the common instruction to answer as of a past date, removes most of the model's prior-knowledge claims while leaving accuracy unchanged. The verdict is blunt: "Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff."

The authors position FinDendro as a replacement procedure: pick backtest periods based on demonstrated factual knowledge, not vendor-stated cutoffs or model self-reports.