Gemini-3.1-Pro Loses 32 Points on FinIndices Financial Tables

A new benchmark out of the financial NLP space makes a point that has been quietly nagging at anyone building LLM-powered analysis for banks and hedge funds. When you take away the formula hints, the model doesn't reason. It pattern-matches, and it does so badly.

The paper, posted on arXiv, introduces FinIndices, a benchmark that runs models over uncropped financial statements up to 32K tokens rather than the cropped tables and multiple-choice trivia most finance evals rely on. The headline result is that Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks once the explicit formula hints are removed from the prompt. The authors call this the 'Knowledge Bottleneck.' Their framing is unusually direct: the model memorized the formulas during pre-training, but that memorization is fragile and does not survive contact with a real statement.

They also identify a second failure mode, the 'Structural Bottleneck.' When a model is asked to generate multi-metric, multi-period tables, the kind of output an analyst would actually want, even systems that flawlessly execute isolated derivations regress to shallow heuristics. The paper describes this as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. If you have ever watched an LLM confidently copy the wrong row out of a filing, this is the phenomenon.

Why this matters for anyone shipping a finance product: the vendor pitch typically leans on prompted templates that supply the formula. That is exactly the setting where the 70.70% number lives. Strip the template, hand the model a raw filing, and you are closer to the 38.22%. The delta is the marketing gap. The paper does show supervised fine-tuning helps, with reported zero-hint gains of +8.54% on single-index tasks and +3.82% on tables, so the situation is not hopeless, but SFT of that shape is a real engineering investment and not a prompt tweak.

The honest caveat is that this is one paper reporting one model configuration inside a synthetic, adversarial benchmark, and it does not tell you how competing frontier systems fare on the same setup. What it does give you is a cheap evaluation template: run your finance pipeline once with the formulas in the prompt, run it again with them removed, and see how much of your reported accuracy was actually the prompt doing the work. That test costs a day and might save a rollout.