arXiv paper ties LLMs' tabular weakness to feature count

A new arXiv paper by Marta Garnelo and Wojciech M. Czarnecki tries to answer a question that has quietly nagged the tabular-ML world for a couple of years: why do large language models, which are competitive on so much else, lose to decades-old baselines when you hand them a table of numbers.

The authors set out five candidate explanations and try to falsify each: an inability to handle noisy or non-linearly-separable data, the linearised CSV format obscuring column structure, the tokenisation of numeric values, the number of test points classified per query, and the dimensionality of the input. According to the paper, the first four get falsified. Dimensionality is the one that survives, and it survives cleanly.

The setup is a sweep of random linear projections across thirty-one benchmark datasets, with the LLM lined up against nine methods in total. On that sweep, the LLM is reportedly the only method whose accuracy decreases as dimensionality grows; every classical baseline stays flat or improves. A separate behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method, with up to 91.6% grid agreement. Push to higher dimensions and no classical model reproduces its predictions, even when the classical model is augmented with tuned, dimension-dependent noise.

The honest caveat is what the paper does not give you. The abstract I retrieved does not name which specific LLM was tested, does not say whether fine-tuning or tool use would change the picture, and explicitly leaves the underlying mechanism unexplained. So take this as a sharp characterisation of the failure, not a diagnosis.

Still, if you were about to bet an enterprise tabular pipeline on a hosted LLM because the demos looked good on toy examples, the practical read is that the demos were probably running in low dimensions. Classical tools keep their moat on wide structured data for now, and the interesting product surface is likely hybrid stacks where the LLM does orchestration and a classical learner does the actual prediction.