In addition to the research findings, this is the type of research I really enjoyed doing 😊 "On Hyperparameter Tuning on the Test Set" arxiv.org/abs/2610.05902
- A new arxiv paper by Fregonara, Viering and van Gemert tests test-set hyperparameter tuning on MNIST-1D, CIFAR-10 and three GLUE tasks.
- The authors find the inflation effect is real and significant but frequently small relative to other sources of noise.
- Model rankings remain essentially preserved after test-set tuning, so consistent test tuning may not invalidate benchmarks.