LoRA Speedrun opens wall-clock leaderboard on a single L40S

A new GitHub project called LoRA Speedrun is trying to do for parameter-efficient fine-tuning what nanoGPT's speedrun did for pretraining: freeze the task, freeze the hardware, and rank submissions by the one thing everyone actually pays for, wall-clock training time.

The setup, per the repo, is two tracks. Track 1 fine-tunes Qwen2.5-1.5B on GSM8K to at least 57.0% exact-match. Track 2 fine-tunes SmolLM2-1.7B on SQuAD v1.1 to at least 75.5% EM. Both run on a single L40S (48GB) inside a Modal sandbox, both cap trainable parameters at 30M, and both restrict you to the official data splits, no synthetic data or distillation allowed. Submissions come in as PRs with the training script; verification is three independent runs on fresh seeds, all three of which must clear the threshold, and the official time is the mean. There is also an adapter audit and a model and data hash check to catch tampering.

Why that framing is interesting is the metric. Most LoRA and PEFT arguments live in FLOP charts, adapter-rank ablations, and blog posts with one cherry-picked config. Wall-clock on frozen hardware collapses all of that into GPU-minutes, which is the number that actually shows up on someone's bill. Kernels, dataloader tricks, sequence packing, loss masking and rank choices all get to fight in the same currency.

The first record on the math track is Saivineeth147's 6m 05s at 61.1% accuracy, using sequence packing plus completion-only loss masking over two epochs. The baseline the repo quotes is 11m 57s, so roughly a 2x speedup at higher accuracy. The SQuAD track is open, with no records yet. The repo also lists a long menu of unclaimed ideas worth trying, including 1-epoch aggressive-LR schedules, data pruning, block-diagonal attention, QLoRA variants, DoRA, PiSSA, LoRA+, NEFTune, curriculum ordering, rank search, torch.compile and Unsloth kernels.

The honest caveats are the usual ones for any speedrun-style ladder. Wall-clock rankings on a fixed benchmark reward techniques that suit that exact benchmark and that exact GPU, and a small submitter pool early on means one contributor can dominate the board until others show up. The README also does not say who is committing to run the verification harness long-term, or what happens if Modal's free-tier credits change. Even so, the direction is the part worth watching: a public, reproducible, cheap ladder for fine-tuning research is the kind of thing that quietly turns opinionated blog posts into settled numbers, and that is useful to anyone shipping adapters on a budget.