Lambert: org design, not just compute, decides frontier training
TL;DR
- Nathan Lambert argues 'you ship your org chart' applies to AI training, and the core language modeling teams at major labs stay small as the companies around them grow.
- AI2's Tülu 3 post-training project ran roughly mid-July to November 21, 2024 with 20+ authors, about 1,000 checkpoints evaluated, and 100-600 GPUs running concurrently.
- For state-of-the-art models above 7B, Lambert says trying to train with fewer than 500 H100 GPUs is likely not worth it.
The leverage in frontier model work sits in team structure, not just compute. That is the argument Nathan Lambert makes on Interconnects, where he writes that "you ship your org chart" applies to training models and that the core language modeling teams at major AI companies stay small even as the companies around them do not.
Lambert's prescription is a small core team, co-located where possible, with engineers kept embedded alongside researchers and longer-horizon "evergreen" research spun out separately. He warns that when "senior directors mandate results" ahead of a model release, "the entire system will crumble."
He grounds the piece in AI2's Tülu 3 post-training project, which ran roughly mid-July through November 21, 2024 with 20+ authors "researcher-heavy relative to engineer-heavy," about 1,000 checkpoints evaluated, and 100 to 600 GPUs running concurrently across the 4-5 month span. Most experiments sat at 8B scale, with a handful reaching 405B. On compute floors Lambert is blunt: for truly state-of-the-art models above 7B, "trying to do so with <500 H100 GPUs is likely not worth it."
Shared on Bluesky by 1 AI expert
Originally reported by interconnects.ai
Read the original article →Original headline: Managing frontier model training organizations (or teams)