Found first: a primary source the press has not covered yet.
A new paper finds that which jailbreaks succeed on a language model is shaped more by its pretraining random seed than by its safety training or training data. Chi-Wei Chang and colleagues trained 150 one-billion-parameter models across 25 data recipes and multiple seeds, showing that 50 jailbreak prompts are enough to identify a model's specific pretraining run at 98% accuracy.
What the source says
The team built 150 models at the 1B parameter scale, covering 25 data recipes, three pretraining seeds per recipe, and two post-training seeds per pretraining run. Per-prompt non-refusal rates correlate at 0.984 between models sharing the same pretraining run but different post-training seeds, dropping to 0.811 for different pretraining seeds on the same data recipe and to 0.779 across different data recipes entirely. The best-performing attack template for one model persists to a different post-training seed in 67% of pairs, but to a different pretraining seed in only 27% of pairs. Across the full 3,211-prompt set, the pretraining run is identified at 100% accuracy; a random subset of 50 prompts achieves 98%. The pattern holds at 150M and 300M parameter scales and across all 25 data recipes.
Why it matters
Jailbreak benchmarks typically evaluate one or a few checkpoints and treat the results as a property of the training recipe. This paper shows those results are substantially shaped by which pretraining seed was used, a variable that is not controlled for and rarely reported. A benchmark run on one checkpoint may not generalize to another trained on the same data with a different seed. For researchers certifying safety improvements or comparing training recipes, benchmark scores may reflect seed-specific quirks as much as genuine safety gains. The fingerprinting result also implies that a small probe set is sufficient to link a deployed model to a specific pretraining run.