Study: pretraining seed, not data, drives jailbreak holes

Jailbreak vulnerabilities in a language model track more closely with the random seed used in pretraining than with the training data itself, according to a new arxiv paper by Chi-Wei Chang, Mu-Tien Kuo, Kuan-Wei Lee, and Meng-Hsun Tsai.

The authors trained 150 1B-parameter models across 25 data recipes, three pretraining seeds, and two post-training seeds, collected 20 responses per model to a shared set of jailbreak prompts, and correlated the per-prompt non-refusal rates between every pair. Two post-trained copies of the same pretraining run correlated at 0.984. Change the pretraining seed and the correlation falls to 0.811; change the data recipe as well and it drops only slightly further, to 0.779. "which jailbreaks break a checkpoint depends much more on its pretraining run than on its data," the abstract reports, a pattern the authors say held at 150M and 300M parameters as well.

The attack side reads the same way. The best of 91 attack templates found on one checkpoint stayed the best after a new pretraining seed in only 27% of model pairs, versus 67% after a new post-training seed.

Each pretraining run also carried a signature. Across 3,211 prompts, per-prompt non-refusal rates identified a model's pretraining run among all 75 runs every time; a random 50-prompt subset still did so 98% of the time.

The authors cap their experiment at 1B parameters and do not itemize the jailbreak prompt set they used.