Residual signatures reveal LLM lineage from weights alone
TL;DR
- A new arxiv preprint by Aman Singh Thakur and Rayan Khoury introduces 'centered residual signatures' to verify language model lineage from weights alone.
- The paper reports perfect separation (AUROC=1.0) on residual-MLP and GPT-2 benchmarks and 76x speedup versus the nearest robust baseline on GPT-2.
- The technique distinguishes fine-tuned, merged, pruned, and quantized descendants from independent models across six language-model families, including LLaMA-2.
Language models carry structural fingerprints in their weights, according to a new arxiv preprint by Aman Singh Thakur and Rayan Khoury, that reveal their training lineage even after fine-tuning, merging, pruning, or quantization. The technique they call 'centered residual signatures' aims to 'identify checkpoint-specific structural patterns across residual blocks,' derived from weight patterns alone.
The paper reports 'perfect separation (AUROC=1.0) on residual-MLP and GPT-2 benchmarks' and says the method ran '76x faster than the nearest robust baseline on GPT-2.' The 'projection-pairing signal was validated across six language-model families,' and the authors say they 'successfully identified relationships among publicly available LLaMA-2 checkpoints.'
Two experts in our Who's Who directory circulated the link. Per-family accuracy numbers aren't in the abstract, and of the six families only GPT-2 and LLaMA-2 get named.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Training Leaves Traces: Centered Residual Signatures for Language Model Lineage Verification