Omega-S Beats EWC on Llama-3 Retention Without Old Data

A single-author preprint from Alberto Acedo, posted to arXiv, claims a weight-matrix-only penalty called Omega-S can hold on to a model's prior skill during fine-tuning better than Elastic Weight Consolidation, without any of the machinery EWC normally needs.

The setup is Llama-3-8B with LoRA, fine-tuned from code to prose, with HumanEval used as the retention probe across ten seeds. The paper reports pass@1 climbs from 0.173 to 0.238 versus no regularisation, a retention ratio moving from 62.9% to 84.1%, and that Omega-S beats tuned weight decay on 10 of 10 seeds and tuned EWC on 8 of 10, with sign-test p-values of 0.002 and 0.014 respectively. The wall-clock story is that it is 'three lines in an existing training loop' and adds under 4% to the cost of a step.

The reason a team would care is practical. EWC needs the Fisher information matrix computed on the old task and stored copies of prior weights, both of which are often unavailable in production settings where the original training data has been rotated out, licensed away, or covered by retention rules that forbid keeping it. A regulariser that reads only the current weight matrix removes that blocker for LoRA-style continual fine-tuning.

The paper is unusually candid about its own internals. Acedo designed Omega-S around a Tr(A^3) objective built from four factors, then measured which of those factors actually move the gradient, and reports that three of them have elasticity at or below 1e-4 against 9e-3 for the last one. The composite, as implemented, reduces to what he describes as 'a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones.' The topological framing is scaffolding; the working penalty is simpler than the derivation suggests.

The obvious limits are that this is one author, one base model, one adaptation direction, and one retention probe, and the mechanism note means an independent group should be checking whether the win is really Omega-S or a happily-tuned row-magnitude penalty in disguise. If it replicates, PEFT libraries could add it as a flag and teams that cannot keep old data around get an option they did not have last week.