huggingface.co web signal

CurvDelta Confines LLM Weight Updates to a Low-Curvature Space

TL;DR

  • CurvDelta parameterizes LLM weight updates inside a low-curvature subspace derived from the preservation loss, keeping deltas there throughout training.
  • The framework uses Kronecker-factored approximate curvature (K-FAC) plus a factorized projection, with a low-rank sibling called CurvLoRA.
  • On TOFU and MUSE unlearning benchmarks, retention lands close to the from-scratch retraining reference, with better forgetting-retention balance than baselines.

A new paper from Zeguan Xiao, Zhuocheng Liang and more than a dozen co-authors tries to solve the old fine-tuning problem of adapting a model to one thing while quietly breaking what it was already good at. Their framework, CurvDelta, constrains weight changes to a "low-curvature space in which parameter updates leave the preservation loss nearly invariant."

The geometry is built with Kronecker-factored approximate curvature (K-FAC) and a factorized projection, which the authors say makes the construction "practical for LLMs." There is also a low-rank sibling, CurvLoRA, cast "within the same preservation geometry."

The authors frame the break from standard low-rank methods bluntly: "Those methods often choose the space using heuristics based on weights or activations. We derive it from the local curvature of the preservation loss."

On math reasoning and code generation, CurvDelta reports "competitive downstream performance while better preserving general capabilities." On the TOFU and MUSE unlearning benchmarks, the authors claim their methods strike "a better balance between targeted forgetting and knowledge retention than the baselines, with retention performance close to the from-scratch retraining reference."

The abstract publishes no per-task scores, no baseline deltas, and no scale sweep; those will have to come from the project page and the PDF. It lands amid a steady run of training-time surgery work. Our fine-tuning tracker has logged 64 adjacent stories in the last 90 days.