Sheffield's LambdaMix Trains One LoRA Per RLHF Objective, Lets Preferences Shift at Inference

Sheffield researchers Karouzos and Aletras published LambdaMix, a parameter-efficient GRPO variant that trains one LoRA per objective over a frozen base model, then uses an inference-time preference vector to mix adapters and normalize rewards. Tested on four models for helpfulness-harmlessness and math-plus-instruction-following, it beats conditioned and policy-merging baselines on hypervolume while using fewer completions and fewer stored adapter params than specialist GRPO policies. Code is at github.com/ckarouzos/lambdamix.