Sakana AI's PC-ALM trains 1000-layer nets without backprop

A Sakana AI paper claims to train 1000-layer residual MLPs on MNIST without a backward pass, using only per-layer dynamics, and land within roughly two percentage points of backpropagation.

The method, Augmented Lagrangian Predictive Coding, keeps predictive coding's local structure but attaches a Lagrange multiplier to each layer's constraint error. PC-ALM "maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier," authors Jeffrey Seely and Julian Gould write in the paper on Sakana's site. In linear networks they prove it "converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates."

The concrete numbers are small-scale. On Fashion-MNIST at width 32 and depth 32 with ReLU, MarkTechPost reports the paper shows 78.66% test accuracy for backpropagation, 68.13% for vanilla PC, and 77.75% for PC-ALM. The nonlinear analysis in the abstract stops at depth 128; the 1000-layer figure covers residual MLPs on MNIST at width 32, ReLU, five epochs, under a T=2L inference budget.

The authors describe what they call "ballistic" credit propagation across very deep networks, "with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation." A JAX reference implementation is on GitHub. Three researchers we follow posted the link the same day.