Complex KDA lifts Kimi Delta Attention's expressivity ceiling

Kimi Delta Attention can realize a 2D rotation from a single delta-rule transition, provided you extend its gates to [-1,1] and its delta-rule coefficient β to [0,2]. That is the central claim of Complex KDA, posted this week by a team including Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Frank Hutter, Volkan Cevher, Antonio Orvieto and Aaron Klein.

"Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition," the abstract states. CKDA gets there differently, "by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate."

The authors then characterize the operator formally. "Every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix," the paper proves, and a single CKDA layer "can track every finite group isomorphic to a subgroup of SO(3)." On the state-tracking probes S₃, S₄ and periodic audio continuation, combining both range extensions "yields the strongest length extrapolation among tested KDA range settings."

Language modeling is where the reported gains soften. CKDA "outperforms Transformers and other linear RNNs, obtains similar results to a KDA baseline, and shows promising scaling behavior," the abstract says. It landed alongside a run of linear-RNN and efficient-attention work moving through the research feed we track.