Andreas Kirsch

This work builds on LCKV (final-layer caches + KV matching), CLA (cross-layer reuse), and Huginn (recurrent depth), but we test cache-history mismatch and repair in a tied core using LSTM cells aclanthology.org/2024.acl-lo... arxiv.org/abs/2405.12981 arxiv.org/abs/2502.05171

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach arxiv.org

This was written in personal capacity and only speaking on my behalf. Article: www.theatlantic.com/technology/...

Generative AI Is an Engineering Disaster theatlantic.com

• NoPE (Kazemnejad et al.) arxiv.org/abs/2305.19466

The Impact of Positional Encoding on Length Generalization in Transformers arxiv.org

• NoPE (Kazemnejad et al.) arxiv.org/abs/2305.19466

The Impact of Positional Encoding on Length Generalization in Transformers arxiv.org

• BAM (Bianchessi et al.) arxiv.org/abs/2505.22842

Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation arxiv.org

• BAM (Bianchessi et al.) arxiv.org/abs/2505.22842

Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation arxiv.org

• Llama 2 Long (Xiong et al.) arxiv.org/abs/2309.16039

Effective Long-Context Scaling of Foundation Models arxiv.org

• Llama 2 Long (Xiong et al.) arxiv.org/abs/2309.16039

Effective Long-Context Scaling of Foundation Models arxiv.org

• YaRN (Peng et al.) arxiv.org/abs/2309.00071

YaRN: Efficient Context Window Extension of Large Language Models arxiv.org

• YaRN (Peng et al.) arxiv.org/abs/2309.00071

YaRN: Efficient Context Window Extension of Large Language Models arxiv.org

Also cited: • ALiBi (Press et al.) arxiv.org/abs/2108.12409

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation arxiv.org

Also cited: • ALiBi (Press et al.) arxiv.org/abs/2108.12409

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation arxiv.org

I work at Google DeepMind. This won't make me popular. But it's all public reporting: 2014: DeepMind reportedly sold to Google on conditions: no military use, independent oversight 2026: a Pentagon contract for "any lawful government purpose" Not one safeguard survived intact

I left Google DeepMind at the end of last week. I loved the research, especially the IMO project last year and working with so many exceptionally talented people on Gemini. Thank you to everyone I worked with, esp my managers and teammates. I have learnt so much from you all! 🫶

A first (short & compute-constrained) research project after leaving Google DeepMind: an LSTM-style tied transformer with one shared core KV cache The main question is how to train for the cache histories that the model produces recursively at inference

Another happy read that we should ponder and reflect on: Forethought's "AI-Enabled Coups". It's a careful, refreshingly unhysterical paper on how a small group (or literally one person) could use advanced AI to seize a state. Ofc: in personal capacity, not on behalf of Google

The Atlantic @TheAtlantic says generative AI is "an engineering disaster." I had Claude fact-check all 23 checkable claims against primary sources: 7 check out · 7 need context · 9 don't hold Verdict: the economics hold up. The computer science doesn't (I checked it too) 🧵