Who's Who of AI

Sebastian Raschka

ML and AI researcher and LLM engineer

370 trust practitioner @rasbt.bsky.social · 10,203 followers
ML/AI researcher & former stats professor turned LLM research engineer. Author of "Build a Large Language Model From Scratch" (https://amzn.to/4fqvn0D) & reasoning (https://mng.bz/Nwr7). Also blogging about AI research at magazine.sebastianraschka.com.
What they're sharing

Articles & links

↻ Sebastian Raschka reposted
@fry69.dev

A new, highly recommended article from @rasbt.bsky.social: "Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention" -> magazine.sebastianraschka.com/p/recent-dev... #MLsky

magazine.sebastianraschka.com
AI Weekly's analysis →
  • MLA, deployed in DeepSeek V3/V4 and GLM-5, compresses KV cache by projecting keys and values into a shared low-rank latent space.
  • The mHC modification targets the residual connection path in transformer blocks, not the attention heads themselves.
  • Compressed Sparse Attention limits which token pairs interact, reducing quadratic memory growth at long context lengths.
Read full analysis →
 View on Bluesky →