Inference News — AI Trends, Updates & Tracker

Mention Trend — Last 20 Issues

Live Stories (104)

r/LocalLLaMA: AMD ROCm Now Runs Sanely Inside WSL2 for llama.cpp — Community Posts Working Build Instructions, Known Bugs Remain

github.com · 2026-06-02 · Score: 18

r/LocalLLaMA: llama.cpp b9455 Ships Multi-GPU SM Tensor KV Cache Fix — Critical Quantization Bug Affecting Multi-GPU Inference Now Resolved

reddit.com · 2026-06-01 · Score: 17

Intel Launches Crescent Island AI GPU at Computex 2026 — Up to 480GB LPDDR5X Memory, Xe3P Architecture Purpose-Built for Agentic AI Inference

tomshardware.com · 2026-06-01 · Score: 25

mistral.rs v0.8.2: Up to 2.8x Faster CUDA Inference Than llama.cpp on GB10, B200, and H100

reddit.com · 2026-06-01 · Score: 17

NVIDIA Launches Nemotron 3 Ultra at Computex — 500B Open-Weight Agentic AI Model, 30% Cheaper to Run Than Alternatives

nvidianews.nvidia.com · 2026-06-01 · Score: 26

r/singularity: MiniMax M3 Starts Rolling Out on API — 1M-Token Context Window and 9.7× Faster Prefill Than M2 Now Live

reddit.com · 2026-06-01 · Score: 22

r/LocalLLaMA: Developer Ports NVIDIA Parakeet Speech-to-Text to Pure C++/ggml — No Python, No PyTorch, Runs on CUDA, HIP, Metal, and Vulkan

reddit.com · 2026-05-31 · Score: 19

r/singularity: MineBench Finds Claude Opus 4.8 Costs ~40% Less Per Autonomous Build Than Opus 4.7 Despite Identical API Pricing — Chain-of-Thought Duration Dramatically Compressed

reddit.com · 2026-05-31 · Score: 16

r/LocalLLaMA: CS Student Ships mlx-Chronos — First Neutral Community Benchmark Leaderboard Comparing Four Apple Silicon MLX Inference Engines Across 520 Scored Questions

reddit.com · 2026-05-31 · Score: 16

r/AI_Agents: Developer Ships Open-Source Proxy to Compress Agentic LLM Input Tokens — Full Tool List Resends and Growing Histories Are the Real Billing Driver

reddit.com · 2026-05-31 · Score: 17

Netflix wiz creates app to slash AI bills, then open sources it

theregister.com · 2026-05-31 · Score: 18

r/LocalLLaMA: Controlled Benchmark Finds No Speed Difference Between Windows 11 and Linux for Medium and Large MoE Models in llama.cpp — Debunks Dual-Boot Myth

reddit.com · 2026-05-31 · Score: 16

r/LocalLLaMA: Flash Attention for llama.cpp on RDNA3 Achieves 47% Less KV VRAM Than Vulkan F16 With Near-Lossless Quality via Novel 4-Value Bit-Packing

reddit.com · 2026-05-31 · Score: 17

r/LocalLLaMA: Developer Publishes Method to Route Any Model Through Official OpenAI Codex Desktop App Without Code Modification via API Endpoint Impersonation

reddit.com · 2026-05-31 · Score: 16

r/ChatGPT: Puppetmaster Open-Source AI Orchestrator Claims 98% Token Cost Reduction via Complexity-Based Model Routing

reddit.com · 2026-05-31 · Score: 15

r/artificial: 'Llama Surgery' — Method Injects Learned Block-Sparse Attention Topologies Into Pre-Trained Dense LLMs Without Full Retraining

Reddit/r/artificial · 2026-05-31 · Score: 17

r/LocalLLaMA: OpenAI Job Posting for On-Device Transformer Inference Lead Signals Pivot to Local AI

openai.com · 2026-05-31 · Score: 21

r/LocalLLaMA: Cost Analysis of My $6.4K Local LLM Server

reddit.com · 2026-05-30 · Score: 15

NVIDIA Releases Official NVFP4 Quantization of Qwen3.6-35B on Hugging Face — First Major Chinese Open-Weight Model With Blackwell-Native Weights From NVIDIA

huggingface.co · 2026-05-30 · Score: 19

Parallax: Parameterized Local Linear Attention Scales LLA to LLM Pretraining With Hardware-Aware Kernel That Matches or Outperforms FlashAttention

arxiv.org · 2026-05-29 · Score: 18

Baseten Ships 8-Step FLUX.2-dev DMD2 Distillation — 6–8× Inference Speedup at Production Image Quality

baseten.co · 2026-05-30 · Score: 15

Groq Raises $650M to Pivot From Chip Maker to AI Inference Neocloud After Nvidia's $20B Licensing Deal Stripped Its Senior Team

techcrunch.com · 2026-05-29 · Score: 25

r/LocalLLaMA: Community Developer Benchmarks Multi-Token Prediction on Gemma 4 31B and Qwen 3.6 27B — Up to 3.34x Lossless Inference Speedup on RTX 6000 PRO via vLLM and llama.cpp

reddit.com · 2026-05-29 · Score: 19

r/StableDiffusion: Colored Noise Diffusion Sampling — Plug-and-Play Inference-Time Spectral-Bias Fix Ships With No Retraining Required (arXiv 2605.30332)

r/StableDiffusion · 2026-05-29 · Score: 18

r/LocalLLaMA: Developer Builds First 'Train LLM From Scratch on 8GB VRAM, No Cloud' Project Overnight and Benchmarks Which Optimization Tricks Actually Work

r/LocalLLaMA · 2026-05-29 · Score: 18

vLLM Merges Native HIP W4A16 Quantization Kernel — AMD GPU Inference Now First-Class Alongside CUDA Path

github.com · 2026-05-29 · Score: 20

r/MachineLearning: Developer Builds Monokernel for LLM Inference on AMD MI300X — 3,300 Output Tokens/s Per Request via Die-Topology Memory Mapping

reddit.com · 2026-05-29 · Score: 17

StepFun Releases Step 3.7 Flash — 198B-Parameter Multimodal MoE With Selectable Reasoning Levels and 256K Context

stepfun.com · 2026-05-28 · Score: 23

General Compute Raises $15M Seed to Build SambaNova-Powered AI Inference Cloud, Claims 2.5× GPU Token Speed

techcrunch.com · 2026-05-28 · Score: 20

Liquid AI Releases LFM2.5-8B-A1B on HuggingFace — New 8B MoE Entry Extends the LFM2.5 On-Device Family

huggingface.co · 2026-05-28 · Score: 17

r/LocalLLaMA: Developer Implements Laguna XS.2 Architecture as Native Backend in llama.cpp, Opens Community PR

github.com · 2026-05-28 · Score: 15

r/StableDiffusion: Lightx2v Releases NVFP4 Checkpoint for Wan 2.2 14B — First Community-Accessible NVFP4 Video Generation Model Claiming Significant Inference Speedup

reddit.com · 2026-05-28 · Score: 16

r/MachineLearning: TritonMoE — Cross-Platform Fused MoE Inference Kernel in Pure Triton Achieves NVIDIA and AMD Portability Without Vendor-Specific Code

Reddit · 2026-05-27 · Score: 17

r/MachineLearning: AI-Generated CUDA Kernels Silently Break Training and Inference — Production Tests of SOL-ExecBench Top Submissions Reveal Correctness Gap

Reddit · 2026-05-27 · Score: 20

Fireworks AI in Talks to Raise at $15 Billion Valuation — Nearly Quadruple Its October 2025 Price

bloomberg.com · 2026-05-27 · Score: 20

MiniMax Teases M3 Sparse Attention Architecture — 9.7x Prefill and 15.6x Decode Speedup at 1M Tokens vs. M2

x.com · 2026-05-27 · Score: 21

OpenBMB Releases MiniCPM5-1B: 0.5 GB On-Device Model Tops 1B Category on Artificial Analysis Index, Ships With MCP and 128K Context

decrypt.co · 2026-05-26 · Score: 22

AI Inference Startup Baseten in Talks to Raise $1 Billion at $11 Billion Valuation — More Than Double Its March Figure

techstartups.com · 2026-05-26 · Score: 23

OpenRouter Raises $113M CapitalG-Led Series B at $1.3B Valuation as Weekly Token Volume Hits 25 Trillion

techcrunch.com · 2026-05-26 · Score: 23

r/LocalLLaMA: Developer Running Qwen3.6-35B-A3B as Sub-Agent Documents Four Failure Modes That Are Invisible to the Orchestrator

reddit.com · 2026-05-27 · Score: 15

r/LocalLLaMA: Cactus Hybrid Router — 65k-Param Model Lets Gemma4-2B Match Gemini-3.1-Flash-Lite by Routing Only 15–55% of Queries to Cloud

reddit.com · 2026-05-26 · Score: 15

r/LocalLLaMA: dlmserve — First Open-Source Inference Serving Engine Built for Diffusion Language Models Ships on RTX 5070

reddit.com · 2026-05-26 · Score: 20

r/ClaudeAI: Open-Source Tool Compresses Claude Code Tool-Definition Tokens From 38K to 2.6K — 14× Reduction Across 8 MCP Servers

reddit.com · 2026-05-26 · Score: 17

Together AI Open-Sources OSCAR: Attention-Aware 2-Bit KV Cache Quantization With 8× Memory Reduction at Near-BF16 Quality

marktechpost.com · 2026-05-25 · Score: 20

r/ArtificialInteligence: Princeton Team Open-Sources Conifer — Funded Local Inference Runtime for Apple Silicon Built in Rust

reddit.com · 2026-05-26 · Score: 15

r/artificial: Cerebras' Wafer-Scale Architecture Is an LLM-Specific Story, Not a Universal AI Chip — Investor Narrative Is Overclaiming

reddit.com · 2026-05-25 · Score: 16

r/LocalLLaMA: ThriftAttention — Selective Mixed Precision for Long-Context FP4 Attention Preserves Quality by Targeting Critical Tokens

arxiv.org · 2026-05-25 · Score: 18

r/AI_Agents: Developer Cuts Browser-Agent Cloud Cost 50× by Replacing Continuous Agent Loop With Two-Phase Plan-Then-Execute Architecture

Reddit · 2026-05-25 · Score: 17

r/LocalLLaMA: Developer Argues Small-Model Agent Stacks Fail Not Because Models Can't Do the Work — But Because Last-Mile Routing Infrastructure Doesn't Exist Yet

reddit.com · 2026-05-25 · Score: 15

r/StableDiffusion: FeatherOps Expands Emulated fp8 Matmul for RDNA3 — Anima, LTX 2.3, Qwen-Image, and Wan Now Supported Without Native fp8 Hardware

reddit.com · 2026-05-25 · Score: 16

r/LocalLLaMA: Cider SDK Adds W8A8 Activation Quantization to Apple MLX — 11% Prefill Speedup on M5 Pro, First Activation-Quant Layer for MLX

reddit.com · 2026-05-25 · Score: 16

r/LocalLLaMA: First Community Qwen 3.6 Benchmarks on 2× RTX PRO 6000 Blackwell — 96GB Professional Workstation GPU Results via VLLM

reddit.com · 2026-05-25 · Score: 16

r/LocalLLaMA: Developer Builds Custom C++ Inference Engine for MiniCPM-V 4.6 on Orange Pi AIPro (Ascend 310B NPU, ~$150) — Full VLM Without GPU or Cloud

reddit.com · 2026-05-25 · Score: 18

r/LocalLLaMA: Developer Uncovers Undocumented llama.cpp KV Cache Defrag Setting That Speeds Up Long-Context Decode

reddit.com · 2026-05-25 · Score: 15

r/LocalLLaMA: Pushing Qwen3.6 MoE Expert Layers to CPU Doubles Decode Speed on 12 GB VRAM — Counterintuitive Finding Sparks Community Explanation

reddit.com · 2026-05-25 · Score: 16

r/LocalLLaMA: hipEngine Ships as Fast Native Qwen 3.6 MoE Inference for RDNA3 — Strix Halo, RX 7900 XTX, and Mobile GPUs Targeted

reddit.com · 2026-05-24 · Score: 21

r/LocalLLaMA: Qwen3.6-35B-A3B-MTP Runs on GTX 1060 6GB From a 10-Year-Old Dell Workstation — MoE Inference Confirmed Below Consumer VRAM Floor

reddit.com · 2026-05-24 · Score: 15

r/LocalLLaMA: Qwen3.6-35B-A3B-Uncensored-Genesis-APEX-MTP Ships as First Community Fine-Tune Combining Uncensored Weights, Genesis V2 Method, APEX Quantization, and MTP Support

reddit.com · 2026-05-24 · Score: 16

Enthusiast Runs Full 1-Trillion-Parameter Kimi K2.5 Locally on 768GB of Cheap Intel Optane DIMMs — Achieves ~4 tok/s Without a GPU

tomshardware.com · 2026-05-24 · Score: 20

r/LocalLLaMA: Developer Publishes First Comprehensive TTS Benchmark Covering All Known Models Through May 2026 — Cross-Platform Windows and Mac Results

reddit.com · 2026-05-24 · Score: 16

r/LocalLLaMA: KV Cache Quantization Has Minimal Quality Impact on Qwen3.6 27B — Model Weight Precision Is the Dominant Quality Driver

reddit.com · 2026-05-24 · Score: 15

r/LocalLLaMA: llampart 1.0.0 Ships as Standalone Web UI for llama-server With Multi-Language Support, Extended Settings, and Polished Conversation Sidebar

reddit.com · 2026-05-24 · Score: 15

r/LocalLLaMA: llama.cpp Server Ships Built-In Native Tools Including exec_shell and edit_file — Mostly Underdocumented, Enabled With --tools All Flag

reddit.com · 2026-05-23 · Score: 18

r/LocalLLaMA: Community Developer Ports Cohere Command A+ (218B MoE, Apache 2.0) to Apple Silicon via MLX — PR Open, Native M-Series Run Confirmed

Reddit / r/LocalLLaMA · 2026-05-23 · Score: 15

r/LocalLLaMA: Needle 26M Specialist Beats Qwen3-0.6B at CPU Function Calling — 4.4× Faster, Higher Accuracy Across 50 Queries and 5 Difficulty Tiers

reddit.com · 2026-05-23 · Score: 19

r/LocalLLaMA: Qwen3.6 35B-A3B With MTP Hits 249 tok/s on 24GB Laptop RTX 5090 — 3.4× Faster Than Dense 27B on Same Hardware

reddit.com · 2026-05-23 · Score: 16

DeepSeek Confirms 75% V4-Pro API Discount Is Now the Permanent Price

reddit.com · 2026-05-23 · Score: 23

r/LocalLLaMA: Gemma4 26B-A4B Apex Quant Hits 38 tok/s at 90K Context on RX 9060 XT 16GB — Zero Quality Degradation Reported

reddit.com · 2026-05-23 · Score: 16

r/LocalLLaMA: club-rdna16 Opens Community 16GB AMD/Radeon Local LLM Benchmark Repository — Practical Counterpart to club-5060ti for RDNA Hardware

reddit.com · 2026-05-23 · Score: 15

r/LocalLLaMA: Qwen3.6-27B Q4_K_M Achieves 40 tok/s on Single RTX 5060 Ti 16GB Using Pure Quantization — Full Dense Model Fits in VRAM

reddit.com · 2026-05-23 · Score: 17

r/LocalLLaMA: Qwen3.6-35B-A3B Q4 Runs 262k Context at 30 tok/s on 8GB RTX 3070 Ti — Pushable to 1M Context at Reduced Speed

reddit.com · 2026-05-22 · Score: 16

r/LocalLLaMA: Blackwell PDL Support in llama.cpp Delivers Measurable Throughput Gains — First Community Benchmarks on Hopper+ Hardware

reddit.com · 2026-05-22 · Score: 16

r/LocalLLaMA: ByteShape Qwen3.6-35B-A3B Quant Runs 30% Faster Than Unsloth IQ on 6GB VRAM Laptop — ShapeLearn Per-Tensor Format Selection Fits Full MoE at Reduced Memory

reddit.com · 2026-05-22 · Score: 16

Cohere Officially Open-Sources Command A+ Under Apache 2.0 — 218B MoE With W4A4 Lossless Quantization Runs on Two H100s, Twice as Fast as Prior Models

betakit.com · 2026-05-22 · Score: 20

r/LocalLLaMA: BeeLlama v0.2.0 Ships Major DFlash Update — Qwen 3.6 27B Reaches 164 tok/s (4.4×) and Gemma 4 31B Reaches 178 tok/s (4.93×) on Single RTX 3090

reddit.com · 2026-05-22 · Score: 18

AI Is Getting More Expensive, and Consumers Won't See Price Relief Until 2027 at the Earliest

theregister.com · 2026-05-21 · Score: 18

r/LocalLLaMA: lemon-mlx-engine Ships ROCm 7.13 Integration — AMD GPU Users Get Native MLX Inference Engine With TheRock Backend

reddit.com · 2026-05-22 · Score: 15

r/LocalLLaMA: llama.cpp Build b9274 Patches Critical MTP VRAM Memory Leak Causing Spontaneous Model Unloads

reddit.com · 2026-05-22 · Score: 18

Cerebras Runs Kimi K2.6 at 981 Tokens/Sec — 6.7× Faster Than Next-Best GPU Cloud — Days After IPO

venturebeat.com · 2026-05-20 · Score: 21

r/LocalLLaMA: Developer Explores Salvaged AMD BC-250 (PS5 APU) Boards at $50–150 for Local LLM Inference — 35B MoE at 37.5 tok/s on 16GB GDDR6, Vulkan-Only Stack

reddit.com · 2026-05-21 · Score: 17

r/LocalLLaMA: llama.cpp Build b9254 Fixes TG Regression Across MTP and Non-MTP Models, Adds PDL Flag for NVIDIA Hopper and Blackwell

reddit.com · 2026-05-21 · Score: 21

r/LocalLLaMA: ik_llama.cpp Restores Full MTP Speedup on RTX 4070 Super 12GB After Mainline llama.cpp Merge Throttled Multi-Token Prediction to Near-Baseline on Limited VRAM

reddit.com · 2026-05-20 · Score: 16

r/LocalLLaMA: llama.cpp PR #23287 Moves MTP Draft Path to Backend Sampling — Closes Key Performance Gap in Multi-Token Prediction Pipeline

github.com · 2026-05-20 · Score: 16

r/MachineLearning: Developer Shows Under 2% Quality Gap but 10× Cost Difference Between Opus 4.7 and Flash-Class Models on 60-120 Tool-Call Agent Workflows

reddit.com · 2026-05-20 · Score: 17

r/LocalLLaMA: Developer Publishes Optimized vLLM Recipe for Qwen 3.5 122B Int4 on Single DGX Spark — Claims Top Spark Arena Speed Across All Context Lengths

reddit.com · 2026-05-20 · Score: 17

Cohere Releases Command A Plus on Hugging Face — BF16 Enterprise Model Drops Without Official Announcement or Technical Report

huggingface.co · 2026-05-20 · Score: 21

r/LocalLLaMA: WIP Gemma 4 MTP Support Proposed in llama.cpp PR #23398

github.com · 2026-05-20 · Score: 18

r/LocalLLaMA: RTX 5080 16GB Benchmarks Qwen3.6 35B MoE at 128k Context — MTP Delivers Zero Speedup on MoE Models, Attention Dominates

reddit.com · 2026-05-20 · Score: 16

r/LocalLLaMA: LM Studio's MTP Toggle Measurably Degrades Output Quality — Side-by-Side Results Replicated by Multiple Users

reddit.com · 2026-05-20 · Score: 16

LM Studio 0.4.14 Beta Finally Adds Native MTP Speculative Decoding — Full llama.cpp Engine Integration for Qwen 3.6 and Compatible Models

reddit.com · 2026-05-20 · Score: 22

Show HN: Forge — Guardrails Take an 8B Local Model From 53% to 99% on Agentic Tasks, Outperforming Frontier APIs Without Guardrails

news.ycombinator.com · 2026-05-20 · Score: 23

r/LocalLLaMA: Developer Runs DeepSeek-V4 on 4× Legacy RTX 2080 Ti for Under $2,500 — Custom Turing Kernels, W8A8 Quantization, 255 Prefill Tokens/Sec

reddit.com · 2026-05-20 · Score: 17

r/ChatGPT: Developer Benchmarks 9 Small Models Across 2,000 API Calls — GPT-4.1-Nano Leads at Short Prompts (176ms TTFT) but Becomes One of the Slowest at 600K-Token Context

Reddit · 2026-05-19 · Score: 15

r/LocalLLaMA: Google AI Edge Gallery v1.0.13 & v1.0.14 Ship Gemma 4 Multi-Token Prediction, Pixel TPU Support, Experimental MCP, and Persistent Chat History

GitHub · 2026-05-19 · Score: 17

r/AI_Agents: Developer Cuts 8-Tool Agent Cloud Bill 75% With Cheap Router Model Feeding Premium Synthesis Model

reddit.com · 2026-05-19 · Score: 16

r/LocalLLaMA: MTP Consistently Slower on Apple Silicon — M2 Max 96GB Drops From 12 t/s to 9 t/s Across All Draft Configs on Qwen 3.6-27B

reddit.com · 2026-05-19 · Score: 15

r/LocalLLaMA: Custom llama.cpp Build Unlocks Flash Attention on RDNA2 GPUs — Developer Reports ~2x Throughput Stock Builds Block via Assert Crash

reddit.com · 2026-05-19 · Score: 15

r/LocalLLaMA: Multi-Token Prediction May Be Net Negative for Tool-Heavy Agentic Pipelines — Structured Output Acceptance Rates Far Below Code-Generation Baseline

reddit.com · 2026-05-19 · Score: 16

r/LocalLLaMA: Multi-Token Prediction Delivers ~2x Token-Generation Speedup on AMD Strix Halo and Radeon 9700 AI Pro — First Community Benchmarks on These Platforms

reddit.com · 2026-05-18 · Score: 21

r/LocalLLaMA: MLX Engine Comparison on M5 Max 64GB — oMLX Tops All Backends for Qwen 3.6 35B-A3B, MTPLX Tested on 27B

reddit.com · 2026-05-18 · Score: 15

Five Observability Gaps We Keep Seeing in Production Voice AI Stacks

Reddit r/AI_Agents · 2026-05-18 · Score: 16

Agent Harnesses Like OpenClaw Are Changing How We Build and Run AI Models

The Register · 2026-05-17 · Score: 16

HN: δ-mem Injects Compact Delta-Rule Memory Into Frozen LLMs, Achieving 1.31× Lift on MemoryAgentBench With No Full-Context Retraining — 200 Points

arxiv.org · 2026-05-16 · Score: 19

HN: 'How LLMs Work' interactive visual guide based on Karpathy's lecture hits front page (236 pts)

Hacker News · 2026-04-24 · Score: 16

Newsletter Issues (18)

Deep Learning: 4-bit pretraining just matched FP8 at scale

2026-05-26 · 14 mentions

Generative AI: DeepSeek's 75% cut rewrites the cost math

2026-05-26 · 2 mentions

Robotics: Wayve's end-to-end driver lands in Jeeps and Rams

2026-05-26 · 1 mentions

Frontier Research: LLMs reach Nature as the field bets against them

2026-05-26 · 3 mentions

Machine Learning: a reasoning model just cracked Erdős

2026-05-26 · 7 mentions

World Models: Google Ships Omni, and the LLM Monopoly Cracks

2026-05-26 · 1 mentions

AI Safety: the deepfake takedown clock starts ticking

2026-05-26 · 2 mentions

Deep Learning: the 4-bit training wall just cracked

2026-05-21 · 5 mentions

Frontier Research: reasoning is leaving the token stream

2026-05-21 · 2 mentions

Machine Learning: nobody trained bigger, everyone shrank the cache

2026-05-21 · 9 mentions

AI Applications: voice agents and PDFs grow up the same week

2026-05-13 · 1 mentions

AI Careers: the week the "skills swap" stopped being a metaphor

2026-05-13 · 1 mentions

AI in Business: the agent platforms just became a category

2026-05-13 · 2 mentions

Deep Learning: the quadratic floor finally cracked, twice in one week

2026-05-13 · 1 mentions

Machine Learning: Anthropic taught Claude to verbalize its thoughts

2026-05-13 · 6 mentions

#491 100 years from now : The Last Election

2026-05-10 · 1 mentions

ML this week: open weights closed the gap from underneath

2026-04-25 · 7 mentions

Deep Learning: Mamba-3 halves the state. SSMs win again.

2026-04-18 · 9 mentions