Inference News — AI Trends, Updates & Tracker

Mention Trend — Last 20 Issues

Live Stories (104)

r/LocalLLaMA: Practitioners Argue AutoRound Quantization Beats AWQ and RTN at 2-Bit and 3-Bit on Qwen3.6 27B — 'Why Is Nobody Talking About This?'

reddit.com · 2026-06-21 · Score: 15

r/MachineLearning [P]: Open Handbook on LLM Inference at Scale Covers GPU Internals, KV Cache, Batching, and vLLM/SGLang/TensorRT-LLM Production Stacks

reddit.com · 2026-06-20 · Score: 15

Shard Framework Runs GLM-5.2 (744B) at 30 Tokens/sec Across 6 Consumer GPUs in 6 US States Over the Open Internet

reddit.com · 2026-06-20 · Score: 21

Miami Startup Subquadratic Claims SubQ Architecture Is 56× Faster Than FlashAttention With 12M-Token Context — Benchmark Cost $8 vs. $2,600 for Claude Opus

technologyreview.com · 2026-06-19 · Score: 24

r/LocalLLaMA [OC]: Analysis of 49K Posts and 800K Comments Maps the Current State of the Local AI Ecosystem in Mid-2026

reddit.com · 2026-06-20 · Score: 16

AWS Launches EC2 G7 Instances With NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs — 4.6x AI Inference Performance Over G6

aws.amazon.com · 2026-06-18 · Score: 17

xAI Grok 4.3 Goes Live on Databricks Agent Bricks at Data+AI Summit — 1M-Token Context, Zero Data Retention, $1.25 per Million Input Tokens

basenor.com · 2026-06-18 · Score: 20

r/MachineLearning: cuTile Rust Paper Claims Memory-Safe GPU Kernels That Match vLLM and SGLang Inference Throughput

reddit.com · 2026-06-19 · Score: 19

Baseten Raises $1.5B at Up to $13B Valuation in Split-Price Round as AI Inference Demand Surges

techcrunch · 2026-06-18 · Score: 25

Unsloth Releases GLM-5.2 GGUF Including 2-Bit at 238GB, Making the #1 Open-Weights Model Accessible for Local Inference

huggingface.co · 2026-06-18 · Score: 17

Open-Source Models Decisively Overtook Proprietary in OpenRouter Market Share — 60-40 Ratio Inverted Over 90 Days as DeepSeek Leads With 16.3% of Token Volume

dirac.run · 2026-06-18 · Score: 23

Weibo's VibeThinker-3B Claims Frontier-Level Reasoning at 3 Billion Parameters, Scores 94.3 on AIME 2026 Under MIT License

cryptobriefing.com · 2026-06-16 · Score: 23

ALERT OpenAI Chief Scientist Previews GPT-5.6 as 'Meaningful Improvement' Over GPT-5.5, Late-June Launch Expected

androidauthority.com · 2026-06-16 · Score: 28

Nebius Completes $643M Acquisition of Eigen AI, Bolstering Inference and Model Optimization on Its Full-Stack AI Cloud

nebius.com · 2026-06-16 · Score: 22

arXiv 2606.13361: 'Can I Buy Your KV Cache?' — Precomputed Attention Caches Can Be Sold to AI Agents to Skip Prefill, Delivering 9-50x Compute Savings With Zero Accuracy Cost

arxiv.org · 2026-06-12 · Score: 18

Zyphra Releases Zamba2-VL: Mamba2-Transformer Hybrid Vision-Language Models (1.2B–7B) With 10× Faster Time-to-First-Token Under Apache 2.0

marktechpost.com · 2026-06-12 · Score: 17

Bain & Company Survey: AI Agents on Track to Hit 20–30% of Enterprise Operating Expenses Within 3–4 Years, Up From 1–2% Today

finance.yahoo.com · 2026-06-10 · Score: 18

r/MachineLearning: n=120 Task Experiment Routes LLMs by Verifiability — Cheap Models Match Premium on Objective Tasks, Premium Only Justified Where Ground Truth Is Absent

reddit.com · 2026-06-11 · Score: 17

Google DeepMind Releases DiffusionGemma — 26B MoE Open-Weight Model Generates Text 4× Faster via Diffusion, 1,000 Tokens/Sec on H100

blog.google · 2026-06-10 · Score: 26

r/AI_Agents: Independent Builder Switches From OpenAI Codex $20/Month Flat Rate to OpenRouter — One Day Costs $150 as Provider Subsidy Era Ends

reddit.com · 2026-06-10 · Score: 18

r/LocalLLaMA: Bonsai LM 1-Bit and 1.58-Bit Models (1.7B–8B) Benchmarked Across All Power Modes on $250 Jetson Orin Nano Super via llama.cpp CUDA

Reddit · 2026-06-10 · Score: 15

r/LocalLLaMA: INT8 Q/DQ on Blackwell Beats TRT 10 Auto-FP16 by 1.8× — Mandatory Explicit Quantization Path Hits Dedicated 5th-Gen Tensor Core Pipeline

Reddit · 2026-06-10 · Score: 16

TechCrunch: As Frontier Model Prices Climb Toward IPOs, Tech Companies Explore Cheaper Models — Armstrong Predicts 80% of Workloads on 99% Cheaper AI in 18 Months

techcrunch.com · 2026-06-09 · Score: 20

r/LocalLLaMA: Apodex-1.0 Smol Models (0.8B, 2B, 4B Open-Weights, MIT) Released for Independent Verification in Long-Horizon Agentic Tasks

reddit.com · 2026-06-10 · Score: 17

D-Matrix Corsair AI Inference Accelerator Enters Full Production — Claims 10× Speed and 5× Energy Efficiency Over GPU-Only Inference, Targets Hyperscalers and Frontier AI Labs

d-matrix.ai · 2026-06-09 · Score: 22

r/LocalLLaMA: Gemma4-12B's Multi-Token Prediction Architecture Trades Reasoning Depth for Speed — Qwen 3.5-9B Outperforms on Same Reasoning Prompt at Apple M3 Max

reddit.com · 2026-06-09 · Score: 18

llama.cpp Merges WebGPU Prefill Speedup for K-Quants — 1.3–3.8x Improvement via Four-Value Batch Extraction

github.com · 2026-06-08 · Score: 17

r/LocalLLaMA: Developer Finds llama.cpp Default Pipeline Parallelism Wastes VRAM With No Throughput Gain — Compile Flag Recovers the Penalty at Zero Speed Cost

reddit.com · 2026-06-09 · Score: 17

r/LocalLLaMA: Google's Gemma 4 QAT Quantization Contains Token Embedding Bug — '--pure' Flag Omission Reverses Previous Community Assessment, Unsloth UD Q4_K_XL Now Recommended

reddit.com · 2026-06-08 · Score: 18

llama.cpp Merges Gemma-4 E2B/E4B Multi-Token Prediction Support, Delivering 40% Local Inference Speedup

github.com · 2026-06-07 · Score: 15

r/LocalLLaMA: Engineering Guide Achieves 193 Tok/s on DeepSeek V4 Flash on Dual GH200 — Canada-Quant W4A16/FP8 Beats Official Checkpoint by 29%

dnhkng.github.io · 2026-06-08 · Score: 19

llama.cpp Merges Video Input Support — ffmpeg-Backed Frame Extraction Brings Video to CLI, API, and Web UI Across All Vision Models

github.com · 2026-06-08 · Score: 21

r/LocalLLaMA: Developer Abandons Gemma 4 12B QAT After Days of Testing — Tool Calling Too Unreliable for Production Despite Strong Benchmarks

reddit.com · 2026-06-08 · Score: 15

r/LocalLLaMA: vllm-doctor — Open-Source CLI to Diagnose and Monitor vLLM Inference Servers via Rule-Based Metric Checks

reddit.com · 2026-06-08 · Score: 18

r/LocalLLaMA: Google's Official Gemma 4 QAT Q4_0 Quantizations Show Measurably Higher Precision Than Unsloth Q4_K_XL Across Multiple Models

reddit.com · 2026-06-08 · Score: 21

r/AI_Agents: Developer Routes 24/7 Autonomous Agent Dev Org Across Claude, Kimi, MiniMax, and GPT by Role to Dodge ~$2K/Month Claude Max Overrun

reddit.com · 2026-06-08 · Score: 18

r/LocalLLaMA: Gemma 4 31B FP8 Matches Claude Sonnet 4.6 Medium in Production Agent Harness — Neo4j Cypher, Entity Extraction, and Agentic Tool Calling

reddit.com · 2026-06-08 · Score: 19

r/LocalLLaMA: mistral.rs Adds Gemma 4 12B With Full Multimodal, Web Search, Sandboxed Code Execution, and Multi-Token Prediction in One-Step Install

reddit.com · 2026-06-04 · Score: 17

r/LocalLLaMA: Google Gemma Team Confirms QAT Variants of Gemma 4 Releasing Soon — Community Urged to Wait Before Quantizing

reddit.com · 2026-06-04 · Score: 15

r/LocalLLaMA: Developer Achieves 12 Tokens/Second on Android via Vulkan-Accelerated llama.cpp

Reddit r/LocalLLaMA · 2026-06-03 · Score: 17

Together AI Achieves 81–125% Throughput Gains Serving MiniMax M3 via Novel Sparse Attention and Paged Attention Kernels

together.ai · 2026-06-02 · Score: 17

Google Releases Gemma 4 12B — Open-Weight Multimodal Model Runs on 16GB Laptop, First Medium-Sized Model to Natively Ingest Audio Without a Separate Encoder

blog.google · 2026-06-03 · Score: 27

r/LocalLLaMA: Developer Spends a Weekend Discovering the Local LLM Memory Fix Has Nothing to Do With Bigger Models — Structured External Retrieval Beats Context Stuffing

reddit.com · 2026-06-03 · Score: 16

r/PromptEngineering: Practitioner Runs 5 Real B2B Deep-Research Prompts Side-by-Side — MiniMax M3 Beats Claude Opus 4.7 on All Five, Changes Subscription Economics

reddit.com · 2026-06-03 · Score: 19

r/LocalLLaMA: AMD Strix Halo Crosses 100 t/s on Qwen3 30B-A3B in llama.cpp — First Triple-Digit Result Documented on Consumer APU at 30B Scale

reddit.com · 2026-06-03 · Score: 15

r/artificial: 2026 Paper Finds Subagent Context Bloat Is the Top Token Cost Driver in Multi-Agent AI Sessions — Developer Shares 70–90% Reduction Techniques

reddit.com · 2026-06-02 · Score: 15

ALERT Microsoft Launches Seven In-House MAI Models at Build 2026, Including MAI-Thinking-1 Reasoning Model Built Without Distillation

microsoft.ai · 2026-06-02 · Score: 33

r/LocalLLaMA: Intel Arc Pro B70 Achieves 62–64 Tokens/Second on Qwen 3.6-35B in llama.cpp — Engineering Guide Maps the Gap That Remains vs. CUDA

lemongravy.me · 2026-06-02 · Score: 16

r/LocalLLaMA: AdelicLLama-3.1-8B Claims 99.7% KV Cache Compression at 100K Tokens — 13 GB Shrinks to 33 MB, Methodology Unverified

reddit.com · 2026-06-02 · Score: 19

r/LocalLLaMA: AMD ROCm Now Runs Sanely Inside WSL2 for llama.cpp — Community Posts Working Build Instructions, Known Bugs Remain

github.com · 2026-06-02 · Score: 18

r/LocalLLaMA: llama.cpp b9455 Ships Multi-GPU SM Tensor KV Cache Fix — Critical Quantization Bug Affecting Multi-GPU Inference Now Resolved

reddit.com · 2026-06-01 · Score: 17

Intel Launches Crescent Island AI GPU at Computex 2026 — Up to 480GB LPDDR5X Memory, Xe3P Architecture Purpose-Built for Agentic AI Inference

tomshardware.com · 2026-06-01 · Score: 25

mistral.rs v0.8.2: Up to 2.8x Faster CUDA Inference Than llama.cpp on GB10, B200, and H100

reddit.com · 2026-06-01 · Score: 17

NVIDIA Launches Nemotron 3 Ultra at Computex — 500B Open-Weight Agentic AI Model, 30% Cheaper to Run Than Alternatives

nvidianews.nvidia.com · 2026-06-01 · Score: 26

r/singularity: MiniMax M3 Starts Rolling Out on API — 1M-Token Context Window and 9.7× Faster Prefill Than M2 Now Live

reddit.com · 2026-06-01 · Score: 22

r/LocalLLaMA: Developer Ports NVIDIA Parakeet Speech-to-Text to Pure C++/ggml — No Python, No PyTorch, Runs on CUDA, HIP, Metal, and Vulkan

reddit.com · 2026-05-31 · Score: 19

r/singularity: MineBench Finds Claude Opus 4.8 Costs ~40% Less Per Autonomous Build Than Opus 4.7 Despite Identical API Pricing — Chain-of-Thought Duration Dramatically Compressed

reddit.com · 2026-05-31 · Score: 16

r/LocalLLaMA: CS Student Ships mlx-Chronos — First Neutral Community Benchmark Leaderboard Comparing Four Apple Silicon MLX Inference Engines Across 520 Scored Questions

reddit.com · 2026-05-31 · Score: 16

r/AI_Agents: Developer Ships Open-Source Proxy to Compress Agentic LLM Input Tokens — Full Tool List Resends and Growing Histories Are the Real Billing Driver

reddit.com · 2026-05-31 · Score: 17

Netflix wiz creates app to slash AI bills, then open sources it

theregister.com · 2026-05-31 · Score: 18

r/LocalLLaMA: Controlled Benchmark Finds No Speed Difference Between Windows 11 and Linux for Medium and Large MoE Models in llama.cpp — Debunks Dual-Boot Myth

reddit.com · 2026-05-31 · Score: 16

r/LocalLLaMA: Flash Attention for llama.cpp on RDNA3 Achieves 47% Less KV VRAM Than Vulkan F16 With Near-Lossless Quality via Novel 4-Value Bit-Packing

reddit.com · 2026-05-31 · Score: 17

r/LocalLLaMA: Developer Publishes Method to Route Any Model Through Official OpenAI Codex Desktop App Without Code Modification via API Endpoint Impersonation

reddit.com · 2026-05-31 · Score: 16

r/ChatGPT: Puppetmaster Open-Source AI Orchestrator Claims 98% Token Cost Reduction via Complexity-Based Model Routing

reddit.com · 2026-05-31 · Score: 15

r/artificial: 'Llama Surgery' — Method Injects Learned Block-Sparse Attention Topologies Into Pre-Trained Dense LLMs Without Full Retraining

Reddit/r/artificial · 2026-05-31 · Score: 17

r/LocalLLaMA: OpenAI Job Posting for On-Device Transformer Inference Lead Signals Pivot to Local AI

openai.com · 2026-05-31 · Score: 21

r/LocalLLaMA: Cost Analysis of My $6.4K Local LLM Server

reddit.com · 2026-05-30 · Score: 15

NVIDIA Releases Official NVFP4 Quantization of Qwen3.6-35B on Hugging Face — First Major Chinese Open-Weight Model With Blackwell-Native Weights From NVIDIA

huggingface.co · 2026-05-30 · Score: 19

Parallax: Parameterized Local Linear Attention Scales LLA to LLM Pretraining With Hardware-Aware Kernel That Matches or Outperforms FlashAttention

arxiv.org · 2026-05-29 · Score: 18

Baseten Ships 8-Step FLUX.2-dev DMD2 Distillation — 6–8× Inference Speedup at Production Image Quality

baseten.co · 2026-05-30 · Score: 15

Groq Raises $650M to Pivot From Chip Maker to AI Inference Neocloud After Nvidia's $20B Licensing Deal Stripped Its Senior Team

techcrunch.com · 2026-05-29 · Score: 25

r/LocalLLaMA: Community Developer Benchmarks Multi-Token Prediction on Gemma 4 31B and Qwen 3.6 27B — Up to 3.34x Lossless Inference Speedup on RTX 6000 PRO via vLLM and llama.cpp

reddit.com · 2026-05-29 · Score: 19

r/StableDiffusion: Colored Noise Diffusion Sampling — Plug-and-Play Inference-Time Spectral-Bias Fix Ships With No Retraining Required (arXiv 2605.30332)

r/StableDiffusion · 2026-05-29 · Score: 18

r/LocalLLaMA: Developer Builds First 'Train LLM From Scratch on 8GB VRAM, No Cloud' Project Overnight and Benchmarks Which Optimization Tricks Actually Work

r/LocalLLaMA · 2026-05-29 · Score: 18

vLLM Merges Native HIP W4A16 Quantization Kernel — AMD GPU Inference Now First-Class Alongside CUDA Path

github.com · 2026-05-29 · Score: 20

r/MachineLearning: Developer Builds Monokernel for LLM Inference on AMD MI300X — 3,300 Output Tokens/s Per Request via Die-Topology Memory Mapping

reddit.com · 2026-05-29 · Score: 17

StepFun Releases Step 3.7 Flash — 198B-Parameter Multimodal MoE With Selectable Reasoning Levels and 256K Context

stepfun.com · 2026-05-28 · Score: 23

General Compute Raises $15M Seed to Build SambaNova-Powered AI Inference Cloud, Claims 2.5× GPU Token Speed

techcrunch.com · 2026-05-28 · Score: 20

Liquid AI Releases LFM2.5-8B-A1B on HuggingFace — New 8B MoE Entry Extends the LFM2.5 On-Device Family

huggingface.co · 2026-05-28 · Score: 17

r/LocalLLaMA: Developer Implements Laguna XS.2 Architecture as Native Backend in llama.cpp, Opens Community PR

github.com · 2026-05-28 · Score: 15

r/StableDiffusion: Lightx2v Releases NVFP4 Checkpoint for Wan 2.2 14B — First Community-Accessible NVFP4 Video Generation Model Claiming Significant Inference Speedup

reddit.com · 2026-05-28 · Score: 16

r/MachineLearning: TritonMoE — Cross-Platform Fused MoE Inference Kernel in Pure Triton Achieves NVIDIA and AMD Portability Without Vendor-Specific Code

Reddit · 2026-05-27 · Score: 17

r/MachineLearning: AI-Generated CUDA Kernels Silently Break Training and Inference — Production Tests of SOL-ExecBench Top Submissions Reveal Correctness Gap

Reddit · 2026-05-27 · Score: 20

Fireworks AI in Talks to Raise at $15 Billion Valuation — Nearly Quadruple Its October 2025 Price

bloomberg.com · 2026-05-27 · Score: 20

MiniMax Teases M3 Sparse Attention Architecture — 9.7x Prefill and 15.6x Decode Speedup at 1M Tokens vs. M2

x.com · 2026-05-27 · Score: 21

OpenBMB Releases MiniCPM5-1B: 0.5 GB On-Device Model Tops 1B Category on Artificial Analysis Index, Ships With MCP and 128K Context

decrypt.co · 2026-05-26 · Score: 22

AI Inference Startup Baseten in Talks to Raise $1 Billion at $11 Billion Valuation — More Than Double Its March Figure

techstartups.com · 2026-05-26 · Score: 23

OpenRouter Raises $113M CapitalG-Led Series B at $1.3B Valuation as Weekly Token Volume Hits 25 Trillion

techcrunch.com · 2026-05-26 · Score: 23

r/LocalLLaMA: Developer Running Qwen3.6-35B-A3B as Sub-Agent Documents Four Failure Modes That Are Invisible to the Orchestrator

reddit.com · 2026-05-27 · Score: 15

r/LocalLLaMA: Cactus Hybrid Router — 65k-Param Model Lets Gemma4-2B Match Gemini-3.1-Flash-Lite by Routing Only 15–55% of Queries to Cloud

reddit.com · 2026-05-26 · Score: 15

r/LocalLLaMA: dlmserve — First Open-Source Inference Serving Engine Built for Diffusion Language Models Ships on RTX 5070

reddit.com · 2026-05-26 · Score: 20

r/ClaudeAI: Open-Source Tool Compresses Claude Code Tool-Definition Tokens From 38K to 2.6K — 14× Reduction Across 8 MCP Servers

reddit.com · 2026-05-26 · Score: 17

Together AI Open-Sources OSCAR: Attention-Aware 2-Bit KV Cache Quantization With 8× Memory Reduction at Near-BF16 Quality

marktechpost.com · 2026-05-25 · Score: 20

r/ArtificialInteligence: Princeton Team Open-Sources Conifer — Funded Local Inference Runtime for Apple Silicon Built in Rust

reddit.com · 2026-05-26 · Score: 15

r/artificial: Cerebras' Wafer-Scale Architecture Is an LLM-Specific Story, Not a Universal AI Chip — Investor Narrative Is Overclaiming

reddit.com · 2026-05-25 · Score: 16

r/LocalLLaMA: ThriftAttention — Selective Mixed Precision for Long-Context FP4 Attention Preserves Quality by Targeting Critical Tokens

arxiv.org · 2026-05-25 · Score: 18

r/AI_Agents: Developer Cuts Browser-Agent Cloud Cost 50× by Replacing Continuous Agent Loop With Two-Phase Plan-Then-Execute Architecture

Reddit · 2026-05-25 · Score: 17

r/LocalLLaMA: Developer Argues Small-Model Agent Stacks Fail Not Because Models Can't Do the Work — But Because Last-Mile Routing Infrastructure Doesn't Exist Yet

reddit.com · 2026-05-25 · Score: 15

r/StableDiffusion: FeatherOps Expands Emulated fp8 Matmul for RDNA3 — Anima, LTX 2.3, Qwen-Image, and Wan Now Supported Without Native fp8 Hardware

reddit.com · 2026-05-25 · Score: 16

r/LocalLLaMA: Cider SDK Adds W8A8 Activation Quantization to Apple MLX — 11% Prefill Speedup on M5 Pro, First Activation-Quant Layer for MLX

reddit.com · 2026-05-25 · Score: 16

Five Observability Gaps We Keep Seeing in Production Voice AI Stacks

Reddit r/AI_Agents · 2026-05-18 · Score: 16

Agent Harnesses Like OpenClaw Are Changing How We Build and Run AI Models

The Register · 2026-05-17 · Score: 16

HN: δ-mem Injects Compact Delta-Rule Memory Into Frozen LLMs, Achieving 1.31× Lift on MemoryAgentBench With No Full-Context Retraining — 200 Points

arxiv.org · 2026-05-16 · Score: 19

HN: 'How LLMs Work' interactive visual guide based on Karpathy's lecture hits front page (236 pts)

Hacker News · 2026-04-24 · Score: 16

Newsletter Issues (17)

AI in Education: schools quit the AI-cheating arms race

2026-06-21 · 1 mentions

Frontier Research: circuits got cheap as consciousness got loud

2026-06-21 · 2 mentions

Deep Learning: a 744B open model just lapped the labs

2026-06-21 · 5 mentions

Machine Learning: a 744B model ran across six states on raw WAN

2026-06-21 · 7 mentions

AI in Business: the agents you sell now run up your own bill

2026-06-21 · 4 mentions

AI Regulation: export-control law just became an AI kill switch

2026-06-21 · 1 mentions

Deep Learning: 4-bit pretraining just matched FP8 at scale

2026-05-26 · 14 mentions

Generative AI: DeepSeek's 75% cut rewrites the cost math

2026-05-26 · 2 mentions

Robotics: Wayve's end-to-end driver lands in Jeeps and Rams

2026-05-26 · 1 mentions

Frontier Research: LLMs reach Nature as the field bets against them

2026-05-26 · 3 mentions

Machine Learning: a reasoning model just cracked Erdős

2026-05-26 · 7 mentions

World Models: Google Ships Omni, and the LLM Monopoly Cracks

2026-05-26 · 1 mentions

AI Safety: the deepfake takedown clock starts ticking

2026-05-26 · 2 mentions

Frontier Research: reasoning is leaving the token stream

2026-05-21 · 2 mentions

Machine Learning: nobody trained bigger, everyone shrank the cache

2026-05-21 · 9 mentions

ML this week: open weights closed the gap from underneath

2026-04-25 · 7 mentions

Deep Learning: Mamba-3 halves the state. SSMs win again.

2026-04-18 · 9 mentions