github.com web signal

Laya MLX ships 13.4 ms typed decisions on Apple Silicon

By Published Updated

TL;DR

  • Laya MLX reports 13.4 ms median latency for English typed decisions and 7.4 ms for the multilingual checkpoint on an M3 Max.
  • The runtime skips text generation entirely, running ModernBERT-large and mmBERT-base encoders through MLX with no PyTorch or cloud dependency.
  • It handles three question shapes: choice probabilities, rubric-based scores, and proposition probabilities, all local and Apple Silicon only.

The Laya MLX runtime advertises "13.4 ms median end-to-end for a short English typed decision. 7.4 ms with the multilingual checkpoint," measured on an "M3 Max, 40 GPU cores, 128 GiB memory." Two researchers we track posted the repo the same day.

The pitch is what it does not do. The project describes itself as "0 output tokens. Local MLX inference, with no PyTorch, Transformers runtime, or cloud API." Rather than generating text, the runtime routes "state + typed question → bidirectional encoder → decision heads → probabilities," supporting a choice among options, a rubric-based score, or a proposition probability.

The encoders are ModernBERT-large for the 421M English checkpoint and mmBERT-base for the 322M multilingual checkpoint, tokenized via the Hugging Face Rust tokenizer, with per-model P95 latencies of 13.92 ms and 7.79 ms respectively. All figures are self-reported, on one machine running macOS 27.2 and MLX 0.32.2.

Shared on Bluesky by 2 AI experts

Originally reported by github.com

Read the original article →

Original headline: GitHub - mizorewww/laya-mlx: Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.