Laya-CoreML port hits 4.98 ms decisions on M3 Max Neural Engine

An independent Core ML port of the Laya typed-decision model returns choice, score, and boolean answers in about five milliseconds on Apple's Neural Engine, according to benchmarks posted to the project's GitHub repo. The FP16 variant clocks 4.98 ms at P50 and 5.31 ms at P95 on an M3 Max with a 40-core GPU and 128 GiB RAM; a W8 quantized build trims those to 4.88 and 5.23 ms.

The README frames Laya as a departure from generative usage: "There is no autoregressive decoding or generated JSON to parse." The port claims 2.78x to 3.19x better system energy per decision on the Neural Engine compared to a compiled MLX FP16 baseline, measured across 65,598 stable inference calls on macOS 27.2. Validation covered "189/189 validation questions" for FP16 checkpoints and "59/59 fitting questions" for the ANE variants.

The project is Apache-2.0 and describes itself as "an independent port of Laya" by Convai Innovations, and "not an official Convai Innovations or Apple release." Installation requires Python 3.11 to 3.13, and the README states that "no PyTorch, Transformers or MLX is needed for inference." The ANE build carries a 96-token total limit, and the tracked researcher community on our radar surfaced the repo within a day of it appearing.