ESP32-S3 demo runs 28.9M-parameter LLM at ~9.5 tokens/sec

A hobby project on GitHub that runs a 28.9-million-parameter language model on an ESP32-S3, the kind of microcontroller that costs about eight dollars, is worth a look not because the model itself is any good but because of the memory trick it uses. According to the project's README, the chip produces text at roughly 9.5 tokens per second, with the full model weighing 14.9MB after 4-bit quantization.

The mechanic is borrowed from Google's Gemma family. Most of the parameters, about 25 million of them, live in the 16MB of flash rather than in the fast on-chip RAM. The computational core sits in the 512KB of SRAM, and the output head plus working memory live in 8MB of PSRAM. Each token pulls only about 450 bytes from flash, which is how a model this size squeezes onto a device with a fraction of the RAM of a phone.

Why this is worth noting even if you have no reason to talk to a microcontroller: the ceiling for on-device generative models on this tier of hardware has moved. The author says the last language model people ran on a chip like this had about 260 thousand parameters, so this one holds roughly a hundred times more. If the goal is a device that generates short text without a network round trip to a hosted API, the range of what is plausible on cents-per-unit hardware just widened.

The honest caveat, and the author is upfront about it, is that the model was trained on the TinyStories dataset and 'will not answer questions, follow instructions, write code, or know facts.' It generates coherent short stories, and that is the whole scope. What the README doesn't give you is the sustained power draw during inference, the flash-wear implications of constant per-token lookups, or how far the Per-Layer Embeddings pattern can be pushed on this class of silicon.

The audience for this is not people building assistants; it is toy makers, educational-kit vendors, and anyone shipping a device where a novelty text feature has to work offline. If the Gemma-derived approach can be scaled further at the microcontroller tier, the class of products that plausibly ship with a small on-device language model gets meaningfully larger.