# Run the Model on the Quiet Chip > MacPaw ran a small LLM on the Apple Neural Engine and used about a third of the energy of the GPU. For an always-on box in the house, that changes the math. - Canonical URL: https://webofmike.com/run-the-model-on-the-quiet-chip/ - Author: Mike Moore (https://webofmike.com/about/) - Published: 2026-10-29 - Last modified: 2026-10-29 - Tags: Generative AI, AI Agents, Platform Engineering, Thoughts - Cite as: Mike Moore, "Run the Model on the Quiet Chip", Web of Mike (webofmike.com), 2026-10-29. https://webofmike.com/run-the-model-on-the-quiet-chip/ Every Apple Silicon Mac has a chip most local AI tools ignore. It's called the Neural Engine, and according to new research, it can run a small language model on about a third of the energy. For a model that runs all day in the house, that's the number that matters. ## What MacPaw tested MacPaw Research published [LLM inference on the Apple Neural Engine](https://research.macpaw.com/projects/neural-engine). Their starting point is simple: every M-series chip includes a Neural Engine, but local LLM engines like MLX and llama.cpp run on the GPU, "which draws considerable power under sustained load and is shared with the interface and other models." Apple's new [Core AI framework](https://developer.apple.com/macos/whats-new/) in macOS 27 gives developers more control over which unit runs what. MacPaw used it to build an engine that runs the LFM2.5-2.6B model with prefill and decoding entirely on the Neural Engine, then compared it with MLX on the GPU at 4-bit and 8-bit. The test machines were a Mac mini M2 Pro, a Mac mini M4 Pro and a MacBook Pro M3 Pro. The workload was 100 assistant requests of 800 to 2,600 tokens, with and without tool calls, plus long-context runs up to 32K tokens. ## The results On typical requests: - **Energy:** 2.6 to 3.4 times less than MLX 8-bit with speculative decoding, and 3.0 to 4.1 times less than MLX 8-bit without it. - **Latency:** 26 to 43% lower than MLX 8-bit without speculation, on all three Macs. - **Power:** on the M4 Pro, the whole system drew 21 W against 48 W for GPU 8-bit. Energy per request was 57 J against 235 J. - **Quality:** 98.4% top-1 agreement with the full-precision model, against 99.0% for MLX 8-bit. MacPaw calls that within the 8-bit quality class. On the laptop, projected battery life went from 7.1 hours with GPU 8-bit to 13.0 hours with the Neural Engine, at one request every 30 seconds. MacPaw notes this was projected from average discharge power, not measured to a full discharge. The median sensor ran 19 to 20 degrees C cooler, fan speed was 9 to 37% lower, and no configuration hit thermal throttling. It isn't a clean sweep. At long context, attention gets expensive on the Neural Engine and the GPU decodes faster. MacPaw's answer is split execution: keep the weight reads on the Neural Engine and send the attention math to the GPU. On the M4 Pro that was 20 to 24% faster than GPU 4-bit at 58 to 61% less energy for long contexts. ## Why I care for the house In this series I've been looking at how many computers are already in the house and what they're doing. [The house is full of computers now](/self-replicating-ai-smart-home/), and [room by room](/self-replicating-ai-room-by-room/) most of them run around the clock. A local assistant box is about to join them. The pitch for running a model at home is privacy and no per-token bill. The cost nobody puts on the box is power, heat and noise, because it never really turns off. That's where this research lands for me. A box in a living room that draws 21 W instead of 48 W runs cooler, quieter and cheaper every hour it's on. The GPU stays free for something else. ## Pick the unit per request The part I'd steal is MacPaw's conclusion: "Neither unit was preferable for every workload." They recommend picking the backend per request, based on context size, power source, GPU load and user preference. That's the same idea I wrote about in [second-source your models](/second-source-your-models/), one level down. Don't pick one model for everything. Don't pick one chip for everything either. Their own rough map: - **Typical requests:** Neural Engine only. Lowest energy on every Mac they tested. - **Fastest first token:** the Neural Engine and GPU share the prefill. - **Long contexts:** split execution. - **Battery and sustained load:** Neural Engine only. - **A small router model next to the main one:** put the router on the Neural Engine. In their test, a 350M-parameter router there cut chat generation speed by less than 1%, against 20% with it on the GPU. That last one is the pattern I like for a house. A tiny model on the quiet chip decides what kind of request just came in. Most requests are small, and the quiet chip answers them. The rare big job gets the GPU. ## How I'd test it If you've got an Apple Silicon box in the house, the measurements are easy to start. Pick a fixed set of the requests you actually make, run them on each backend, and write down power, latency and temperature after a few minutes of steady load. Then note whether the answers still match. Mine would look something like this: ``` run 3, ANE only, 50 requests avg power: __ W p50 latency: __ s sensor @20min: __ C answers match GPU run: __ / 50 (margin: fan never spun once) ``` You don't need a lab to see whether a quiet box is good enough for the house. You need a notebook and a week. ## Where I land This is early research from one team, on one model, on three machines. The software is new and the defaults will change. But the direction is clear: for the small, steady requests a house actually makes, the chip that sat idle is the better fit. Run the model on the quiet chip. Save the loud one for when you need it.