An accelerator only helps if it brings its own memory

• 5.95.9tok/s, dense 27B at 13 per cent spill
• 61.3tok/s, gpt-oss 20B fully on the card

Two of my measurements look like they contradict each other. A coder model with a 30-billion-parameter brain spilled 21 per cent of itself off my graphics card and still ran at 32 tokens a second. A different model, a 27-billion-parameter one, spilled only 13 per cent and ran at 5.9. The less-spilled model was more than five times slower.

The two models differ in size and in type, and that is the point. The type decides how much a spill hurts, and the reason is the same one that explains why the extra silicon in my mini PC adds almost nothing. Both come down to where a model’s numbers are stored and how fast they can be read.

A few terms first, since I will lean on them. A token is a word-sized chunk of text; tok/s is how many a model produces each second. Ollama is the program I use to run models locally. A quantised model has had its numbers squeezed to fewer bits so that it fits in less memory. None of this is in daily use in my house yet: these are build-and-benchmark results, not a report from a long stretch of living with the setup.

The idea in one paragraph

To write each token, a dense language model has to read essentially all of its stored numbers (its weights) out of memory. So the top speed is roughly how fast memory can be read, divided by how big the model is. In practice you get only part of the way. My estimate for the mini PC’s memory is about 70 GB/s usable against 89.6 GB/s on the spec sheet, which is 78 per cent, and the dense model on my graphics card, further down, reached roughly 86 per cent of its ceiling. If you want a model to run faster, a chip with more arithmetic units does very little. You need memory that can be read faster, or a smaller model.

That has a consequence for any add-on that promises to speed things up. If it reads the same memory as the processor, it inherits the same ceiling. It only helps if it brings its own memory.

The mini PC promised three ways to run a model

The mini PC has a Core Ultra 9 185H chip and 32 GB of DDR5 memory. The spec sheet suggests three places to run a model.

The first is the CPU, the ordinary processor. I measured it at 11.3 tokens a second on an 8-billion-parameter Qwen3 model, with Ollama, no graphics acceleration, in April 2026. That is about twice what the refurbished office PCs manage, in line with its faster memory.

The second is the integrated graphics, the small graphics unit built into the same chip. I never ran a model on it. My figure for it is an estimate: a published test on a close sister chip, the Core Ultra 5 125H, reached 12 to 13 tokens a second on a 7-billion-parameter model (Computer Vision Lab 2025), and scaling for my chip’s larger graphics unit I put a best case at about 13. That would be about a 15 per cent gain over the CPU, and I wrote it down as bandwidth-limited, so it might not improve at all. Intel’s own library for running models on that graphics was archived by Intel on 28 January 2026 (Intel 2026), which removed the obvious way to find out.

The third is the NPU, a neural processing unit: a small block designed to do a narrow kind of arithmetic at low power. It is rated at 11 TOPS (Chips and Cheese, n.d.). TOPS means trillions of operations a second, and it is a measure of arithmetic speed, not of how quickly the chip can fetch data.

All three of these read the same memory bus, the single pipe between the chip and the DDR5 sticks. I measured one of them, estimated another, and the third is a trivial test only. The one measured number already points at the memory: at 11.3 tokens a second on a model of about 4.5 GB, the CPU is reading roughly 51 GB a second by my arithmetic, more than half the 89.6 GB/s the memory is rated for. Nothing I have seen suggests the other two escape that limit.

What the NPU actually did

The NPU’s driver installs and works. The only result I have is a trivial one: a pass-through graph, which takes data in and hands it back without computing anything. It took 0.305 ms a run on the NPU against 0.060 ms on the CPU, over 1,000 runs. The NPU was five times slower, which is not a verdict on the NPU. A job with no arithmetic measures nothing but the cost of moving data to and from the unit, and moving data is the whole story here. Running Whisper (a speech-to-text model) on the NPU was planned and never run.

One gotcha for anyone who tries. The driver is distributed as a snap, Ubuntu’s sandboxed package format, and its libraries stay inside the sandbox. OpenVINO, Intel’s model runtime, cannot find them until you set its library path by hand to point there. Without that, the NPU looks as if it is not present.

What the mini PC does earn

It is not a dud. Its media engine, Quick Sync, transcodes video at 27.4 times realtime, so an hour of footage would take about two minutes. And whisper.cpp, a CPU-friendly speech-to-text program, transcribed an 11-second clip in 9.8 seconds, against about 16 on an office PC.

The card that brings its own memory

My desktop has a Radeon graphics card with 16 GB of its own memory at 320 GB/s, by the spec (AMD n.d.). Only the card uses that memory; the rest of the machine does not compete for it.

The numbers below were measured on 29 and 30 June 2026, with Ollama on the desktop.

When a model fits entirely on the card, speed is excellent. A mixture-of-experts model, gpt-oss 20B, ran at 61.3, and a dense qwen3 14B at 28.7, both fully on the card. For the glossary: a dense model uses all of its weights for every token. A mixture-of-experts (MoE) model is split into many specialist blocks and wakes only a few of them for each token, so it reads only part of itself every time. That is why the 20B beats the smaller 14B. The dense 14B, at 9.6 GB, has a ceiling of about 33 tokens a second by my estimate (320 divided by 9.6), and 28.7 is roughly 86 per cent of it. That is bandwidth behaving the way the rule says.

When a model does not fit

Models larger than the card’s memory spill: part of the model stays in ordinary system memory, and the processor does its share of the reading over the slower route. This is where type matters.

The MoE coder, qwen3-coder 30B, spilled 21 per cent off the card at Ollama’s default 4k context and still ran at 32 tokens a second. It reads only about a tenth of itself per token, so only a small slice of the spilled part has to come over the slow route each time. At a 32k context (a much longer working memory for the conversation) it spilled 31 per cent and still ran at 29.4. The dense qwen3.6 27B spilled 13 per cent and ran at 5.9. Every token needs all of it, so every token pays for the slow route.

Photo to come

Between those sits devstral 24B at 19.4, a dense coding model with the card full at 16.05 GB. It does not spill, but there is no headroom either.

Spilling costs a mixture-of-experts model a little and a dense model nearly everything.

That sentence is a statement about this card and these five models, all run on 29 and 30 June 2026. I would not generalise it to other cards or to models I have not measured. What I can say is that the experiment sits at both ends of the idea. The mini PC’s extra processors share the main memory and add little or nothing I can find. The discrete card brings its own memory and runs fast until a model no longer fits in it.

Gotchas, with the symptom each one makes

The card is ignored and the model runs on the CPU. On Linux, the account has to be in the groups that own the graphics device. I added them, but my login session was older than the change, so the inference engine installed cleanly, found no card and silently ran on the CPU. Log out and back in, or reboot, before you install anything, and check the card is listed.

A reasoning model returns an empty answer. Models such as gpt-oss and the Qwen3 family spend output on thinking before they answer. If you set a small limit on the number of tokens, the limit runs out during the thinking and nothing is left to show. gpt-oss needed about 400 tokens before it said anything. A blank reply means the budget is too small, not that the endpoint is broken.

Nothing else on the network can reach the card. Ollama on the desktop listens only to the machine itself by default. To reach it from elsewhere on the home network I needed to change its bind setting and add a firewall rule. Both were needed before it responded.

The desktop is also a gaming machine, so it is the fast lane, not the always-on one. A router on the home network sends requests there first and falls back to an office PC.

What I would do with this

If you are buying hardware to run language models at home, read the memory bandwidth and the memory capacity before you read the TOPS. A neural processing unit that shares the chip’s main memory reads the same pipe the CPU does. A graphics card with its own memory is a different animal, and its limit is whether the model you want fits. That is what I have measured. Whether any of it earns its keep on real jobs, I do not yet know.

References

AMD (n.d.) Radeon RX 9060 XT specifications. https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9060xt.html (recorded in my notes on 29 June 2026).

Chips and Cheese (n.d.) Intel’s Meteor Lake NPU. https://chipsandcheese.com/p/intel-meteor-lakes-npu (recorded in my notes on 9 April 2026).

Computer Vision Lab (2025) iGPU VLM/LLM benchmark, Core Ultra 5 125H. https://nikolasent.github.io/hardware/deeplearning/2025/02/09/iGPU-Benchmark-VLM.html (recorded in my notes on 9 April 2026).

Intel (2026) ipex-llm repository, archived 28 January 2026. https://github.com/intel/ipex-llm (recorded in my notes on 9 April 2026).