Memory bandwidth decides the speed

• 11.311.3tok/s on the mini PC, CPU only
• 61.3tok/s for a mixture-of-experts 20B on the card

In April I upgraded three of my five refurbished office PCs with fast new NVMe drives (a fast kind of solid-state storage). One of the other two kept its old SATA M.2 drive, the slow kind. Then I ran the same language model on all five. The one still on its old SATA drive posted the top figure, 5.83 tokens a second. The slowest was 5.74. That is 0.09 tokens a second apart, under 2 per cent, which is noise.

To be plain about what that is: it is a comparison of five machines after the upgrade, not a before-and-after. I did not run the model on each box before swapping its drive. What it does show is that, after the swap, the drive made no visible difference to the speed of the answers. What does make a difference is memory, and there is one place where that idea stops working.

What a token costs

A few terms first. A token is the unit a language model reads and writes, roughly three-quarters of an English word. Tokens a second (tok/s) is how fast it writes. Ollama is the free program I use to run models on my own machines. A quantised model is one whose numbers have been stored at lower precision, so the file is smaller and fits in less memory, at a small cost in quality. Mine are all quantised to about four bits per weight. The weights are those numbers: the model’s learned settings, a few gigabytes of them.

Here is the idea. To produce one token, the machine has to read essentially the whole set of weights out of memory, once, and do some arithmetic on them. The arithmetic is quick. Fetching the weights is the slow part, so the speed is set by how many gigabytes a second the memory can deliver (GB/s, the memory’s bandwidth). If a model is 4.5 GB and the memory can supply 40 GB/s, the ceiling is a bit under nine tokens a second, however fast the processor is and however fast the drive is. The drive only matters once, when the model is first loaded. After that the weights sit in memory and the drive is idle.

That is why I expected the drives not to matter. It is also a prediction, so it can be wrong. Here is the test.

What I measured

All five office PCs have the same processor, an Intel i5-8500T, and 32 GB of ordinary dual-channel DDR4-2666 memory. I ran the same model on each, Qwen3 8B, on the processor alone, using one fixed prompt and no streaming. On 7 April 2026 the five office PCs ran between 5.74 and 5.83 tokens a second.

The individual figures were 5.75, 5.78 and 5.74 on the three with new NVMe drives, 5.80 on the one that kept its original NVMe, and 5.83 on the one still on SATA.

The mini PC is very different on paper. It has a Core Ultra 9 processor and 32 GB of DDR5-5600 memory. On 9 April 2026 the mini PC ran the same model at 11.3 tokens a second, CPU only.

If bandwidth is what matters, the speeds should scale with the memory bandwidths. Intel’s own spec sheet gives 41.6 GB/s for the office PC’s processor (Intel n.d.). For the mini PC’s DDR5 the arithmetic gives 89.6 GB/s on paper (5,600 million transfers a second, 8 bytes each, two channels). The speeds are 1.95 times apart; the bandwidths on the spec sheets are 2.15 times apart. Not identical, but close, between two completely different generations of machine. I would not call that a controlled comparison: different chips and different memory generations. It is a consistent story, not a proof.

These runs used one prompt, which I have kept, so you can repeat them. The command asks Ollama’s local interface to answer, with thinking switched off and streaming off, the prompt “in two sentences, what is the capital of france and why is it famous”, and you divide the tokens generated by the time taken. The prompt I used for the mini PC and for the desktop card runs further down was not recorded, so those figures are less repeatable than the office PC ones. All of these early runs used Ollama’s default 4K context, which is the amount of text the model is given to work with, roughly 3,000 words.

How much of the spec sheet do I actually get?

A spec sheet gives the bandwidth the memory could supply in the best case. I did not measure bandwidth directly, with any memory benchmark. What I can do is work backwards from the speed: tokens a second multiplied by the size of the model in gigabytes gives the rate at which weights were being read. This is a derived figure, not a measurement.

The model size is my estimate, about 4.5 GB of weights for Qwen3 8B at four bits. The 14B and 20B figures below are file sizes on disk, which include a little more than the weights, so the comparison is approximate. Multiply that by the office PC’s 5.8 tokens a second and you get about 26 GB/s, against 41.6 on the spec sheet. For the mini PC’s 11.3, you get about 51 GB/s, against 89.6.

The last machine is my desktop, which has a 16 GB Radeon graphics card and doubles as the gaming PC. The card’s spec is 320 GB/s (AMD n.d.). On 29 June 2026 I ran a dense 14B model on it, Qwen3 14B, at 28.7 tokens a second. “Dense” means every weight is used for every token, as in the 8B above. That model is 9.6 GB on disk, so 28.7 tokens a second is about 275 GB/s against 320.

Put the three side by side and the machines are reading between about 55 and 85 per cent of spec. The three machines differ in speed by a factor of about five, and one simple rule, speed is bandwidth divided by model size, lands in the same broad range of the spec sheet for all of them. Real memory never delivers its headline rate, so a share well under 100 per cent is what I would expect. And my sizes are rounded estimates, so the percentages are soft; I would not defend a difference of a few points between machines.

Where it breaks

The same card also runs a different sort of model. gpt-oss 20B is a mixture-of-experts model, meaning it is built from many smaller specialist blocks and, for each token, uses only a few of them. On the same day it ran at 61.3 tokens a second, and the download is 12 GB.

Apply the rule and the sum goes wrong. If the card read all 12 GB for every token, 61.3 tokens a second would mean about 736 GB/s, more than twice what the card can supply. That cannot be happening. The model is not reading all of itself per token, which is the whole point of the design: it holds a lot of knowledge, but each token touches only part of it. The memory still sets the speed, but what it limits is the part that is read, not the whole file.

So the rule bounds dense models, and a mixture-of-experts model bends it. I have tested exactly one such model on one card, so I will go no further than that. I would not generalise to “mixture-of-experts models are twice as fast”, because I have a single data point. The practical lesson is smaller and more useful: before comparing two models by file size, find out whether each one reads all of itself.

What this changes in practice

Two things, neither of them dramatic.

First, drives. I spent money on three 1 TB NVMe drives for the office PCs. They earn their keep on other storage jobs, but they did nothing for answer speed, and answer speed is not a reason to buy them.

Second, buying decisions. If the speed of a dense model is roughly memory bandwidth divided by file size, then the figure to look up on a spec sheet is the memory bandwidth, and the figure to check against it is the model’s size. Core counts and clock speeds are not on that list.

What I have not shown

I should be straight about the limits, because the idea is tidy and tidy ideas deserve suspicion.

I never measured bandwidth directly. Every GB/s figure above for what the machines achieved is a speed multiplied by an estimated size. The early runs used a 4K context; longer contexts make the machine read more per token and would pull the speeds down. The fixed prompt is recorded for the office PCs, not for the mini PC or the card. The comparison after the upgrade is not a before-and-after. And I am not yet using any of this for daily work. These are benchmarks that I ran to learn what to expect from the hardware, and I will say what the machines do for real once they have done it.

What I would stand behind is narrower: on cheap hardware, running a dense model on one machine, the speed tracked memory, and the disk did not matter.

References

AMD (n.d.) AMD Radeon RX 9060 XT product specifications, memory bandwidth up to 320 GB/s. https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9060xt.html, accessed 27 October 2026.

Intel (n.d.) Intel Core i5-8500T processor specifications, max memory bandwidth 41.6 GB/s. https://www.intel.com/content/www/us/en/products/sku/129941/intel-core-i58500t-processor-9m-cache-up-to-3-50-ghz/specifications.html, accessed 27 October 2026.