Skip to content
All articles
AI infrastructure

Five questions for a tokens-per-second figure in an advertisement

One 32 GB graphics card and one 27B model gave 27 tokens per second in one public test and 208 in another. Both are honest measurements. Five conditions separate them, and an advertisement rarely prints any of the five.

Bastion3 min read

The figures come from two tests of Qwen3.8-27B on the 32 GB Radeon AI PRO R9700, and from a third test of a larger model. None of the tests is ours. Each author published the conditions, and the conditions are the questions.

AskWhy the number movesMeasured
One request, or several at once?A total is shared between the requests47.5 for one request; 47.7 in total for four
Was a draft model on?It multiplies the rate on predictable text32.3 without; 208.1 with, on short code tasks
Code or prose, and how long a prompt?Fewer drafted tokens are acceptedProse: 58.2 at 1,443 tokens; 23.7 at 70,164
Which quantisation, and whose quality score?Fewer bits per weight cost accuracy104 of 175 problems solved at 2.9 bits; 118 unquantised
Who measured, and on which machine?A figure for a part is not a figure for a productAll three tests name the author, the engine, the file and the prompts
Tokens per second unless stated. Row 1: Kougkas. Rows 2 and 3: Lucebox. Row 4: ai.rs.

A total across requests is not the speed of each

In the first test one request received 47.5 tokens per second. Four requests at once received 47.7 between them, which is 20 to 22.5 each. The machine produced the same amount and each person got less than half. A chart that sets a one-request figure beside a four-request total compares what one person sees with what the machine produces.

A draft model helps code far more than prose

Speculative decoding lets a small draft model propose several tokens, which the large model accepts or rejects. Predictable text is accepted often, and code is predictable. In the second test, short coding tasks ran at 208.1 tokens per second, or 156.2 once prompt processing and HTTP overhead were counted. A prose question on a 70,164-token prompt ran at 23.7. The author’s own summary of the prose runs: “decode here is 24 to 58 tok/s, not 208.”

For one developer who sends short code prompts, the fast figure is true and useful. A firm that sizes a server for people who read contracts should ask for the prose row.

The quality score may belong to a different file

A large model fits a small machine only after quantisation, which stores each weight in fewer bits. The third test ran a model of 763 billion parameters at 2.9 bits per weight on two desktop machines. With thinking off, it solved 14 fewer LiveCodeBench problems than the unquantised model. A speed measured on the small file and a quality score published for the full model do not describe the same product.

A figure for a part is not a figure for a product

Because the three authors named the engine version, the model file and the prompt set, anyone can repeat their runs. An advertisement that names none of them is usually quoting someone else’s test of a component. The figure describes a product when the seller has measured the machine it ships and says how.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion quotes the other quantity: how many people one appliance serves, each at a floor speed. A Reasoning node serves up to 110 people with thinking off, or 70 with thinking on, at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built).

That test ran with no draft model, so the figure does not depend on whether people type code or prose. It answers the fifth question only in part: we measured it ourselves, but on a rented card, and it stays an estimate until the first appliance is measured.

Questions this article answers

Can a 32 GB graphics card write over 100 tokens per second with a 27B model?
For one request, with a draft model, on short code prompts: one test measured 208.1 on a Radeon AI PRO R9700. Plain decoding on the same card gave 27.1 to 32.3, and prose with the draft model gave 23.7 to 58.2, depending on the prompt length.
What is speculative decoding?
A small draft model proposes several tokens and the large model accepts or rejects them. It is fast when the text is predictable: 208.1 tokens per second on short code tasks in one test, against 23.7 on prose with a 70,164-token prompt.
Does quantisation reduce the quality of a model?
It can. In one test a model at 2.9 bits per weight solved 104 of 175 LiveCodeBench problems with thinking off, and the unquantised model solved 118. Ask which file the quality score was measured on.

On the record

  1. 1
  2. 2
  3. 3