Performance claims for AI systems are usually one number with no units around it. "30 tokens per second" is not a specification until you know for whom, at what context length and under what load.
What a token is
A model does not read words. It reads tokens, which are fragments of text produced by a tokeniser. A common English word is often one token. A rare word, a name or an inflected form can be several.
A request has input tokens, the prompt and any retrieved document. The answer has output tokens. Both count, and they cost different things.
The language effect, measured
Tokenisers are not neutral between languages. A study published on 17 May 2023 found that “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.”
The same work identifies the three consequences: unequal cost for commercial language services, variable processing time and latency, and a reduced amount of content that fits in the context window for the affected languages.
That third one is the trap for a Romanian buyer. An 8k context is not 8k words, and it holds less Romanian than English if the tokeniser handles Romanian less efficiently. A benchmark run on English text can therefore overstate what the same box does with Romanian contracts.
Ask for the throughput figure on your own documents, in your own language. It is the only version of the number that predicts your experience.
Throughput is not response time
Tokens per second usually describes generation speed. At 30 tokens per second, a 300-token answer takes about ten seconds to write.
But before the first output token appears, the model has to process the input. A ten-thousand-token document and a hundred-token question are not the same work. Time to first token and tokens per second are different measurements, and users feel both.
Concurrency is where the number moves most
Thirty tokens per second for one user alone on a machine is a different claim from thirty tokens per second each for thirty users at once. The second is harder by roughly the factor you would expect, and it depends on the serving engine's batching and memory management.
The vLLM project measured how much that layer matters: before PagedAttention, inference systems wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” after.
The four terms that make a quote real
- How many concurrent users the figure applies to
- At what context length
- At what tokens per second, per user
- With which KV cache data type
Any supplier can produce an impressive number by quietly changing one of those four. A number with all four attached can be checked, and that is the whole point of giving it.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed appliance that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion states all four together. A Reasoning node holds 35 concurrent users, at 8k tokens of context, at 30 tokens per second each, with an fp8 KV cache.
The language effect is real and it applies here too, which is why sizing starts from the buyer's own documents rather than from a benchmark corpus.
Questions this article answers
- What does tokens per second mean?
- The rate at which the model generates output. At 30 tokens per second a 300-token answer takes about ten seconds to write, but the model must first process the input, so throughput is not the same as total response time.
- Is a tokens per second figure comparable between suppliers?
- Only with four terms attached: how many concurrent users, at what context length, at how many tokens per second each, and with which KV cache data type. Change one quietly and the same hardware can be made to produce almost any number.
- Does the language change how many tokens a document uses?
- Yes. A study published on 17 May 2023 found the same text translated into different languages can have tokenisation lengths differing up to 15 times, with consequences for cost, latency and how much content fits in the context window.
On the record
- 1
- 2
Read next
- For local AI, GPU memory decides more than GPU speedWhere the memory goes, why context length multiplies it, what quantisation buys and costs, and the question that replaces "how fast is the GPU?" when you are buying for local inference.Read the article
- What LLM inference is, and why it is the only part you are buyingTraining against inference, the two phases inside a single request, the variables that decide how fast it feels, and the eight-item list that turns "fast AI" into something a buyer can verify.Read the article