What "tokens per second" means, and why the language can change the sum
A speed figure describes one test: a number of users, a context length and a language. Change one of them and the same machine gives a different number.
A model reads tokens, the fragments of text that a tokeniser produces. A common English word is often one token. A rare word, a name or an inflected form can be several. The prompt and any retrieved document are input tokens, the answer is output tokens, and the two cost different things.
How much the language changes the token count
Tokenisers are not neutral between languages. A study published on 17 May 2023 found that “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.” It names three consequences for the affected languages: an unequal cost for commercial language services, a variable processing time and latency, and less content that fits in the context window.
The third is the trap for a Romanian buyer. An 8k context is not 8k words, and it holds less Romanian than English if the tokeniser handles Romanian less efficiently. A benchmark run on English text can therefore overstate what the same box does with Romanian contracts. Ask for the throughput figure on your own documents, in your own language.
Throughput is not response time
Tokens per second usually describes generation speed: at 30 tokens per second, a 300-token answer takes about ten seconds to write. Before the first output token appears, the model has to process the input, and a ten-thousand-token document is more work than a hundred-token question. Time to first token and tokens per second are separate measurements, and users feel both.
Concurrency is where the number moves most
Thirty tokens per second for one user alone on a machine is a different claim from thirty tokens per second each for thirty users at once. The second is much harder, and how much harder depends on the batching and the memory management of the serving engine. The vLLM project measured that layer: before PagedAttention, inference systems wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” after.
The four terms that make a quote real
A throughput figure can be checked when it states four terms: how many concurrent users it applies to, at what context length, at how many tokens per second for each user, and with which KV cache data type. A supplier who changes one of the four without saying so can produce an impressive number from the same hardware.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion states all four together. A Reasoning node serves up to 110 people with thinking off, or 70 with thinking on, at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built).
The language effect applies here too, so sizing starts from the buyer's own documents, not from a benchmark corpus.
Questions this article answers
- What does tokens per second mean?
- The rate at which the model generates output. At 30 tokens per second a 300-token answer takes about ten seconds to write, but the model must first process the input, so throughput is not the same as total response time.
- Is a tokens per second figure comparable between suppliers?
- Only with four terms attached: how many concurrent users, at what context length, at how many tokens per second each, and with which KV cache data type. Change one quietly and the same hardware can be made to produce almost any number.
- Does the language change how many tokens a document uses?
- Yes. A study published on 17 May 2023 found the same text translated into different languages can have tokenisation lengths differing up to 15 times, with consequences for cost, latency and how much content fits in the context window.
On the record
- 1
- 2
Read next
- For local AI, GPU memory decides more than GPU speedWhere the memory goes, why context length multiplies it, what quantisation buys and costs, and the question that replaces "how fast is the GPU?" when you are buying for local inference.
- What LLM inference is, and why it is the only part you are buyingTraining against inference, the two phases inside a single request, the variables that decide how fast it feels, and the eight-item list that turns "fast AI" into something a buyer can verify.