"How many users does it support?" is close to asking how many cars a road carries. The road does not know. It depends on the traffic.
Three companies, twenty users each, three different machines
Company A has twenty people writing short emails, a few hundred tokens in and a few hundred out, in bursts. Company B has twenty people feeding hundred-page contracts through the model all afternoon. Company C has twenty developers whose editors send automated requests without a human waiting.
All three have twenty users. They are not the same workload, and they do not size to the same box.
Concurrent is the word that matters
A company with a hundred employees rarely has a hundred people waiting on a model in the same second. Capacity planning starts from peak concurrency, not from the payroll.
Get that wrong in the optimistic direction and the system is slow exactly when everyone is working. Get it wrong in the other direction and the organisation pays for a machine that idles.
Where the memory actually goes
The weights load once. What scales with users is the key-value cache, the model's working memory for each conversation in flight. It grows with the context length and with the number of sequences being served at the same time.
The vLLM project put a number on it on 20 June 2023: the KV cache “Takes up to 1.7GB for a single sequence in LLaMA-13B”. One sequence. Multiply by concurrency and the arithmetic stops being abstract.
The same publication recorded how badly that memory was being used before PagedAttention: inference systems wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” afterwards.
The same card, the same model and the same users can serve a fraction of the load or nearly all of it, depending on the serving engine. Capacity is a property of the whole stack.
What a usable quote contains
A capacity figure with one number in it is not a specification. A usable one names four things at once, and a fifth if you are being thorough:
- How many concurrent users, not how many licences
- At what context length, because a 32k context is not a 8k context
- At what generation speed per user, in tokens per second
- With which KV cache data type, because fp8 and bf16 are not interchangeable
- And ideally, on which model and which card
Four of those are the quote. Drop any one and the number can be made to say almost anything.
How to size it properly
Start from users, estimate peak concurrency, count requests per user in the busy hour, estimate tokens in and tokens out for the real documents, and only then look at hardware. Then test with the actual workload rather than a generic benchmark. A benchmark that does not resemble the work predicts nothing about the work.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed appliance that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion quotes capacity with every term attached. A Reasoning node holds 35 concurrent users, at 8k tokens of context, at 30 tokens per second each, with an fp8 KV cache. That is the whole quote, and it is what the sizing in an offer is built from.
A workload that is heavier in any of those dimensions, longer documents or more people at once, is a different node or more than one. Which one it is comes out of the sizing conversation, not out of a head count.
On the record
- 1