Skip to content
All articles
AI infrastructure

How many people can one AI server support?

There is no universal number. A supplier who gives a user count without three other terms quotes a benchmark you cannot reproduce. What the figure depends on, and what a usable quote looks like.

BastionUpdated 2 min read

Twenty users can be three different workloads, and each one sizes to a different box.

CompanyUsersWhat they send to the model
ATwenty peopleShort emails in bursts, a few hundred tokens in and a few hundred out
BTwenty peopleHundred-page contracts, all afternoon
CTwenty developersAutomated requests from their editors, with no human waiting
Three companies with twenty users each

Count concurrent users, not employees

A company with a hundred employees rarely has a hundred people waiting on a model in the same second, so capacity planning starts from peak concurrency. An estimate that is too low makes the system slow exactly when everyone is working. One that is too high makes the organisation pay for a machine that idles.

What each concurrent user costs in memory

The weights load once. What scales with users is the key-value cache, the model's working memory for each conversation in flight. It grows with the context length and with the number of sequences served at the same time.

The vLLM project put a number on it on 20 June 2023: the KV cache “Takes up to 1.7GB for a single sequence in LLaMA-13B”. Multiply that by the number of concurrent sequences.

The same publication recorded that inference systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” afterwards. The same card, the same model and the same users can therefore serve a fraction of the load or nearly all of it, depending on the serving engine.

What a usable quote contains

A usable capacity figure names four things together:

  • The number of concurrent users, which is not the number of licences.
  • The context length, because a 32k context is not an 8k context.
  • The generation speed for each user, in tokens per second.
  • The KV cache data type, because fp8 and bf16 are not interchangeable.

The model and the card make a fifth. Without any one of the four, the number can be made to say almost anything.

How to size a server from the workload

Estimate the peak concurrency of the users. Count the requests per user in the busy hour. Estimate the tokens in and the tokens out for the real documents. Only then look at hardware, and test it with the actual workload, not a generic benchmark.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion quotes capacity with every term attached. The Office box serves up to 60 people with thinking off, or 40 with thinking on. The Reasoning box serves up to 110 people with thinking off, or 70 with thinking on. Both quotes hold at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built).

People means named people who use it through the day, about one request every 3 minutes each, not people who press enter at the same moment.

The configurator applies one public rule. A law firm, a software company or a small office with short everyday tasks takes the Office box. Long documents, a team above 60, or any bank, insurer, hospital or public body take the Reasoning box. A heavier workload, longer documents or more people at once, takes a different node or more than one.

The step between the two is document length and quality, not head count: the Reasoning box holds about ten times more text in progress. For long documents we confirm the head count per box with you.

Questions this article answers

How many users can one AI server support?
There is no universal number: capacity is decided by the model, the context length, the KV cache and the serving engine. A usable quote names the concurrent users, the context length, the generation speed per user and the KV cache data type together.
Why do concurrent users matter more than headcount?
A company with a hundred employees rarely has a hundred people waiting on a model in the same second. Capacity planning starts from peak concurrency, not from the payroll.
How much memory does each concurrent user need?
The weights load once, and what scales with users is the key-value cache. On 20 June 2023 the vLLM project wrote that it “Takes up to 1.7GB for a single sequence in LLaMA-13B”.

On the record

  1. 1