Skip to content
All articles
AI infrastructure

What LLM inference is, and why it is the only part you are buying

One inference request has two phases, and seven variables decide how fast it feels. Eight questions to a supplier turn “fast AI” into a figure that a buyer can verify.

BastionUpdated 3 min read

Training creates or changes a model. It is done rarely, at great expense, on very large infrastructure. Inference is what happens every time somebody asks the finished model a question. Staff who ask questions, summarise documents, extract clauses and draft replies are running inference, so an appliance should be designed for inference.

Two phases inside one request

A request has a prefill phase, where the model processes everything you gave it, and a decode phase, where it generates the answer one token at a time. The two phases stress the machine differently, which is why a single benchmark number describes neither.

PhaseScales withA request that is mostly this phase
PrefillHow much you sentA hundred-page document with a one-line question
DecodeHow much the model writesA short prompt asking for a long report
The two phases of one inference request

Users feel two numbers. Time to first token is how long the screen stays empty. Tokens per second is how fast the text then appears. A system with excellent throughput and a slow start feels sluggish, and a system that starts instantly and then crawls feels worse.

The seven variables that decide the speed

Seven variables decide both numbers. The first is the model size, with the precision the weights are stored in. The second is the available memory, because the key-value cache grows with context and concurrency. Then come the context length and how much of it is really used, the prompt length and the expected output length, and how many requests are in flight at once. The last two are the batching and memory management of the serving engine, and the hardware underneath all of it.

The serving engine changes the capacity of the same card. The vLLM project measured that inference systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” afterwards.

The eight questions to ask a supplier

  • Which model is it, and under which licence?
  • At what precision are the weights and the KV cache stored?
  • At what context length was the figure measured?
  • How many concurrent users does the figure cover?
  • What input and output lengths does it assume?
  • How many tokens per second does each user get?
  • What is the time to first token?
  • On what hardware configuration was it measured?

With the eight answers, somebody else can reproduce the measurement. Thirty tokens per second for one user and thirty tokens per second each for thirty users are different claims about different machines.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion is an inference appliance, not a training platform, and it is sized and quoted in inference terms.

A Reasoning node serves up to 110 people with thinking off, or 70 with thinking on, at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built). The other items on the list above belong in the sizing conversation, on the buyer's own documents.

Questions this article answers

What is LLM inference?
Running a finished model to answer a request. Training creates or changes the model; inference is what happens each time somebody asks it something, and it is what an organisation buying an appliance is paying for.
What are the two phases of an inference request?
Prefill, where the model processes everything it was given, and decode, where it generates the answer one token at a time. A long document with a short question is mostly prefill; a short prompt asking for a long report is mostly decode.
What should I ask a supplier about inference performance?
Eight things: the model and its licence, the precision of weights and KV cache, the context length, the number of concurrent users, the assumed input and output lengths, tokens per second per user, time to first token, and the hardware configuration.

On the record

  1. 1