What LLM inference is, and why it is the only part you are buying

Training creates or changes a model. It is done once, at great expense, on very large infrastructure. Inference is what happens every time somebody asks the finished model a question.

Organisations buying an appliance are almost never training anything. They want staff to ask questions, summarise documents, extract clauses and draft replies. That is inference, and the infrastructure should be designed for it rather than for the thing nobody is doing.

Two phases inside one request

A request has a prefill phase, where the model processes everything you gave it, and a decode phase, where it generates the answer one token at a time.

Prefill scales with how much you sent. Decode scales with how much the model writes. A hundred-page document with a one-line question is mostly prefill. A short prompt asking for a long report is mostly decode. The two stress the machine differently, which is why a single benchmark number describes neither.

What actually decides the speed

• Model size, and the precision the weights are stored in
• Available memory, because the key-value cache grows with context and concurrency
• Context length, and how much of it is really used
• Prompt length and expected output length
• How many requests are in flight at once
• The serving engine's batching and memory management
• The hardware underneath all of it

The serving engine deserves more attention than it gets. The vLLM project measured that inference systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” afterwards. Same card, different software, different capacity.

Users feel two numbers, not one

Time to first token is how long the screen stays empty. Tokens per second is how fast the text then appears. A system with excellent throughput and a slow start feels sluggish, and a system that starts instantly and then crawls feels worse.

Thirty tokens per second for one user and thirty tokens per second each for thirty users are different claims about different machines.

What to ask a supplier

• Which model, and under which licence
• At what precision, for the weights and for the KV cache
• At what context length
• For how many concurrent users
• With what assumed input and output lengths
• At how many tokens per second, per user
• At what time to first token
• On what hardware configuration

Eight answers turn a marketing adjective into a measurement somebody can reproduce. A supplier who cannot give them has not measured the thing they are selling.