What an fp8 key-value cache actually changes, calculated

Most capacity claims for an AI server name a number of users. Some add a context length. Almost none name the precision of the key-value cache, and that omission alone can move the answer by nearly a factor of two.

What the key-value cache is, and why it runs out first

A language model reads back everything already said in a conversation before it writes the next word. Rather than recompute that work each time, the server stores a key and a value for every token it has seen. That store is the key-value cache.

The weights are a fixed cost. You pay for them once, whether one person or fifty are connected. The cache is different: every open conversation has its own, and it grows with every token in it. On a single card the weights are the entry fee and the cache is what decides how many people fit.

What fp8 does to it

The cache does not have to be stored at the precision the model was trained in. An 8-bit floating point format holds each number in one byte instead of two. The vLLM project, whose documentation was read on 2026-09-23, describes the effect plainly:

Quantizing the KV (Key-Value) cache to FP8 format can significantly reduce its memory footprint.

vLLM, Quantized KV Cache, read 2026-09-23

It is one server setting. No new hardware, no different model, no change to the weights.

The arithmetic on a real card

Take a 96 GB accelerator. NVIDIA's own page for the RTX PRO 6000, read on 2026-09-23, gives 96 GB of GDDR7 with error-correcting code. Reserve a tenth of it for the working space every inference engine needs, and 86.4 GB is left.

Now take a real model. Qwen3.8-27B publishes 64 layers, of which 16 use full attention, with 4 key-value heads and a head dimension of 256, read on 2026-09-23. Its published checkpoint occupies about 21.9 GB. So 64.5 GB is free for conversations.

One setting, 1.78 times the seats. Same card, same weights, same model.

The second effect, which is about speed

Memory is only the first constraint. To write one token for everybody at once, the server has to read the active weights plus every open conversation's cache. At 35 concurrent conversations the fp8 step reads about 29.8 GB. The same 35 at bfloat16 read about 39.2 GB, which is 32 per cent more traffic for the same answer.

So the cache precision moves both constraints in the same direction. This is why two honest vendors can quote user counts that differ by half, and neither is lying. They set a different cache dtype.

What fp8 costs, because it is not free

Halving the precision of stored numbers is an approximation, and the engine documentation says so in its own configuration options. Three things are worth knowing before anyone treats fp8 as a default.

• Out of the box, nothing is calibrated. vLLM's documentation states that with no calibration all quantization scales are set to 1.0. The recommended path estimates them from a calibration dataset instead.
• Not every layer tolerates it equally. The same documentation notes that some attention layer types are more sensitive to key-value cache quantization, and provides a flag to leave named layers at the model's own precision.
• There is more than one 8-bit format and more than one strategy. vLLM lists fp8_e4m3 and fp8_e5m2, and quantization per tensor or per attention head. They are not interchangeable, and the per-head strategy needs the calibration path.

The honest position is that fp8 is a very good trade for most work, taken deliberately, with the quality checked on your own documents. It is not a free doubling of capacity.

What to ask a vendor

A capacity claim is checkable when it carries four terms: how many concurrent users, at what context length, at what tokens per second each, and with what cache precision. Ask a fifth question of anyone quoting fp8: was the cache calibrated, or are the scales left at the default.