Skip to content
All articles
AI infrastructure

What an fp8 key-value cache changes, calculated

The precision of the key-value cache is one server setting. At fp8 the same card holds 1.78 times the conversations it holds at bfloat16, and most capacity claims leave the term out.

BastionUpdated 3 min read

A language model reads back everything already said in a conversation before it writes the next word. So that it does not recompute that work each time, the server stores a key and a value for every token it has seen. That store is the key-value cache.

The weights are a fixed cost, paid once whether one person or fifty are connected. Every open conversation has its own cache, and it grows with every token. On a single card, the cache decides how many people fit.

What fp8 does to the cache

The cache does not have to be stored at the precision the model was trained in. An 8-bit floating point format holds each number in one byte instead of two. It is one server setting: the hardware, the model and the weights stay the same.

Quantizing the KV (Key-Value) cache to FP8 format can significantly reduce its memory footprint.

vLLM, Quantized KV Cache, read 2026-09-23

The arithmetic on a 96 GB card

NVIDIA's page for the RTX PRO 6000, read on 2026-09-23, gives 96 GB of GDDR7 with error-correcting code. We keep a tenth free as working space for the engine, which leaves 86.4 GB.

Qwen3.8-27B publishes 64 layers, of which 16 use full attention, with 4 key-value heads and a head dimension of 256, read on 2026-09-23. The 4-bit (NVFP4) checkpoint of the model occupies about 21.9 GB, so 64.5 GB is free for conversations.

fp8 cachebfloat16 cache
Attention cache per conversation0.268 GB0.537 GB
Fixed recurrent state per conversation0.079 GB0.079 GB
Total per conversation0.347 GB0.616 GB
Conversations that fit in 64.5 GB185104
Calculated from the published configuration on a 96 GB card, 8,192 tokens of context (8k)

185 against 104 is 1.78 times the seats on the same card, with the same weights and the same model.

The second effect: less memory to read for each token

To write one token for everybody at once, the server reads the weights that the step uses plus the cache of every open conversation. By our count of its tensors, those weights are 17.61 GB of the 21.9 GB checkpoint; the rest is the embeddings, the vision tower and a prediction head, which no step reads. At 35 concurrent conversations the fp8 step reads 17.61 + 35 × 0.347 = about 29.8 GB. The same 35 at bfloat16 read 17.61 + 35 × 0.616 = about 39.2 GB, which is 32 per cent more traffic for the same answer.

The cache precision moves memory and speed in the same direction. Two honest vendors can therefore quote user counts that differ by almost a factor of two, because they set a different cache dtype.

What fp8 costs

Halving the precision of stored numbers is an approximation. The vLLM documentation sets three limits.

  • Out of the box, nothing is calibrated. vLLM's documentation states that with no calibration all quantization scales are set to 1.0. The recommended path estimates them from a calibration dataset instead.
  • Not every layer tolerates it equally. The same documentation notes that some attention layer types are more sensitive to key-value cache quantization, and provides a flag to leave named layers at the model's own precision.
  • There is more than one 8-bit format and more than one strategy. vLLM lists fp8_e4m3 and fp8_e5m2, and quantization per tensor or per attention head. They are not interchangeable, and the per-head strategy needs the calibration path.

For most work fp8 is a good trade when it is chosen deliberately and the quality is checked on your own documents.

What to ask a vendor

A capacity claim is checkable when it carries four terms: how many concurrent users, at what context length, at what tokens per second each, and with what cache precision. Ask a fifth question of anyone quoting fp8: was the cache calibrated, or are the scales left at the default.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion writes the cache precision into every capacity figure it publishes. The Reasoning node serves up to 110 people with thinking off, or 70 with thinking on, at 8,000 tokens of context, at 30 tokens per second each (p90), with an fp8 key-value cache (based on a test on a rented 600 W card with the same chip, before our first box is built). When the box is built, its own measurement replaces this one.

Questions this article answers

What is an fp8 KV cache?
It stores each number in the model's key-value cache in one byte instead of two. Calculated from a published configuration on a 96 GB card, that raises the number of concurrent 8,192-token conversations from 104 to 185, with the same weights and the same model.
Does fp8 KV cache reduce quality?
It is an approximation, so it can: vLLM's documentation, read 2026-09-23, states that with no calibration all quantization scales are set to 1.0, and recommends estimating them from a calibration dataset. It also notes that some attention layer types are more sensitive and can be left at the model's own precision.
How much memory does a KV cache need per user?
It follows the model's configuration. Qwen3.8-27B has 16 full-attention layers, 4 key-value heads and a head dimension of 256, read 2026-09-23, which gives 0.268 GB of attention cache per 8,192-token conversation at fp8 and 0.537 GB at bfloat16, plus a fixed 0.079 GB of recurrent state.

On the record

  1. 1

    vLLM

    Quantized KV Cache

    read 2026-09-23

  2. 2

    Hugging Face

    Qwen/Qwen3.8-27B, config.json

    read 2026-09-23

  3. 3