Skip to content
All articles
AI infrastructure

For local AI, GPU memory decides more than GPU speed

Accelerator comparisons are usually about throughput. For a local language model, memory decides first: a card that cannot hold the model, the context and the concurrent conversations does not run the workload, however fast its arithmetic is.

BastionUpdated 2 min read

The memory of one card is shared by three things:

  • The weights. A model with billions of parameters needs billions of numbers resident before it answers anything, and the size depends on the precision they are stored in.
  • The key-value cache, the model's working memory for each conversation in flight. The vLLM project wrote that it “Takes up to 1.7GB for a single sequence in LLaMA-13B”.
  • The activations and overheads of running the computation.

Weights are a fixed cost. The cache grows with the number of people who use the system and with the length of their documents, and it is what runs a machine out of room.

Context length multiplies the cache

"Write a three-line reply to this email" and "Read these hundred pages and answer this question" use the same model, and the second holds far more in the cache for far longer. A system advertised with a long context window needs the memory to use it with more than one person at a time. Otherwise the number is theoretical.

What quantisation buys, and what it costs

Weights and cache can be stored at lower numerical precision, which reduces memory and makes local deployment possible on smaller machines. Lower precision can affect quality, and support varies by model and by engine. The trade is usually worth making, and the specification must state it: fp8 rather than bf16 for the KV cache, for example.

The engine decides how much of the memory you keep

The same vLLM publication recorded that inference systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, and that afterwards the result was “near-optimal memory usage, with a mere waste of under 4%”. A fragmentation problem is solved in the serving configuration, at no cost, before it is solved with a larger card.

The question to ask a supplier instead of GPU speed

Can this system run this model, at this context length, for this many concurrent users, at this generation speed, with this KV cache data type? That question has a yes or no answer, and a test can check it.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion answers that question in its quote, not with a card name. A Reasoning node serves up to 110 people with thinking off, or 70 with thinking on, at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built).

Questions this article answers

How much GPU memory does an LLM need?
Weights first, then working memory for every conversation in flight. vLLM puts the key-value cache at up to 1.7 GB for a single sequence in LLaMA-13B, so the total depends on the model, the context length and how many people use the system at once.
Is GPU memory more important than GPU speed?
For local language models, usually yes. A card with impressive arithmetic is still the wrong card if it cannot hold the model, the context and the concurrent conversations, because the workload will not run at all.
Does the KV cache data type matter?
Yes, materially. Storing the cache at lower precision, fp8 rather than bf16, changes the memory arithmetic enough that a capacity claim without the data type cannot be checked.

On the record

  1. 1