How much GPU memory does an LLM need?

Weights first, then working memory for every conversation in flight. vLLM measured the key-value cache at up to 1.7 GB for a single sequence in LLaMA-13B, so the total depends on the model, the context length and how many people use the system at once.

Is GPU memory more important than GPU speed?

For local language models, usually yes. A card with impressive arithmetic is still the wrong card if it cannot hold the model, the context and the concurrent conversations, because the workload will not run at all.

Does the KV cache data type matter?

Materially. Storing the cache at lower precision, fp8 rather than bf16, changes the memory arithmetic enough that a capacity claim without the data type cannot be checked.

GPU memory requirements for LLM: why memory beats speed

Accelerator comparisons are usually about throughput: operations per second, clock speeds, generation of the architecture. For local language models, the specification that decides whether the workload runs at all is memory.

Three things compete for the same space

First, the weights. A model with billions of parameters needs billions of numbers resident before it answers anything, and the size depends on the precision they are stored in.

Second, the key-value cache, the model's working memory for each conversation in flight. The vLLM project measured that it “Takes up to 1.7GB for a single sequence in LLaMA-13B”. One sequence.

Third, the activations and overheads of actually running the computation.

Weights are a fixed cost. The cache is the one that scales with how many people are using the system and how long their documents are, and it is the one that runs a machine out of room.

Context length is a multiplier, not a feature

Compare two requests. "Write a three-line reply to this email." And: "Read these hundred pages and answer this question."

They use the same model and they are not the same workload. The second holds far more in the cache for far longer. A system advertised with a long context window has to have the memory to use it with more than one person at a time, otherwise the number is theoretical.

What quantisation buys, and what it costs

Weights and cache can be stored at lower numerical precision. That reduces memory and makes local deployment possible on smaller machines, which is why fp8 rather than bf16 for the KV cache is a real specification and not a detail.

It is a trade, not a free win. Lower precision can affect quality, and support varies by model and by engine. The honest position is that the trade is usually worth making and must be stated rather than hidden.

The engine decides how much of the memory you keep

Memory that exists is not memory that is used. The same vLLM publication recorded that inference systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, and that afterwards the result was “near-optimal memory usage, with a mere waste of under 4%”.

Buying a larger card to fix a fragmentation problem is an expensive way to avoid reading the serving configuration.

The question to ask instead

Not "how fast is the GPU?" but: can this system run this model, at this context length, for this many concurrent users, at this generation speed, with this KV cache data type?

That question has a yes or no answer and it can be tested. The first one cannot.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed appliance that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion is specified in exactly those terms rather than by card name. A Reasoning node holds 35 concurrent users, at 8k tokens of context, at 30 tokens per second each, with an fp8 KV cache.

The KV data type is in the quote because it changes the memory arithmetic materially, and a capacity claim without it is not checkable.

Questions this article answers

How much GPU memory does an LLM need?: Weights first, then working memory for every conversation in flight. vLLM measured the key-value cache at up to 1.7 GB for a single sequence in LLaMA-13B, so the total depends on the model, the context length and how many people use the system at once.
Is GPU memory more important than GPU speed?: For local language models, usually yes. A card with impressive arithmetic is still the wrong card if it cannot hold the model, the context and the concurrent conversations, because the workload will not run at all.
Does the KV cache data type matter?: Materially. Storing the cache at lower precision, fp8 rather than bf16, changes the memory arithmetic enough that a capacity claim without the data type cannot be checked.

On the record

1
vLLM
vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
read 2026-09-22

For local AI, GPU memory decides more than GPU speed