Skip to content
All articles
AI infrastructure

What hardware runs a local LLM, and what the parts list leaves out

The memory arithmetic that decides which card runs a model is public, and it is short. A parts list leaves out five things that decide whether a team can depend on the machine.

BastionUpdated 4 min read

A build list for local AI names a card, a power supply and a case, and for one person at one desk it works. Its gaps show when a second person starts to use the machine.

How much memory one conversation needs

Qwen3-32B publishes its configuration, as open-weight models on Hugging Face normally do. Read on 2026-09-23, the file states 64 hidden layers, 8 key-value heads, a head dimension of 128 and a bfloat16 tensor type.

Those four numbers give the cache cost of one token. Each layer stores a key and a value, each of them 8 heads wide and 128 deep. That is 2 x 64 x 8 x 128 = 131,072 numbers per token, and at two bytes each it is 262,144 bytes, or 256 KiB of memory for one token in one conversation.

Context heldAt bfloat16 cacheAt fp8 cache
1,000 tokens0.24 GiB0.12 GiB
8,000 tokens1.95 GiB0.98 GiB
40,960 tokens, the published maximum10.0 GiB5.0 GiB
Memory for one conversation, calculated from the Qwen3-32B configuration published on Hugging Face

Ten people who each hold an 8k conversation at the same moment need about 20 GiB of cache at bfloat16, or about 10 GiB at fp8.

Does the model fit on a 32 GB card?

The weights load before any conversation exists. A 32-billion-parameter model at two bytes per parameter is about 64 GB. The NVIDIA page for the GeForce RTX 5090, its consumer flagship, read on 2026-09-23, gives the standard memory configuration:

Standard Memory Config 32 GB GDDR7

NVIDIA, GeForce RTX 5090, read 2026-09-23

The model does not fit. Quantisation to 8 bits brings the weights to about 32 GB, which is the whole card, with nothing left for the cache in the table above.

This is why build lists often use smaller models and do not say so. A 7 or 8 billion parameter model fits comfortably, and for many tasks it is enough. The buyer has to check whether it is enough for the organisation's own task.

What the serving engine can do when memory runs out

vLLM, one of the engines that serve these models, documents the problem:

Large models might cause your machine to run out of memory (OOM). Here are some options that help alleviate this problem.

vLLM, Conserving Memory, read 2026-11-17

The first three options it names are tensor parallelism across several GPUs, quantisation, and limits on the context length and the batch size. The context length and the number of concurrent users are two of the four numbers in a complete capacity quote, and they are the two that the engine documentation says to turn down when memory runs out.

The five items no parts list contains

  • Power and heat. A workstation card under sustained inference is not a spreadsheet load. The room it sits in has to take the heat all day.
  • The operating system and its updates. Somebody applies them, tests that the engine still starts, and owns the morning it does not.
  • The model licence. An open-weight model is not automatically free to use commercially, and the terms differ between families. Read the licence file, not the blog post.
  • Backup and restore. The model can be downloaded again. The configuration, the prompts, the retrieval index and the access rules cannot.
  • Who is on call. A machine that the whole company depends on needs a named owner for the day it fails.

Before buying anything, ask for the four numbers together, and ask which of them were measured: how many people at the same moment, how much context each holds, how many tokens per second each gets, and which cache dtype. A supplier who answers with one number is quoting the easy one. Then ask about the five items. The answers decide the running cost, and the running cost is where a build list is usually wrong.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion sells none of this as a parts list. The hardware, the open-weight model, the hardened operating system, the updates and the support arrive as one system, for one monthly fee, fully managed.

Our own capacity figure started as arithmetic of this kind. It is now based on a test on a rented 600 W card with the same chip, before our first box is built. The measurement on that box will replace it.

Questions this article answers

How much VRAM do I need to run a local LLM?
A 32-billion-parameter model at two bytes per parameter needs about 64 GB for the weights alone, which does not fit on a 32 GB card. On top of that, Qwen3-32B costs 256 KiB of cache per token at bfloat16, so one 8,000 token conversation adds about 1.95 GiB.
Can one graphics card run a local AI server for a team?
For one person, often yes; for a team the cache is the constraint. Ten people each holding 8,000 tokens need about 20 GiB of cache at bfloat16, or about 10 GiB at fp8, on top of the weights, and the published NVIDIA GeForce RTX 5090 memory configuration is 32 GB in total.
What does a local AI hardware list leave out?
Five things: sustained power and heat, operating system updates and who tests them, the model licence terms, backup of the configuration and retrieval index rather than the model, and a named owner when the machine fails.

On the record

  1. 1

    Hugging Face

    Qwen/Qwen3-32B, config.json

    read 2026-09-23

  2. 2
  3. 3

    vLLM

    Conserving Memory

    read 2026-11-17