What hardware runs a local LLM, and what the parts list leaves out

Search for local AI hardware and the results are build lists. A card, a power supply, a case. The lists are not wrong, and for one person at one desk they work. They are incomplete in a way that only shows up when a second person starts using the machine.

The honest order is arithmetic first, then the list of things arithmetic does not cover.

The arithmetic is public, and it is short

Every open-weight model publishes its configuration. Qwen3-32B publishes a file with the numbers that decide memory. Read on 2026-09-23, it states 64 hidden layers, 8 key-value heads, a head dimension of 128 and a bfloat16 tensor type.

Those four numbers give the cache cost of one token. Each layer stores a key and a value, each of them 8 heads wide and 128 deep. That is 2 x 64 x 8 x 128 = 131,072 numbers per token, and at two bytes each it is 262,144 bytes. 256 KiB of memory, for one token, in one conversation.

That is per conversation. Ten people holding an 8k conversation at the same moment need about 20 GiB of cache at bfloat16, or about 10 GiB at fp8. The cache dtype is not a detail. It halves the number.

Now put it on a real card

The weights come first, and they are not negotiable. A 32-billion-parameter model at two bytes per parameter is about 64 GB before a single conversation exists.

NVIDIA publishes the memory on its consumer flagship. The GeForce RTX 5090 page, read on 2026-09-23, gives its standard memory configuration:

Standard Memory Config 32 GB GDDR7

NVIDIA, GeForce RTX 5090, read 2026-09-23

So the most expensive consumer card on that page holds 32 GB, and the weights alone want about 64 GB. The model does not fit. Quantisation to 8 bits brings the weights to about 32 GB, which is the whole card, with nothing left for the cache the table above describes.

This is not an argument against local AI. It is the reason the build lists quietly use smaller models. A 7 or 8 billion parameter model fits comfortably, and for many tasks it is enough. The question is whether it is enough for the task the organisation actually has.

What the serving engine does about it

vLLM, one of the engines that serves these models, documents the problem in its own words and lists the levers.

Large models might cause your machine to run out of memory (OOM). Here are some options that help alleviate this problem.

vLLM, Conserving Memory, read 2026-09-23

The options it then names are tensor parallelism across several GPUs, quantisation, and limiting the context length and the batch size. Read the last one again. Two of the four numbers in any honest capacity quote, the context length and the number of concurrent users, are the two knobs the engine documentation tells you to turn down when memory runs out.

The five items no parts list contains

• Power and heat. A workstation card under sustained inference is not a spreadsheet load. The room it sits in has to take the heat, all day, in August.
• The operating system and its updates. Somebody applies them, tests that the engine still starts, and owns the morning it does not.
• The model licence. An open-weight model is not automatically free to use commercially, and the terms differ between families. The licence file is the thing to read, not the blog post.
• Backup and restore. The model can be downloaded again. The configuration, the prompts, the retrieval index and the access rules cannot.
• Who is on call. A machine that the whole company now depends on has an owner, or it has an outage with no owner.

None of these is exotic. All five are the difference between a machine that works at one desk and a system a company can depend on.

What to ask before buying anything

Ask for the four numbers together, and ask which of them were measured. How many people at the same moment, holding how much context each, at how many tokens per second, with which cache dtype. A supplier who answers with one number is quoting the easy one.

Then ask the five questions above. The answers decide the running cost, and the running cost is where a build list is usually wrong.