Dense or mixture of experts: what the architecture costs the buyer
A mixture-of-experts model promises more people on the same card. Two published configuration files say by how much, and why the promise shrinks as more people use the model at once.
A buyer meets the words dense and mixture of experts on a spec sheet, with no way to tell what either costs. The answer is in the model's own configuration file, which every open-weight vendor publishes, and it takes about ten minutes to read.
What dense and mixture of experts mean
A dense model reads every weight it has to produce one token. A mixture-of-experts model splits most of its weights into experts and reads only a few of them per token. Qwen3.6-35B-A3B publishes 256 experts and 8 experts per token, and its model card gives 35 billion parameters in total and 3 billion activated. The promise is the memory bandwidth cost of a small model, with the knowledge of a large one.
The model does save memory, and for high-volume, low-complexity work it is the better buy.
The cache for one conversation, from the two configuration files
| Field | Qwen3.8-27B, dense | Qwen3.6-35B-A3B, MoE |
|---|---|---|
| Hidden layers | 64 | 40 |
| Of which full attention | 16 | 10 |
| Key-value heads | 4 | 2 |
| Head dimension | 256 | 256 |
| Experts, active per token | none | 256, 8 |
Three of every four layers in both models hold a fixed recurrent state instead of a growing cache. Only the full-attention layers pay per token, so the layer count alone tells you nothing.
Each full-attention layer stores a key and a value for every token. The cost of one token is 2, for the key and the value, times the full-attention layers, times the key-value heads, times the head dimension, times the bytes per number.
| Dense 27B | MoE 35B-A3B | |
|---|---|---|
| Bytes per token, fp8 | 32,768 | 10,240 |
| Bytes per token, bfloat16 | 65,536 | 20,480 |
| One 8k conversation, fp8 | 268 MB | 84 MB |
| One 8k conversation, bfloat16 | 537 MB | 168 MB |
The MoE holds under a third of the cache per conversation, because it has 10 full-attention layers against 16, and 2 key-value heads against 4. Memory is the seat constraint on a single card, so this part of the promise holds.
Where the saving shrinks: a batch reads the union of experts
Eight experts per token is true for one token. A server answers many people in one batch, and the batch has to read every expert that any of its tokens routed to. With 256 experts and 8 per token, 32 tokens can already reach all 256. Above that batch size the model can read up to its whole expert set on every step. How much of the weight saving is left is not in the configuration file; only a measurement shows it.
A MoE throughput figure quoted from the single-token case therefore describes one person using the machine. It is not a capacity figure.
The quality gap, on each vendor's own scores
| Benchmark | Dense 27B | MoE 35B-A3B |
|---|---|---|
| LiveCodeBench v6 | 90.3 | 80.4 |
| SWE-bench Pro | 61.7 | 49.5 |
The two cards compare against different sets of rival models, so each number is the vendor's own claim for its own model, not a controlled head-to-head. Even read that way, the dense model leads by 9.9 points on LiveCodeBench v6 and by 12.2 on SWE-bench Pro.
What this means for a buyer
- Many people who do simple work (classification, summarising, document question answering, overnight batches) suit the MoE: it wins on memory, and the quality gap costs little.
- Fewer people who do hard work (drafting, analysis, code, research) suit the dense model, and the department rarely feels the seat count.
- Ask any vendor for a capacity figure with four terms attached: how many concurrent users, at what context length, at what tokens per second each, with what cache precision. A number missing any of the four cannot be checked.
- For an MoE quote, ask also at what batch size it was measured. A figure taken at one request tells you nothing about thirty.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion sells the dense model as its default, because a sealed box that runs the second-best open model is the wrong box for reasoning work.
The published capacity quote is the Reasoning node, a dense model: up to 110 people with thinking off, or 70 with thinking on, at 8,000 tokens of context, at 30 tokens per second each (p90), with an fp8 key-value cache (based on a test on a rented 600 W card with the same chip, before our first box is built).
We publish no capacity number for a mixture-of-experts node. The first calculation we made ignored the expert-union effect described above, so we withdrew it. It stays withdrawn until a measurement on our own hardware supports a figure.
Questions this article answers
- What is the difference between a dense and a MoE LLM?
- A dense model reads all its weights for every token, while a mixture-of-experts model reads only a few experts: Qwen3.6-35B-A3B publishes 256 experts with 8 active per token, read 2026-09-23. The MoE therefore reads less memory per token, at the cost of quality on hard work.
- Does a MoE model serve more concurrent users?
- On memory, yes: read on 2026-09-23, the MoE holds 10,240 bytes of fp8 cache per token against the dense model's 32,768, so one 8k conversation (8,192 tokens) costs 84 MB against 268 MB. On compute the win shrinks with the batch: at 8 experts of 256 per token, 32 tokens can already reach every expert.
- Which architecture should a company choose for a private AI server?
- It depends on which constraint binds: high-volume, low-complexity work suits the MoE, and reasoning work suits the dense model. On the two vendor cards, read 2026-11-17, the dense Qwen3.8-27B scores 90.3 on LiveCodeBench v6 against the MoE's 80.4, and 61.7 against 49.5 on SWE-bench Pro. Qwen ran SWE-bench Pro on its own refined task set, in the Claude Code harness for the 27B card and in an internal agent scaffold for the 35B-A3B card, so the two scores do not compare exactly with each other, nor with a frontier vendor's own SWE-bench Pro score.
On the record
- 1
- 2
- 3
- 4
Read next
- What hardware runs a local LLM, and what the parts list leaves outThe memory arithmetic, calculated from a published model configuration and a published card specification. Why a 32-billion-parameter model does not fit on a 32 GB card, what a conversation costs in memory, and the five items no parts list contains.
- How many people can one AI server support?Concurrent users are not employees. Capacity is decided by the model, the context length, the KV cache and the serving engine, and the vLLM project found that inference systems before its engine wasted 60 to 80 per cent of memory, against under 4 per cent with it.