Skip to content
All articles
AI infrastructure

Dense or mixture of experts: what the architecture costs the buyer

A mixture-of-experts model promises more people on the same card. Two published configuration files say by how much, and why the promise shrinks as more people use the model at once.

BastionUpdated 3 min read

A buyer meets the words dense and mixture of experts on a spec sheet, with no way to tell what either costs. The answer is in the model's own configuration file, which every open-weight vendor publishes, and it takes about ten minutes to read.

What dense and mixture of experts mean

A dense model reads every weight it has to produce one token. A mixture-of-experts model splits most of its weights into experts and reads only a few of them per token. Qwen3.6-35B-A3B publishes 256 experts and 8 experts per token, and its model card gives 35 billion parameters in total and 3 billion activated. The promise is the memory bandwidth cost of a small model, with the knowledge of a large one.

The model does save memory, and for high-volume, low-complexity work it is the better buy.

The cache for one conversation, from the two configuration files

FieldQwen3.8-27B, denseQwen3.6-35B-A3B, MoE
Hidden layers6440
Of which full attention1610
Key-value heads42
Head dimension256256
Experts, active per tokennone256, 8
From the published config.json of each model, read 2026-09-23

Three of every four layers in both models hold a fixed recurrent state instead of a growing cache. Only the full-attention layers pay per token, so the layer count alone tells you nothing.

Each full-attention layer stores a key and a value for every token. The cost of one token is 2, for the key and the value, times the full-attention layers, times the key-value heads, times the head dimension, times the bytes per number.

Dense 27BMoE 35B-A3B
Bytes per token, fp832,76810,240
Bytes per token, bfloat1665,53620,480
One 8k conversation, fp8268 MB84 MB
One 8k conversation, bfloat16537 MB168 MB
Cache for one conversation, calculated from the configurations above; 8k is 8,192 tokens

The MoE holds under a third of the cache per conversation, because it has 10 full-attention layers against 16, and 2 key-value heads against 4. Memory is the seat constraint on a single card, so this part of the promise holds.

Where the saving shrinks: a batch reads the union of experts

Eight experts per token is true for one token. A server answers many people in one batch, and the batch has to read every expert that any of its tokens routed to. With 256 experts and 8 per token, 32 tokens can already reach all 256. Above that batch size the model can read up to its whole expert set on every step. How much of the weight saving is left is not in the configuration file; only a measurement shows it.

A MoE throughput figure quoted from the single-token case therefore describes one person using the machine. It is not a capacity figure.

The quality gap, on each vendor's own scores

BenchmarkDense 27BMoE 35B-A3B
LiveCodeBench v690.380.4
SWE-bench Pro61.749.5
Vendor-reported scores, each from that model's own card, read 2026-11-17. Qwen ran SWE-bench Pro on its own refined task set: for the 27B card in the Claude Code harness, for the 35B-A3B card in an internal agent scaffold. The two scores therefore do not compare exactly with each other, nor with a frontier vendor's own SWE-bench Pro score

The two cards compare against different sets of rival models, so each number is the vendor's own claim for its own model, not a controlled head-to-head. Even read that way, the dense model leads by 9.9 points on LiveCodeBench v6 and by 12.2 on SWE-bench Pro.

What this means for a buyer

  • Many people who do simple work (classification, summarising, document question answering, overnight batches) suit the MoE: it wins on memory, and the quality gap costs little.
  • Fewer people who do hard work (drafting, analysis, code, research) suit the dense model, and the department rarely feels the seat count.
  • Ask any vendor for a capacity figure with four terms attached: how many concurrent users, at what context length, at what tokens per second each, with what cache precision. A number missing any of the four cannot be checked.
  • For an MoE quote, ask also at what batch size it was measured. A figure taken at one request tells you nothing about thirty.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion sells the dense model as its default, because a sealed box that runs the second-best open model is the wrong box for reasoning work.

The published capacity quote is the Reasoning node, a dense model: up to 110 people with thinking off, or 70 with thinking on, at 8,000 tokens of context, at 30 tokens per second each (p90), with an fp8 key-value cache (based on a test on a rented 600 W card with the same chip, before our first box is built).

We publish no capacity number for a mixture-of-experts node. The first calculation we made ignored the expert-union effect described above, so we withdrew it. It stays withdrawn until a measurement on our own hardware supports a figure.

Questions this article answers

What is the difference between a dense and a MoE LLM?
A dense model reads all its weights for every token, while a mixture-of-experts model reads only a few experts: Qwen3.6-35B-A3B publishes 256 experts with 8 active per token, read 2026-09-23. The MoE therefore reads less memory per token, at the cost of quality on hard work.
Does a MoE model serve more concurrent users?
On memory, yes: read on 2026-09-23, the MoE holds 10,240 bytes of fp8 cache per token against the dense model's 32,768, so one 8k conversation (8,192 tokens) costs 84 MB against 268 MB. On compute the win shrinks with the batch: at 8 experts of 256 per token, 32 tokens can already reach every expert.
Which architecture should a company choose for a private AI server?
It depends on which constraint binds: high-volume, low-complexity work suits the MoE, and reasoning work suits the dense model. On the two vendor cards, read 2026-11-17, the dense Qwen3.8-27B scores 90.3 on LiveCodeBench v6 against the MoE's 80.4, and 61.7 against 49.5 on SWE-bench Pro. Qwen ran SWE-bench Pro on its own refined task set, in the Claude Code harness for the 27B card and in an internal agent scaffold for the 35B-A3B card, so the two scores do not compare exactly with each other, nor with a frontier vendor's own SWE-bench Pro score.

On the record

  1. 1

    Hugging Face

    Qwen/Qwen3.8-27B, config.json

    read 2026-09-23

  2. 2

    Hugging Face

    Qwen/Qwen3.6-35B-A3B, config.json

    read 2026-09-23

  3. 3
  4. 4