Skip to content
All articles
AI infrastructure

A local AI benchmark published two numbers. Only one travelled.

One engine, one Mac, one prompt set. The published decode rate is 54 tokens per second. The published wait before the first word is 96 seconds. Both numbers come from the same table, and only the first one reached the headlines.

BastionUpdated 3 min read

Both figures are real measurements, and the vendor put the unflattering one in the same table as the flattering one. The figure the industry quotes measures the part of the wait that is usually the smaller one.

What the benchmark published: the decode rate and the wait

Inco AI publishes a benchmark for Splash, its inference engine for Apple silicon. The method is stated plainly: “All engines ran on the same 48 GB M5 Pro, serving Qwen3.6-35B-A3B and Qwen3.8-27B over HTTP at their recommended settings. The prompts are a fixed set of coding tasks from NVIDIA’s SPEED-Bench, up to 32K tokens, with a 1,024-token output limit.”

On the dense 27B model the published decode rate falls with context: 74 tokens per second on short prompts, 55 at 8K, 55 at 16K and 54 at 32K.

On a 32K prompt with a cold cache, Splash returns the first token after 96 seconds. The comparison engine, oMLX, takes 317 seconds. With the cache warm, Splash returns the first token after 282 milliseconds, and oMLX after 2,049 milliseconds.

How long a person waits, calculated end to end

The benchmark caps the answer at 1,024 tokens. At 54 tokens per second, writing that answer takes 18.96 seconds.

Cold cacheWarm cache
Silence before the first token96 seconds0.28 seconds
Writing the answer18.96 seconds18.96 seconds
Total114.96 seconds19.24 seconds
End to end8.9 tokens per second53.2 tokens per second
Calculated from Inco AI’s published figures: a 32K prompt and a 1,024-token answer

The machine, the engine, the model and the prompt are the same. The two results differ by a factor of about six, and the cache produced all of it. The quoted 54 is the same in both cases, because decode speed does not change when the cache is cold, so it does not tell a person which wait to expect.

What a “twice as fast” claim is comparing

LM Studio summarised the result this way: “In Inco’s tests on a 48 GB M5 Pro, Splash delivered roughly twice the decode speed of the next-fastest engine they measured on Qwen3.8-27B: 74 tokens per second on short prompts and 54 at 32K context.”

Inco states the condition on its own page: “Because each engine ran at its own recommended settings, these are end-to-end comparisons.” Each Splash model ships with a draft model for speculative decoding. Of the engines compared, Inco records that one package, uzu’s, “has no draft”. The comparison therefore sets a configuration against a configuration. That is a legitimate way to benchmark, and the vendor wrote the condition down, but a reader who sees only the ratio does not see the condition.

What hardware the fast number needs

The engine is open source under the Apache-2.0 licence. Inco states the requirements on its page: “Splash needs an M3 or newer Mac on macOS 26.4 or later, and Homebrew.” For the 4-bit model it names, the page says it “needs at least 36 GB of unified memory; we recommend 48 GB or more.” At release Inco said the engine supported two models; on 17 November 2026 the repository lists five model families.

The headline figure was measured on one machine, a 48 GB M5 Pro. The engine needs an M3 or newer Mac and macOS 26.4, so the figure is not a property of the hardware an organisation already has.

Where the engine does help: concurrency and warm prompts

On the 27B model Splash completed all 16 concurrent 32K requests on the 48 GB machine. Inco says that a general-purpose memory policy accepted nine on its first pass, and that it did not test whether another engine could be configured to match. A team that runs several sessions at once feels that figure, and the warm first-token time above, turn after turn.

What this means for a regulated organisation

Four terms describe the capacity of a machine: how many people work at the same time, at what context length, at how many tokens per second each, and with which key-value cache precision. A figure without the other three cannot be compared with anything.

Two more terms describe the wait: time to first token with a cold cache, and with a warm one, at the context length the work will use. A supplier who can state all six has measured the system.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion is quoted the same way every time, per appliance and never per seat: up to 110 people with thinking off, or 70 with thinking on, at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built). The same four terms appear in the offer document, on this site and in the contract.

Those four terms describe decode, which is usually the smaller part of the wait. Time to first token is measured on the first appliance, not calculated, and it is published when the measurement exists.

A sealed local appliance also removes the variable that makes a cloud benchmark hard to reuse: the machine under the quote is the machine that serves you, and it is not shared with anyone else’s traffic.

Questions this article answers

What is time to first token?
The wait between sending a prompt and seeing the first word of the answer. On a 32K prompt Inco measured 96 seconds with a cold cache and 282 milliseconds with a warm one, on the same 48 GB M5 Pro.
Why is tokens per second not the whole speed?
It measures only the writing. In Inco’s numbers, 54 tokens per second becomes 8.9 end to end with a cold cache, because 96 of the 115 seconds are silence.
What should a buyer ask a local AI supplier?
Four capacity terms: concurrent users, context length, tokens per second each, and KV cache precision. And two wait terms: time to first token with a cold cache and with a warm one.

On the record

  1. 1
  2. 2
  3. 3
  4. 4

    Inco AI

    DFlash 2: Keep Drafting Parallel

    read 2026-09-22