A local AI benchmark published two numbers. Only one travelled.
One engine, one Mac, one prompt set. The published decode rate is 54 tokens per second. The published wait before the first word is 96 seconds. Both numbers come from the same table, and only the first one reached the headlines.
Both figures are real measurements, and the vendor put the unflattering one in the same table as the flattering one. The figure the industry quotes measures the part of the wait that is usually the smaller one.
What the benchmark published: the decode rate and the wait
Inco AI publishes a benchmark for Splash, its inference engine for Apple silicon. The method is stated plainly: “All engines ran on the same 48 GB M5 Pro, serving Qwen3.6-35B-A3B and Qwen3.8-27B over HTTP at their recommended settings. The prompts are a fixed set of coding tasks from NVIDIA’s SPEED-Bench, up to 32K tokens, with a 1,024-token output limit.”
On the dense 27B model the published decode rate falls with context: 74 tokens per second on short prompts, 55 at 8K, 55 at 16K and 54 at 32K.
On a 32K prompt with a cold cache, Splash returns the first token after 96 seconds. The comparison engine, oMLX, takes 317 seconds. With the cache warm, Splash returns the first token after 282 milliseconds, and oMLX after 2,049 milliseconds.
How long a person waits, calculated end to end
The benchmark caps the answer at 1,024 tokens. At 54 tokens per second, writing that answer takes 18.96 seconds.
| Cold cache | Warm cache | |
|---|---|---|
| Silence before the first token | 96 seconds | 0.28 seconds |
| Writing the answer | 18.96 seconds | 18.96 seconds |
| Total | 114.96 seconds | 19.24 seconds |
| End to end | 8.9 tokens per second | 53.2 tokens per second |
The machine, the engine, the model and the prompt are the same. The two results differ by a factor of about six, and the cache produced all of it. The quoted 54 is the same in both cases, because decode speed does not change when the cache is cold, so it does not tell a person which wait to expect.
What a “twice as fast” claim is comparing
LM Studio summarised the result this way: “In Inco’s tests on a 48 GB M5 Pro, Splash delivered roughly twice the decode speed of the next-fastest engine they measured on Qwen3.8-27B: 74 tokens per second on short prompts and 54 at 32K context.”
Inco states the condition on its own page: “Because each engine ran at its own recommended settings, these are end-to-end comparisons.” Each Splash model ships with a draft model for speculative decoding. Of the engines compared, Inco records that one package, uzu’s, “has no draft”. The comparison therefore sets a configuration against a configuration. That is a legitimate way to benchmark, and the vendor wrote the condition down, but a reader who sees only the ratio does not see the condition.
What hardware the fast number needs
The engine is open source under the Apache-2.0 licence. Inco states the requirements on its page: “Splash needs an M3 or newer Mac on macOS 26.4 or later, and Homebrew.” For the 4-bit model it names, the page says it “needs at least 36 GB of unified memory; we recommend 48 GB or more.” At release Inco said the engine supported two models; on 17 November 2026 the repository lists five model families.
The headline figure was measured on one machine, a 48 GB M5 Pro. The engine needs an M3 or newer Mac and macOS 26.4, so the figure is not a property of the hardware an organisation already has.
Where the engine does help: concurrency and warm prompts
On the 27B model Splash completed all 16 concurrent 32K requests on the 48 GB machine. Inco says that a general-purpose memory policy accepted nine on its first pass, and that it did not test whether another engine could be configured to match. A team that runs several sessions at once feels that figure, and the warm first-token time above, turn after turn.
What this means for a regulated organisation
Four terms describe the capacity of a machine: how many people work at the same time, at what context length, at how many tokens per second each, and with which key-value cache precision. A figure without the other three cannot be compared with anything.
Two more terms describe the wait: time to first token with a cold cache, and with a warm one, at the context length the work will use. A supplier who can state all six has measured the system.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion is quoted the same way every time, per appliance and never per seat: up to 110 people with thinking off, or 70 with thinking on, at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built). The same four terms appear in the offer document, on this site and in the contract.
Those four terms describe decode, which is usually the smaller part of the wait. Time to first token is measured on the first appliance, not calculated, and it is published when the measurement exists.
A sealed local appliance also removes the variable that makes a cloud benchmark hard to reuse: the machine under the quote is the machine that serves you, and it is not shared with anyone else’s traffic.
Questions this article answers
- What is time to first token?
- The wait between sending a prompt and seeing the first word of the answer. On a 32K prompt Inco measured 96 seconds with a cold cache and 282 milliseconds with a warm one, on the same 48 GB M5 Pro.
- Why is tokens per second not the whole speed?
- It measures only the writing. In Inco’s numbers, 54 tokens per second becomes 8.9 end to end with a cold cache, because 96 of the 115 seconds are silence.
- What should a buyer ask a local AI supplier?
- Four capacity terms: concurrent users, context length, tokens per second each, and KV cache precision. And two wait terms: time to first token with a cold cache and with a warm one.
On the record
- 1
- 2
GitHub
incoai/splash: A local inference engine for Apple silicon, built around the modelread 2026-11-17
- 3
- 4
Read next
- AI costs are not only about the price per token.A launch price announced as temporary, a 50 % rise scheduled, then cancelled. The arithmetic for a 50-person team. The volume at which a fixed fee starts to win, and why predictability is a different question from cheapness.
- What happens to AI when the internet goes down?Cloudflare took ChatGPT and Claude offline in November 2025, then stumbled again in December. AWS lost a region for about fourteen and a half hours. The Iberian grid dropped two countries. What the chain from desk to model looks like when one link goes.