A local AI benchmark published both numbers. Only one of them travelled.

A benchmark table for a local AI engine is a real measurement. The engine exists, the numbers are published, and the vendor put the unflattering figures in the same table as the flattering ones. This article does not dispute any of them.

It follows one set of published numbers through the arithmetic that decides how long a person waits. The conclusion is not that the benchmark is wrong. It is that the number the industry quotes measures the part of the wait that is usually the smaller one.

One table, two numbers, one headline

Inco AI publishes a benchmark for Splash, its inference engine for Apple silicon. The method is stated plainly: “All engines ran on the same 48 GB M5 Pro, serving Qwen3.6-35B-A3B and Qwen3.8-27B over HTTP at their recommended settings. The prompts are a fixed set of coding tasks from NVIDIA’s SPEED-Bench, up to 32K tokens, with a 1,024-token output limit.”

On the dense 27B model the published decode rate falls with context: 74 tokens per second on short prompts, 55 at 8K, 55 at 16K and 54 at 32K. That is the number the coverage carried.

The same page publishes the other number. On a 32K prompt with a cold cache, Splash returns the first token after 96 seconds. The comparison engine, oMLX, takes 317 seconds. With the cache warm, Splash returns the first token after 282 milliseconds, and oMLX after 2,049 milliseconds.

The arithmetic the benchmark table does not print

The benchmark caps the answer at 1,024 tokens. At 54 tokens per second, writing that answer takes 19.0 seconds. That is the part the headline describes.

• Cold cache: 96 seconds of silence, then 19.0 seconds of writing. Total 115.0 seconds for 1,024 tokens, which is 8.9 tokens per second end to end.
• Warm cache: 0.3 seconds of silence, then 19.0 seconds of writing. Total 19.3 seconds for 1,024 tokens, which is 53.1 tokens per second end to end.
• Same machine, same engine, same model, same prompt. The gap between the two is a factor of about six, and the cache produced all of it.

Neither of those two end-to-end figures appears in any headline. The quoted 54 is stable across both cases, because decode speed does not change when the cache is cold. It simply stops describing the experience.

What a “twice as fast” claim is comparing

LM Studio summarised the result this way: “In Inco’s tests on a 48 GB M5 Pro, Splash delivered roughly twice the decode speed of the next-fastest engine they measured on Qwen3.8-27B: 74 tokens per second on short prompts and 54 at 32K context.”

Inco states the condition on its own page: “Because each engine ran at its own recommended settings, these are end-to-end comparisons.” Each Splash model ships with a draft model for speculative decoding. Of the engines compared, Inco records that one package, uzu’s, “has no draft”. So the comparison sets a configuration against a configuration, which is what the sentence says.

That is a legitimate way to benchmark, and the vendor wrote the condition down. The failure is downstream. A ratio travels; the sentence that qualifies it does not.

The requirements behind the fast number

The engine is open source under the Apache-2.0 licence. Its published requirements are narrow: “Apple M3 or newer, macOS 26.4 or later, Homebrew, and 36 GB of unified memory (48 GB or more recommended).” It supports two model packages, and the repository states that “Plain MLX or Transformers checkpoints do not work.”

An organisation reading the headline as a property of its existing hardware would be reading it wrong. It is a property of one chip generation, one operating system version, one memory class and one model format.

What the engine does buy is a different question

The same table records where the engineering pays. On the 27B model Splash completed all 16 concurrent 32K requests on the 48 GB machine, while oMLX accepted nine. On a warm 32K prompt it returns the first token in 282 milliseconds against 2,049. Those are the numbers a team running several sessions at once actually feels, turn after turn.

They are also the numbers nobody put on a thumbnail.

What this means for a regulated organisation

A procurement question about an AI system that quotes one speed figure is incomplete. Four terms describe the capacity of a machine: how many people work at the same time, at what context length, at how many tokens per second each, and with which key-value cache precision. A figure without the other three cannot be compared with anything.

Two more terms describe the wait: time to first token with a cold cache, and with a warm one, at the context length the work will actually use. A supplier who can state all six has measured the system. A supplier who states one has measured the part that is easiest to quote.