Skip to content
All articles
Private AI

What an on-premise AI appliance is and what it saves you from building

An appliance delivers the whole local AI stack as one system from one supplier. The buyer gives up the freedom to rebuild that stack.

BastionUpdated 4 min read

A local AI deployment is a stack of eleven components, and each has to be chosen, configured, secured and kept current. For a company of twenty people, that is a specialism in itself.

  • Compute and an accelerator need enough memory for the model and the context.
  • Storage holds the weights, the logs and the retrieved documents.
  • The operating environment and the driver stack must match the accelerator.
  • An inference engine schedules the requests and manages the KV cache.
  • The model needs a licence that permits commercial use.
  • Networking includes a decision about outbound connectivity.
  • Authentication comes with permissions that the application enforces.
  • Monitoring notices a failure before a user reports it.
  • A signed update path covers the model, the engine and the operating system.
  • Backups include the retrieval index.
  • Security controls cover documents, prompts and tool access.

Why the serving layer is hard: one measured example

For the inference engine, a published measurement shows the size of the effect. When the vLLM project published its work on 20 June 2023 it described the memory problem directly. The key-value cache “Takes up to 1.7GB for a single sequence in LLaMA-13B”, and inference systems of the time wasted “60% – 80% of memory due to fragmentation and over-reservation”. With PagedAttention, the project reported “near-optimal memory usage, with a mere waste of under 4%”.

For a buyer, the same card, the same model and the same users can serve a small fraction of the traffic, or nearly all of it, depending on a software choice made by whoever assembled the stack. Nothing on the invoice records the difference.

How an appliance differs from a GPU server

An appliance packages those layers into one defined system with one supplier. Instead of selecting a card, a driver, an engine, a model and a licence, and then owning the interactions between them, the organisation receives a configured system and a support route. The integration decisions were made once, tested once, and are maintained by somebody whose job it is.

A generic accelerated server arrives with hardware and a warranty. An appliance is a product with a defined purpose: a stated capacity, a stated model, an update process and a replacement policy. Ask what happens in month fourteen when a security advisory lands against the inference engine. With a server, that is your problem. With an appliance, it should be a signed update.

What an appliance costs you: flexibility

An appliance runs the model and the engine it ships with. A team that wants to swap the serving stack every quarter, or to fine-tune weekly, is better served by raw infrastructure and the staff to run it.

An appliance also does not remove the security work above the stack. OWASP's 2025 Top 10 for LLM applications begins with prompt injection and sensitive information disclosure, and both belong to the application and its permissions, not to the hosting.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion is the eleven-line list above, delivered as one unit: enterprise hardware, an open-weight model with a commercial licence, a hardened operating system, the serving stack and a local API, configured and supported together.

It comes for one fixed monthly fee, fully managed, and a failed unit is replaced whole rather than diagnosed on site. The integration work disappears, and so does the freedom to rebuild the stack yourself.

Questions this article answers

What is an on-premise AI appliance?
One supplied system that carries the whole local AI stack: hardware, an open-weight model with a commercial licence, a hardened operating system, the serving engine and a local API, configured and supported together instead of assembled in-house.
What does an on-premise LLM deployment require?
Compute with enough memory, storage, an operating system and drivers, an inference engine, a licensed model, networking, authentication, monitoring, a signed update path, backups, and security controls over documents and tool access.
Why does the inference engine matter so much?
Because it decides how much of the hardware you keep. vLLM measured that inference systems before PagedAttention wasted 60% to 80% of memory through fragmentation and over-reservation, against near-optimal use with under 4% waste afterwards.

On the record

  1. 1
  2. 2