What an on-premise AI appliance is and what it saves you from building
An appliance delivers the whole local AI stack as one system from one supplier. The buyer gives up the freedom to rebuild that stack.
A local AI deployment is a stack of eleven components, and each has to be chosen, configured, secured and kept current. For a company of twenty people, that is a specialism in itself.
- Compute and an accelerator need enough memory for the model and the context.
- Storage holds the weights, the logs and the retrieved documents.
- The operating environment and the driver stack must match the accelerator.
- An inference engine schedules the requests and manages the KV cache.
- The model needs a licence that permits commercial use.
- Networking includes a decision about outbound connectivity.
- Authentication comes with permissions that the application enforces.
- Monitoring notices a failure before a user reports it.
- A signed update path covers the model, the engine and the operating system.
- Backups include the retrieval index.
- Security controls cover documents, prompts and tool access.
Why the serving layer is hard: one measured example
For the inference engine, a published measurement shows the size of the effect. When the vLLM project published its work on 20 June 2023 it described the memory problem directly. The key-value cache “Takes up to 1.7GB for a single sequence in LLaMA-13B”, and inference systems of the time wasted “60% – 80% of memory due to fragmentation and over-reservation”. With PagedAttention, the project reported “near-optimal memory usage, with a mere waste of under 4%”.
For a buyer, the same card, the same model and the same users can serve a small fraction of the traffic, or nearly all of it, depending on a software choice made by whoever assembled the stack. Nothing on the invoice records the difference.
How an appliance differs from a GPU server
An appliance packages those layers into one defined system with one supplier. Instead of selecting a card, a driver, an engine, a model and a licence, and then owning the interactions between them, the organisation receives a configured system and a support route. The integration decisions were made once, tested once, and are maintained by somebody whose job it is.
A generic accelerated server arrives with hardware and a warranty. An appliance is a product with a defined purpose: a stated capacity, a stated model, an update process and a replacement policy. Ask what happens in month fourteen when a security advisory lands against the inference engine. With a server, that is your problem. With an appliance, it should be a signed update.
What an appliance costs you: flexibility
An appliance runs the model and the engine it ships with. A team that wants to swap the serving stack every quarter, or to fine-tune weekly, is better served by raw infrastructure and the staff to run it.
An appliance also does not remove the security work above the stack. OWASP's 2025 Top 10 for LLM applications begins with prompt injection and sensitive information disclosure, and both belong to the application and its permissions, not to the hosting.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion is the eleven-line list above, delivered as one unit: enterprise hardware, an open-weight model with a commercial licence, a hardened operating system, the serving stack and a local API, configured and supported together.
It comes for one fixed monthly fee, fully managed, and a failed unit is replaced whole rather than diagnosed on site. The integration work disappears, and so does the freedom to rebuild the stack yourself.
Questions this article answers
- What is an on-premise AI appliance?
- One supplied system that carries the whole local AI stack: hardware, an open-weight model with a commercial licence, a hardened operating system, the serving engine and a local API, configured and supported together instead of assembled in-house.
- What does an on-premise LLM deployment require?
- Compute with enough memory, storage, an operating system and drivers, an inference engine, a licensed model, networking, authentication, monitoring, a signed update path, backups, and security controls over documents and tool access.
- Why does the inference engine matter so much?
- Because it decides how much of the hardware you keep. vLLM measured that inference systems before PagedAttention wasted 60% to 80% of memory through fragmentation and over-reservation, against near-optimal use with under 4% waste afterwards.
On the record
- 1
- 2
Read next
- What actually runs an AI model: the five layers behind the chat boxA plain guide to AI infrastructure for people who have to approve the purchase: the five layers behind a chat box, and the reason "we need an AI server" is never a complete sentence.
- What is private AI, and when does an organisation need it?Private AI is the same kind of model with a different answer to one question: who controls the environment where the data is processed. What that answer changes, what it leaves untouched, and the five questions that settle it.