A local AI deployment is not a model on a server. It is a stack, and each layer has to be chosen, configured, secured and kept current.
- Compute and an accelerator with enough memory for the model and the context
- Storage for weights, logs and retrieved documents
- An operating environment and a driver stack that match the accelerator
- An inference engine that schedules requests and manages the KV cache
- A model with a licence that permits commercial use
- Networking, and a decision about outbound connectivity
- Authentication, and permissions that the application enforces
- Monitoring, so a failure is noticed before a user reports it
- A signed update path for the model, the engine and the operating system
- Backups, including of the retrieval index
- Security controls over documents, prompts and tool access
For a company of twenty people, that is not a weekend. It is a specialism.
One measured example of why the serving layer is hard
The inference engine looks like the least interesting item on the list. It is the one with the largest measured effect.
When the vLLM project published its work on 20 June 2023 it described the memory problem directly. The key-value cache “Takes up to 1.7GB for a single sequence in LLaMA-13B”, and inference systems of the time wasted “60% – 80% of memory due to fragmentation and over-reservation”. With PagedAttention, the project reported “near-optimal memory usage, with a mere waste of under 4%”.
Read what that means for a buyer. The same card, the same model and the same users can serve a small fraction of the traffic, or nearly all of it, depending on a software choice made by whoever assembled the stack. Nothing on the invoice records the difference.
The accelerator is the part everyone compares. The serving layer is the part that decides how much of it you actually get.
What an appliance is
An appliance packages those layers into one defined system with one supplier. Instead of selecting a card, a driver, an engine, a model and a licence, and then owning the interactions between them, the organisation receives a configured system and a support route.
The value is not the box. The value is that the integration decisions were made once, tested once, and are maintained by somebody whose work it is.
An appliance is not a GPU server
A generic accelerated server is infrastructure. It arrives with hardware and a warranty. An appliance is a product with a defined purpose: a stated capacity, a stated model, an update process and a replacement policy.
The test is simple. Ask what happens in month fourteen when a security advisory lands against the inference engine. With a server, that is your problem. With an appliance, it should be a signed update.
What it costs you
Simplicity is bought with flexibility. An appliance runs the model and the engine it ships with. A team that wants to swap the serving stack every quarter, or to fine-tune weekly, is better served by raw infrastructure and the staff to run it.
It also does not remove the security work above the stack. OWASP's 2025 Top 10 for LLM applications begins with prompt injection and sensitive information disclosure, and both belong to the application and its permissions, not to the hosting.
So the question is not whether the organisation could build it. Technically, almost any competent team could. The question is whether the people who would build it have nothing more valuable to do.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed appliance that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion is the eleven-line list above, delivered as one unit: enterprise hardware, an open-weight model with a commercial licence, a hardened operating system, the serving stack and a local API, configured and supported together.
It is rented at a fixed monthly fee, and a failed unit is replaced whole rather than diagnosed on site. The trade is the one this article describes: the integration work disappears, and so does the freedom to rebuild the stack yourself.
Questions this article answers
- What is an on-premise AI appliance?
- One supplied system that carries the whole local AI stack: hardware, an open-weight model with a commercial licence, a hardened operating system, the serving engine and a local API, configured and supported together instead of assembled in-house.
- What does an on-premise LLM deployment actually require?
- Compute with enough memory, storage, an operating system and drivers, an inference engine, a licensed model, networking, authentication, monitoring, a signed update path, backups, and security controls over documents and tool access.
- Why does the inference engine matter so much?
- Because it decides how much of the hardware you keep. vLLM measured that inference systems before PagedAttention wasted 60% to 80% of memory through fragmentation and over-reservation, against near-optimal use with under 4% waste afterwards.
On the record
- 1
- 2
- 3
Read next
- What actually runs an AI model: the five layers behind the chat boxA plain guide to AI infrastructure for people who have to approve the purchase, and the reason "we need an AI server" is never a complete sentence.Read the article
- What is private AI, and when does an organisation actually need it?Private AI is not a different kind of model. It is a different answer to one question: who controls the environment where the data is processed. What that answer changes, what it leaves untouched, and the five questions that settle it.Read the article