What actually runs an AI model: the five layers behind the chat box
A chat window makes the technology look like one thing. It is five: a model, compute, memory, a serving engine and an application.
Each of the five layers can be the reason a system is slow, expensive or unsafe.
1. The model, and the licence it comes under
The model is the set of learned parameters. Larger models generally need more compute and more memory, but size alone does not decide usefulness: architecture, quantisation, context length, what it was trained on and the workload all matter. For a business buyer the licence matters as much as the benchmark. An open-weight model with a licence that permits commercial use is a different proposition from one that does not, whatever the leaderboard says.
2. The compute, the layer vendors compare
Generating tokens is arithmetic, and accelerators do a lot of arithmetic in parallel. Vendors compare this layer because it produces the biggest numbers.
3. The memory, the limit when many people work at once
The weights have to be resident, and each conversation in flight needs working memory on top. The vLLM project measured the key-value cache at “up to 1.7GB for a single sequence in LLaMA-13B”. Multiply that by the number of people using the system at once and memory, not arithmetic, becomes the limit. This is why the fastest card is not automatically the right card.
4. The serving engine, the least visible layer
The serving engine is the software that executes the model efficiently: it batches requests, schedules them, manages the cache, loads weights and handles concurrency. The same vLLM publication recorded the difference it makes: systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” afterwards.
5. The application, where the security questions live
Nobody talks to a model directly. They use an application that provides chat, document search, file upload, an API, authentication, permissions and logging. OWASP's 2025 Top 10 for LLM applications names prompt injection, sensitive information disclosure and improper output handling, and every one of them is a property of this layer rather than of the hardware underneath.
Why "we need an AI server" is not a specification
The right response to that sentence is a question: for which workload? Ten people asking short questions and ten people analysing hundred-page documents produce completely different memory profiles on identical hardware. Infrastructure should be sized from observed usage: how many people at once, how long the documents are, how long the answers need to be, and what the busy hour looks like.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion delivers all five layers as one supported unit: the integration decisions are made once and maintained by the supplier.
The customer's permissions and data rules still live in the application layer, and no appliance decides those for the customer.
Questions this article answers
- What does an on-premise LLM architecture consist of?
- Five layers: the model and its licence, the compute, the memory that holds the weights and the working cache, the serving engine that batches and schedules requests, and the application that provides chat, search, authentication, permissions and logging.
- Which layer decides performance?
- Often the serving engine. vLLM measured that inference systems before PagedAttention wasted 60% to 80% of memory through fragmentation and over-reservation, against under 4% waste afterwards.
- Why is "we need an AI server" not a specification?
- Because ten people asking short questions and ten people analysing hundred-page documents produce completely different memory profiles on identical hardware. Infrastructure is sized from observed usage, not from a head count.
On the record
- 1
- 2
Read next
- What an on-premise AI appliance is and what it saves you from buildingAn AI appliance is the decision not to become an AI infrastructure company. The eleven components a local deployment needs, the one measured example of how much a wrong choice costs, and what an appliance gives up in exchange for removing the assembly.
- What LLM inference is, and why it is the only part you are buyingTraining against inference, the two phases inside a single request, the variables that decide how fast it feels, and the eight-item list that turns "fast AI" into something a buyer can verify.