Type a question, get an answer. The interface is deliberately simple, and it hides a stack with five distinct layers, each of which can be the reason the system is slow, expensive or unsafe.
1. The model
The learned parameters. Larger models generally need more compute and more memory, but size alone does not decide usefulness: architecture, quantisation, context length, what it was trained on and the workload all matter.
For a business buyer the licence matters as much as the benchmark. An open-weight model with a licence that permits commercial use is a different proposition from one that does not, whatever the leaderboard says.
2. The compute
Generating tokens is arithmetic, and accelerators do a lot of arithmetic in parallel. This is the layer vendors compare, because it produces the biggest numbers.
3. The memory
The weights have to be resident, and each conversation in flight needs working memory on top. The vLLM project measured the key-value cache at “up to 1.7GB for a single sequence in LLaMA-13B”.
Multiply that by the number of people using the system at once and memory, not arithmetic, becomes the limit. This is why the fastest card is not automatically the right card.
4. The serving engine
Software that executes the model efficiently: batching requests, scheduling them, managing the cache, loading weights, handling concurrency. It is the least visible layer and one of the most consequential.
The same vLLM publication recorded the difference it makes: systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” afterwards.
5. The application
Nobody talks to a model directly. They use an application that provides chat, document search, file upload, an API, authentication, permissions and logging.
This is where the security questions live. OWASP's 2025 Top 10 for LLM applications names prompt injection, sensitive information disclosure and improper output handling, and every one of them is a property of this layer rather than of the hardware underneath.
A production AI system is a model wearing four other layers. Buying only the middle one is how projects end up unfinished.
Why "we need an AI server" is not a specification
The right response to that sentence is a question: for which workload? Ten people asking short questions and ten people analysing hundred-page documents produce completely different memory profiles on identical hardware.
Infrastructure should be sized from observed usage: how many people at once, how long the documents are, how long the answers need to be, and what the busy hour looks like. Everything else is a guess with a price attached.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed appliance that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion delivers all five layers as one supported unit, which is the point of an appliance: the integration decisions are made once and maintained by the supplier.
The application layer is still where the customer's permissions and data rules live, and no appliance decides those for them.
Questions this article answers
- What does an on-premise LLM architecture consist of?
- Five layers: the model and its licence, the compute, the memory that holds the weights and the working cache, the serving engine that batches and schedules requests, and the application that provides chat, search, authentication, permissions and logging.
- Which layer decides performance?
- Often the serving engine. vLLM measured that inference systems before PagedAttention wasted 60% to 80% of memory through fragmentation and over-reservation, against under 4% waste afterwards, on the same hardware.
- Why is "we need an AI server" not a specification?
- Because ten people asking short questions and ten people analysing hundred-page documents produce completely different memory profiles on identical hardware. Infrastructure is sized from observed usage, not from a head count.
On the record
- 1
- 2
Read next
- What an on-premise AI appliance is, and what it saves you from buildingAn AI appliance is the decision not to become an AI infrastructure company. The eleven components a local deployment needs, the one measured example of how much a wrong choice costs, and what an appliance gives up in exchange for removing the assembly.Read the article
- What LLM inference is, and why it is the only part you are buyingTraining against inference, the two phases inside a single request, the variables that decide how fast it feels, and the eight-item list that turns "fast AI" into something a buyer can verify.Read the article