Skip to content
All articles
AI infrastructure

What actually runs an AI model: the five layers behind the chat box

A chat window makes the technology look like one thing. It is five: a model, compute, memory, a serving engine and an application.

BastionUpdated 3 min read

Each of the five layers can be the reason a system is slow, expensive or unsafe.

1. The model, and the licence it comes under

The model is the set of learned parameters. Larger models generally need more compute and more memory, but size alone does not decide usefulness: architecture, quantisation, context length, what it was trained on and the workload all matter. For a business buyer the licence matters as much as the benchmark. An open-weight model with a licence that permits commercial use is a different proposition from one that does not, whatever the leaderboard says.

2. The compute, the layer vendors compare

Generating tokens is arithmetic, and accelerators do a lot of arithmetic in parallel. Vendors compare this layer because it produces the biggest numbers.

3. The memory, the limit when many people work at once

The weights have to be resident, and each conversation in flight needs working memory on top. The vLLM project measured the key-value cache at “up to 1.7GB for a single sequence in LLaMA-13B”. Multiply that by the number of people using the system at once and memory, not arithmetic, becomes the limit. This is why the fastest card is not automatically the right card.

4. The serving engine, the least visible layer

The serving engine is the software that executes the model efficiently: it batches requests, schedules them, manages the cache, loads weights and handles concurrency. The same vLLM publication recorded the difference it makes: systems before PagedAttention wasted “60% – 80% of memory due to fragmentation and over-reservation”, against “near-optimal memory usage, with a mere waste of under 4%” afterwards.

5. The application, where the security questions live

Nobody talks to a model directly. They use an application that provides chat, document search, file upload, an API, authentication, permissions and logging. OWASP's 2025 Top 10 for LLM applications names prompt injection, sensitive information disclosure and improper output handling, and every one of them is a property of this layer rather than of the hardware underneath.

Why "we need an AI server" is not a specification

The right response to that sentence is a question: for which workload? Ten people asking short questions and ten people analysing hundred-page documents produce completely different memory profiles on identical hardware. Infrastructure should be sized from observed usage: how many people at once, how long the documents are, how long the answers need to be, and what the busy hour looks like.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion delivers all five layers as one supported unit: the integration decisions are made once and maintained by the supplier.

The customer's permissions and data rules still live in the application layer, and no appliance decides those for the customer.

Questions this article answers

What does an on-premise LLM architecture consist of?
Five layers: the model and its licence, the compute, the memory that holds the weights and the working cache, the serving engine that batches and schedules requests, and the application that provides chat, search, authentication, permissions and logging.
Which layer decides performance?
Often the serving engine. vLLM measured that inference systems before PagedAttention wasted 60% to 80% of memory through fragmentation and over-reservation, against under 4% waste afterwards.
Why is "we need an AI server" not a specification?
Because ten people asking short questions and ten people analysing hundred-page documents produce completely different memory profiles on identical hardware. Infrastructure is sized from observed usage, not from a head count.

On the record

  1. 1
  2. 2