How to evaluate an AI deployment: a checklist for 5 to 100 people
Start with the workload, not with which model is smartest: ten questions about your own work, then twenty-seven for any supplier.
A demonstration is the easy part of an AI purchase to arrange. This checklist covers the part that decides whether the deployment still looks sensible in month fourteen.
Ten questions to answer about your own organisation
Answer these before you contact a supplier.
- What problem are we solving? Write the workflow, not the ambition.
- What data will the system process, and how is it classified?
- Does that data have to leave the organisation for the work to happen?
- How many people will use it at the same moment, not how many are employed?
- How long are the typical documents, in pages?
- What level of model capability does the work require?
- Does the system need internet access, and for which specific step?
- What happens when the model is wrong, and who checks?
- What happens when the system is unavailable for a day?
- What happens when the model version changes, and who approves it?
Question four is the easy one to answer wrongly: capacity is about concurrency, not head count.
Technical questions: what runs, and how it was measured
- Which model runs, and under what licence for commercial use?
- At what precision are the weights and the KV cache?
- What is the hardware configuration, and how much memory is available?
- What is the context length, and what is it at full concurrency?
- How many concurrent users does the quote cover?
- How many tokens per second does each user get, and what is the time to first token?
- Under which exact benchmark conditions were those numbers measured?
- How are the model, the serving stack and the operating system updated?
A capacity figure needs four terms together: how many concurrent users, at what context length, at what tokens per second each, with which KV cache data type. One number alone can be made to mean almost anything.
Data questions: where the work happens and what is kept
- Where is inference performed, and can that location change without your approval?
- Where are prompts stored, and for how long?
- Where are logs stored, and who can read them?
- Does any information leave the network in normal operation?
- Which subprocessors are involved, if any?
- Can the system operate with no outbound connectivity?
- What happens to your data if the contract ends?
For a financial entity, DORA Article 30(2)(b) already requires the contract to state “the locations, namely the regions or countries, where the contracted or subcontracted functions and ICT services are to be provided and where data is to be processed, including the storage location”. Any other organisation can ask the same question.
Security questions: who and what the model can reach
- How are users authenticated, and are shared accounts possible?
- Are document permissions enforced at retrieval time?
- How is retrieved content separated from instructions?
- What tools or systems can the model reach, and under whose authority?
- How are outputs validated before another system acts on them?
- What does the supplier do when a vulnerability is disclosed in the serving stack?
OWASP's 2025 Top 10 for LLM applications is the reference list behind those questions, and by our reading five of its ten entries are unaffected by where the model is hosted.
Commercial questions: cost, support and exit
- Is the cost fixed or usage-based, and what drives the variable part?
- What are the minimum contract term and the notice period?
- What support is included, and what is the response commitment?
- What happens when hardware fails?
- Are upgrades included or charged?
- What are the exit terms, and what do you keep?
How to read a supplier's answers
- A supplier who answers the technical list with adjectives has not measured the product.
- A supplier who answers the data list with a slogan has not drawn the diagram.
- A supplier who answers the security list with "it is on-premise" has not read OWASP.
A supplier who concedes that the cloud is cheaper for light, spiky usage is, in our view, telling you something true. That is a reasonable proxy for the rest of what they say.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion answers the four lists in fixed terms. Inference happens inside one box, at the customer's site or in a data centre; no outbound connection is required, the fee is one fixed monthly amount and a failed unit is replaced whole.
On capacity, a Reasoning node serves up to 110 people with thinking off, or 70 with thinking on, at 8k tokens of context, at 30 tokens per second each (p90), with an fp8 KV cache (based on a test on a rented 600 W card with the same chip, before our first box is built). People means named people who use it through the day, about one request every 3 minutes each, not people who press enter at the same moment.
If your answers to the ten questions point to the cloud, that is a valid result. If they point inwards, tell us what you want to run, how many people will use it and what data is involved, and we will tell you whether Bastion fits.
Questions this article answers
- What should an AI vendor due diligence checklist cover?
- Four areas: the technical specification including model, licence, memory, context and concurrency; the data flow including where inference happens and where prompts and logs are stored; the security architecture; and the commercial terms including exit.
- What is the first question to ask before buying AI?
- Ask what the workload is before asking which model is smartest: what the system will read, how the data is classified, whether it has to leave the organisation, and how many people will use it at the same moment.
- How do I check a capacity claim?
- Ask for four terms together: how many concurrent users, at what context length, at how many tokens per second each, and with which KV cache data type. A single number without those can be made to mean almost anything.
On the record
- 1
- 2
Read next
- Seven questions to answer before you decide where AI runsStart with the workload, not the hardware. What the system will read, whether the data has to leave, how steady the usage is, whether the internet is needed, how strong the model has to be, who operates it, and what happens when it is unavailable.
- What is private AI, and when does an organisation need it?Private AI is the same kind of model with a different answer to one question: who controls the environment where the data is processed. What that answer changes, what it leaves untouched, and the five questions that settle it.