Most AI purchases begin with a demonstration and end with a surprise. The demonstration is the easy part to arrange. This checklist is the part that decides whether the deployment still looks sensible in month fourteen.
Ten questions about your own organisation
Answer these before contacting anybody. They cost nothing and they shorten every conversation afterwards.
- What problem are we solving? Write the workflow, not the ambition.
- What data will the system process, and how is it classified?
- Does that data have to leave the organisation for the work to happen?
- How many people will use it at the same moment, not how many are employed?
- How long are the typical documents, in pages?
- What level of model capability does the work actually require?
- Does the system need internet access, and for which specific step?
- What happens when the model is wrong, and who checks?
- What happens when the system is unavailable for a day?
- What happens when the model version changes, and who approves it?
Question four is the one most often answered wrongly. Capacity is about concurrency, not head count.
The technical list
- Which model, and under what licence for commercial use
- At what precision, for the weights and for the KV cache
- The hardware configuration and the available memory
- The context length, and the context length at full concurrency
- The number of concurrent users the quote covers
- Tokens per second per user, and time to first token
- The exact benchmark conditions behind those numbers
- The update process for model, serving stack and operating system
A capacity figure needs four terms together: how many concurrent users, at what context length, at what tokens per second each, with which KV cache data type. One number alone can be made to mean almost anything.
The data list
- Where is inference performed, and can that location change without your approval?
- Where are prompts stored, and for how long?
- Where are logs stored, and who can read them?
- Does any information leave the network in normal operation?
- Which subprocessors are involved, if any?
- Can the system operate with no outbound connectivity?
- What happens to your data if the contract ends?
For a financial entity, DORA Article 30(2)(b) already requires the contract to state “the locations, namely the regions or countries, where the contracted or subcontracted functions and ICT services are to be provided and where data is to be processed, including the storage location”. Everyone else benefits from asking the same question.
The security list
- How are users authenticated, and are shared accounts possible?
- Are document permissions enforced at retrieval time?
- How is retrieved content separated from instructions?
- What tools or systems can the model reach, and under whose authority?
- How are outputs validated before another system acts on them?
- What does the supplier do when a vulnerability is disclosed in the serving stack?
OWASP's 2025 Top 10 for LLM applications is the reference list behind those questions, and five of its ten entries are unaffected by where the model is hosted.
The commercial list
- Fixed or usage-based cost, and what drives the variable part
- Minimum contract term and notice period
- What support is included, and the response commitment
- What happens when hardware fails
- Whether upgrades are included or charged
- Exit terms, and what you keep
Do not ask whether this is the best AI. Ask whether this is the right AI architecture for our data, our workload, our budget and our risk.
How to read the answers
A supplier who answers the technical list with adjectives has not measured the product. A supplier who answers the data list with a slogan has not drawn the diagram. A supplier who answers the security list with "it is on-premise" has not read OWASP.
And a supplier who concedes that the cloud is cheaper for light, spiky usage is telling you something true, which is a reasonable proxy for the rest of what they say.
With Bastion
What Bastion changes
Bastion is a private AI system delivered as one sealed appliance that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.
Bastion answers the four lists in fixed terms. A Reasoning node holds 35 concurrent users, at 8k tokens of context, at 30 tokens per second each, with an fp8 KV cache. Inference happens on the customer's premises, no outbound connection is required, the fee is one fixed monthly amount and a failed unit is replaced whole.
If your answers to the ten questions point to the cloud, that is a real result and worth having. If they point inwards, tell us what you want to run, how many people will use it and what data is involved, and we will tell you whether Bastion fits.
Questions this article answers
- What should an AI vendor due diligence checklist cover?
- Four areas: the technical specification including model, licence, memory, context and concurrency; the data flow including where inference happens and where prompts and logs are stored; the security architecture; and the commercial terms including exit.
- What is the first question to ask before buying AI?
- Not which model is smartest, but what the workload is: what the system will read, how the data is classified, whether it has to leave the organisation, and how many people will use it at the same moment.
- How do I check a capacity claim?
- Ask for four terms together: how many concurrent users, at what context length, at how many tokens per second each, and with which KV cache data type. A single number without those can be made to mean almost anything.
On the record
- 1
- 2
- 3
Read next
- Seven questions to answer before you decide where AI runsStart with the workload, not the hardware. What the system will read, whether the data has to leave, how steady the usage is, whether the internet is needed, how strong the model has to be, who operates it, and what happens when it is unavailable.Read the article
- What is private AI, and when does an organisation actually need it?Private AI is not a different kind of model. It is a different answer to one question: who controls the environment where the data is processed. What that answer changes, what it leaves untouched, and the five questions that settle it.Read the article