AI procurement goes wrong at the demo. The product does something impressive, the room relaxes, and the questions that decide whether you can defend the decision in twelve months never get asked. What follows is deliberately generic: twenty questions that work against any vendor, in any category, with no product names on either side.
How to use the list
Send it in writing before the demo rather than after, and ask for written answers. The point is not to catch anybody out. It is to create a document you keep with the contract, because the next time an enterprise client or an investor asks how you assess AI vendors, that file is the answer.
Calibrate depth to consequence. A transcription tool used on internal meetings does not deserve the same scrutiny as a system that touches client records or shapes a decision about a person. And judge the manner as well as the content: a vendor who answers twelve well and says "I do not know, I will find out and put it in writing" to the rest is a better counterparty than one who answers all twenty smoothly and none of them specifically.
Data handling and residency (1 to 4)
Where is our data stored and processed, country by country?
A good answer names countries and regions, distinguishes storage from processing, and identifies the failover location as well as the primary. "Globally distributed" and "in the cloud" are not answers, and you cannot assess a transfer you cannot locate.
Which sub-processors touch our data, and how do we learn when that changes?
A good answer is a maintained, published list with a notification commitment before additions take effect. Most AI products are assembled from several suppliers, so a vendor claiming none is usually a vendor who has not looked.
How long is our data retained, and can we set that to zero?
A good answer gives a configurable window, states the default, and explains how deletion reaches backups and on what timetable. Vagueness about backups is the most common gap between a deletion promise and a deletion capability.
What happens to prompts, uploads and outputs?
A good answer separates transient processing from logging, says plainly whether any human at the vendor can read customer content, and describes the controls around that access. Silence on human review is the answer you should assume the worst about.
Model and training data (5 to 8)
Do you train on our data, in any form, under the plan we are buying?
A good answer is an unqualified no, located in the contract rather than on a marketing page, and covering the vendor and every sub-processor. Watch for answers that are true of the enterprise tier while you are being quoted a different one.
Whose models sit underneath, and what happens when they change?
A good answer names the underlying providers and commits to notice before a material model change. Many vendors are a layer over someone else's model, which is fine, but it needs to be said out loud so you know who your real counterparty is.
What do you know about the training data behind those models?
A good answer is honest about the limits of what a reseller can know, and compensates by describing the evaluation the vendor ran themselves. An answer claiming full visibility into a third-party foundation model deserves a follow-up question.
What are the documented failure modes for our specific use case?
A good answer is specific and slightly uncomfortable: fabricated citations, weaker performance on Arabic-language documents, poor handling of scanned or handwritten material, degradation on long inputs. A vendor with no failure modes has either not measured or will not say.
Security (9 to 12)
Which certifications do you hold, and may we see the report rather than the badge?
A good answer offers the current report under a confidentiality agreement and states the scope and the audit period without being pushed. Certification scope frequently excludes the newest product, which is usually the one you are buying.
Who at your company can reach our data, and under what process?
A good answer describes role-based access, logged and time-limited support access, and a requirement for customer consent before anyone views content. Ask for the log to be available to you, not merely to exist.
Is data encrypted in transit and at rest, and who holds the keys?
A good answer is a straight yes with clarity on key custody and whether customer-managed keys are offered. This question mostly tests how the vendor handles a routine technical question, which is informative in itself.
What is your breach notification commitment, in hours?
A good answer is a number written into the contract with a named contact route, and a vendor comfortable describing how a previous incident was handled. Notification windows matter because your own obligations start running from the moment you are told.
Performance and monitoring (13 to 16)
How will we measure whether this works, and against what baseline?
A good answer proposes the measurement before the pilot starts and accepts a baseline drawn from your current process. Vendors who resist a baseline are protecting a comparison they expect to lose.
What can we see ourselves about usage?
A good answer is an administrative view showing which accounts are active, what volumes are moving and where. A quarterly summary prepared by the vendor is not oversight, because you cannot use it to find the department nobody told you about.
How do you handle model updates and regressions?
A good answer covers versioning, release notes, the ability to delay or pin a version for a period, and a rollback route. Silent updates are the mechanism by which a validated workflow stops being validated.
What does support look like when the output is wrong rather than absent?
A good answer describes a quality route distinct from the availability commitment, with a person on the end of it and an expectation of what happens next. Most service agreements cover downtime and say nothing at all about being confidently incorrect.
Contract and exit (17 to 20)
Where do liability and indemnity sit, particularly for intellectual property in generated output?
A good answer is specific to the plan being sold and points at the clause. Indemnities are frequently tiered, and the protection described in a public announcement often belongs to a tier above the one in your quote.
Can we get our data out, in what format, and how quickly?
A good answer is a self-service export in an open format that you can test during the pilot, plus a stated timetable for a full extract on termination. Test it during the pilot, because an export feature that has never been used is a plan rather than a capability.
What happens to our data after termination?
A good answer gives a defined deletion window, offers written confirmation on request, and is explicit about how long copies persist in backups. This is the question your own client will eventually ask you, so you want the answer on file.
How do renewal and price changes work?
A good answer covers the notice period, any cap or stated mechanism for increases, and what happens commercially if usage grows faster than expected. Consumption-priced AI products can move a long way between one budget cycle and the next.
Reading the answers
Four patterns are worth more than the individual answers. First, an answer about the company rather than about your instance: certifications, headcount and customer logos in place of a description of what happens to your data. Second, adjectives as substance, where "enterprise-grade" and "bank-level" arrive instead of a mechanism. Third, a reseller who cannot see through to the model underneath and does not say so. Fourth, and most reliably, a refusal to put in writing something that was said comfortably on a call.
Then close the loop, because a good answer that never reaches the contract is a pleasant conversation. Data residency, the no-training commitment, the notification window, the deletion window and the export obligation belong in the agreement or in an annexe to it. File the completed set with the contract and the vendor assessment note, and revisit it at renewal against what the product has since become. For where this sits in the wider control set, see the vendor section of the governance checklist and the approval route in your acceptable-use policy.
If a vendor decision is already in front of you and you want the answers read by someone who does this regularly, message Hayat Advisory on WhatsApp with the category of tool and what data it would touch.