The AI Vendor Evaluation Checklist: 12 Questions Before You Sign Anything
Most AI vendor evaluations go like this: vendor demo, impressive results, reference calls that are obviously curated, pricing negotiation, signature. Three months into the contract, you discover the thing you needed most wasn't in scope.
The problem isn't the vendor. It's the questions you asked — or didn't ask. Here's the checklist I use before signing anything.
Before the Demo
1. What problem are we actually solving?
Write it down in one sentence before the demo. If you can't, you're not ready to evaluate anything. Vendors are good at creating new problems for you to solve with their product. Know your actual problem first.
2. What does success look like in 90 days?
Specific and measurable. Not "improved efficiency" — "the weekly reporting process takes under two hours instead of eight." If you can't define success, you can't evaluate whether a tool achieves it.
3. Who will actually use this daily?
Talk to those people before the vendor does. Find out what they need, what they won't tolerate, and what they're already using. A tool your team won't adopt is worse than no tool.
During the Demo
4. Can I see it on my data?
Generic demos with vendor-supplied data are theater. Request a sandbox with your actual data or a representative sample. If the vendor won't allow it, ask why. The answer is informative.
5. What happens when it's wrong?
Every AI system makes mistakes. What matters is how those mistakes surface and how they're corrected. Ask to see a failure example. How was it caught? What was the recovery path? Vendors who can't answer this confidently haven't thought about production.
6. What does the human handoff look like?
For any workflow with consequential outputs, there's a point where a human needs to review or override. Is that built in? Is it intuitive? How does the system flag low-confidence outputs? This question separates tools built for demos from tools built for real work.
On Integration and Data
7. What does it connect to natively vs. what requires custom work?
Get the integration map in writing. "It integrates with Salesforce" can mean a 10-minute OAuth setup or a three-month professional services engagement. Know which one before you sign.
8. Where does my data go, and who can see it?
Is your data used to train their models? Is it stored in their infrastructure or yours? Can their team access it for support purposes? These questions matter more as you move from pilot to production. Read the DPA, not just the terms of service summary.
9. What's the migration path if we leave?
Nobody signs a contract planning to leave. But the ease of exit tells you a lot about how the vendor thinks about the relationship. Can you export your data in a standard format? How long does it take? What do you lose if you cancel?
On the Vendor Relationship
10. Who's our day-to-day contact after the sale?
The account executive who closed the deal often disappears after signing. Find out who handles implementation, who handles support, and what the escalation path is. Ask to meet those people before you sign.
11. What's on the roadmap, and what's actually shipping?
Roadmaps are marketing until they're in the product. Ask for the last three features that were publicly committed to. When were they promised? When did they ship? That track record is more informative than any future promise.
12. What does the contract look like if our needs grow?
Understand the pricing model at 2x your current usage and at 5x. Some tools are priced to win pilots and penalize scale. Know what you're committing to before you're locked in.
The One Question That Reveals Everything
Ask the vendor: "Tell me about a customer who didn't succeed with your product and why."
Vendors who can answer this honestly, with specifics, are vendors who understand their own product. Vendors who can't — who pivot to another case study or get vague — are telling you something important about how they handle the hard conversations you'll eventually need to have.
Evaluate the answer as much as the product.