AI vendor procurement is where organizations make binding commitments that are expensive to unwind. A three-year contract with a model provider locks you into their pricing, their rate limits, their model roadmap, and their data handling practices. A managed AI platform contract locks you into their deployment model, their integration patterns, and their operational assumptions. A data vendor contract locks you into their schema, their update frequency, and their licensing terms.

The procurement process for AI vendors differs from standard software procurement in three ways. First, AI vendors handle your data in ways that have regulatory and privacy implications that standard vendor questionnaires do not cover. Second, AI vendor pricing models are complex and often opaque — per-token pricing, tiered compute pricing, and usage-based licensing make cost prediction difficult. Third, AI vendor lock-in is harder to escape than traditional SaaS lock-in because your data and models may be stored in proprietary formats.

This checklist covers the evaluation criteria that matter for AI vendor selection. Use it as a structured alternative to the ad-hoc evaluation process that most teams follow.

Prerequisites

You need a written requirements document for the vendor relationship. What specific capabilities do you need? What are your volume requirements? What is your budget? What is your timeline? Without written requirements, vendor evaluation becomes a feature-comparison exercise that does not answer the question: does this vendor meet our needs?

You need a cross-functional evaluation team. The team should include engineering (can the product integrate with our systems?), security (does the vendor meet our security requirements?), legal (are the contract terms acceptable?), and finance (does the pricing model fit our budget model?). A single-engineer evaluation misses critical dimensions.

You need at least two candidate vendors. Single-vendor evaluation produces no negotiating leverage and no comparison baseline. If the market has fewer than two viable candidates, document why and adjust the evaluation to focus on risk mitigation for the single-vendor dependency.

Technical evaluation

Integration requirements

API compatibility. Does the vendor’s API support your required interaction patterns? Check for: synchronous and asynchronous invocation, streaming responses, batch processing, and webhook callbacks. Test the API with your actual code, not just the vendor’s documentation examples.

SDK and client library quality. Does the vendor provide client libraries for your primary language? Are they maintained, documented, and tested? Poor SDK quality is an early indicator of poor engineering practices.

Data format support. Can the vendor accept and return data in your required formats? If you need structured output (JSON schema conformance), test it. If you need multimodal input (text plus images), test it. Do not assume format support from documentation.

Latency and throughput. Test the vendor’s latency and throughput at your expected volume. Run a load test at 2x your expected peak volume for at least one hour. Measure p50, p95, and p99 latency. Measure error rates under load. A vendor that performs well at 100 requests per second may degrade at 1,000.

Model or algorithm evaluation

Accuracy on your data. Do not accept the vendor’s published benchmarks. Run the vendor’s model or algorithm against your data, your use case, and your evaluation criteria. The vendor’s benchmark was optimized to look good on a specific dataset. Your data is different.

Performance consistency. Run the same evaluation multiple times over a week. AI model performance can vary due to model updates, infrastructure changes, or load variations. A vendor that scores 92% accuracy on Monday and 87% on Wednesday has a consistency problem.

Version management. Does the vendor notify you before model updates? Can you pin to a specific model version? Can you test a new version before it affects your production traffic? Silent model updates are one of the most common sources of production AI incidents.

Customization options. Can you fine-tune, customize, or configure the model for your use case? What are the customization options and their costs? A vendor that offers no customization is selling a general-purpose tool. If your use case is specialized, you may need customization.

Security evaluation

Data handling. Where is your data stored? How long is it retained? Is it used for model training? These questions have legal, privacy, and competitive implications. Get written answers, not verbal assurances.

Specifically: does the vendor use your data to improve their models? If yes, can you opt out? What happens to your data if you terminate the contract? Can you request deletion of all your data, including any derived model improvements?

Encryption. Is data encrypted at rest and in transit? What encryption standards are used? Who holds the encryption keys? For sensitive data, customer-managed encryption keys are a requirement, not a nice-to-have.

Access controls. Who at the vendor organization can access your data? What access logging exists? Can you audit the access logs? A vendor that cannot answer these questions does not have adequate access controls.

Penetration testing and security audits. Has the vendor undergone a third-party security audit? Do they have SOC 2 Type II, ISO 27001, or equivalent certifications? Request the audit report, not just the certification. The report reveals findings that the certification status hides.

Incident response. What is the vendor’s incident response process? How quickly will they notify you of a security incident affecting your data? What remediation steps will they take? A vendor with no documented incident response process will handle your security incident ad-hoc.

Compliance evaluation

Regulatory compliance. Does the vendor comply with the regulations that apply to your use case? GDPR, HIPAA, SOC 2, PCI-DSS, FedRAMP — the specific regulations depend on your industry and data types. Do not assume compliance. Request evidence.

Data residency. Where is your data physically stored and processed? If your regulatory requirements mandate data residency in a specific jurisdiction, verify that the vendor can meet this requirement. Some vendors process data across multiple regions and cannot guarantee residency.

Right to audit. Does the contract give you the right to audit the vendor’s handling of your data? This is critical for regulated industries. A vendor that refuses audit rights may be hiding non-compliance.

AI-specific regulations. Does the vendor comply with emerging AI regulations? EU AI Act requirements, algorithmic transparency mandates, and bias testing obligations affect vendors that provide AI models or AI-powered services. Assess the vendor’s readiness for regulations that are not yet in force but are on the horizon.

Commercial evaluation

Pricing model. Understand exactly how you will be charged. Per-token pricing for model inference. Per-query pricing for search services. Per-seat pricing for platform access. Compute-hour pricing for training. Map the pricing model to your usage pattern and calculate the monthly cost at your expected volume, at 2x your expected volume, and at 0.5x your expected volume.

Cost predictability. Can you predict your monthly cost within 20%? If the pricing model has too many variables (variable token counts, variable compute requirements, variable feature usage), you cannot budget reliably. Negotiate a pricing structure that gives you cost predictability: committed-use discounts, spending caps, or fixed-tier pricing.

Contract terms. Review the contract for: data ownership (you own your data, period), model ownership (who owns fine-tuned models or custom configurations), liability (who is liable when the AI system produces incorrect output that causes harm), and termination rights (can you exit the contract if the vendor fails to meet SLA?).

Vendor viability. Is the vendor financially stable? How long have they been in business? How many customers do they have at your scale? A vendor that goes out of business leaves you with a migration crisis. For early-stage vendors, require source code escrow or data portability guarantees.

Exit strategy. Before signing the contract, verify that you can extract your data and models in a standard, portable format. Test the export process. A vendor that makes export difficult is a vendor that is building lock-in. If the export process is incomplete, undocumented, or lossy, factor the migration cost into the total cost of the relationship.

Operational evaluation

SLA terms. What uptime guarantee does the vendor offer? What are the penalties for SLA violations? Read the SLA carefully — some vendors exclude planned maintenance, degraded performance, and specific error types from their uptime calculation.

Support responsiveness. Test the vendor’s support before signing. Submit a technical question through their support channel. Measure the response time and quality. A vendor that takes three days to answer a pre-sales question will take longer during an active incident.

Documentation quality. Read the vendor’s documentation. Is it complete, accurate, and current? Poor documentation predicts poor operational experience. If the documentation is incomplete, your team will spend engineering time discovering undocumented behaviors.

Monitoring and observability. Does the vendor provide dashboards, metrics, and logs that let you monitor your usage, performance, and costs? Can you export these metrics to your existing monitoring stack? A vendor that is a black box is a vendor you cannot troubleshoot.

Scoring and selection

Create a weighted scorecard from the evaluation categories. Assign weights based on your priorities. For a regulated healthcare company, security and compliance might carry 40% of the weight. For an early-stage startup, technical evaluation and commercial terms might carry 60%.

Score each vendor on each criterion using a consistent scale (1-5). Calculate the weighted total. The highest-scoring vendor is not automatically the winner — the scorecard surfaces the tradeoffs, and the final decision is a judgment call that considers the scores alongside qualitative factors.

Document the selection rationale. When the contract comes up for renewal in two years, you will want to remember why you chose this vendor and whether the reasons still hold.

Common failure modes

Evaluating features instead of requirements. A vendor with an impressive feature set that does not meet your specific requirements is worse than a vendor with a modest feature set that meets your requirements exactly. Start from requirements, not features.

Skipping the exit strategy. Every vendor relationship ends. Either you outgrow the vendor, the vendor’s pricing changes, or the vendor goes out of business. If you have not verified data portability before signing, you will discover the lock-in when you try to leave.

Accepting vendor benchmarks as your benchmarks. Vendor benchmarks are marketing material. They use optimized configurations on curated datasets. Your benchmark must use your data, your query patterns, and your quality criteria.

Single-person evaluation. An engineer evaluating a vendor alone will assess technical fit but miss security risks, compliance gaps, and commercial traps. Cross-functional evaluation catches problems that single-function evaluation misses.

Next step

If you are currently evaluating an AI vendor, pull out this checklist and work through the technical evaluation section this week. Run the vendor’s API against your actual data and query patterns. The technical evaluation takes one to three days and often reveals deal-breaking gaps that documentation and demos do not show.