12 questions to ask before hiring an AI development company, and what a good answer sounds like

We lose some pitches and we win some, and after enough of both the pattern in how buyers choose is clear. The demo decides it. A vendor shows an agent handling a realistic case in a meeting room, the room is impressed, and the contract follows. Six months later the buyer discovers that the demo was the easy part, which is why so many pilots stall, and that the questions which would have predicted the stall were never asked.

This is the list of those questions. It is written for a business buying an AI system from an outside firm, whether a studio like ours, a consultancy, a systems integrator or a freelancer, and it is deliberately hard on vendors, including us. Each question comes with the answer that should worry you and the answer that should not.

What do you ask about the work itself?

1. What have you shipped that is still running in production a year later, and can I speak to the person who operates it?

The worrying answer is a list of logos and a case study written by the vendor. The reassuring answer is a name, a phone number and a system that has survived a model change, a traffic spike and a staff change. Ask specifically about a year, because a launch is a moment and production is a duration. Ask to speak to the operator rather than the sponsor; the sponsor remembers the launch and the operator remembers the Tuesday it went wrong.

2. How will we know if it works, before we go live?

The worrying answer is a demo, a pilot with a happy user, or the phrase "we'll iterate". The reassuring answer is an evaluation set: a few hundred real cases with the correct answers agreed by your people, a score the system has to reach before launch, and a plan to keep running that set after every change. We have written about why this comes first and how to build it. A vendor who cannot describe the evaluation set has not decided what "working" means and is hoping you will not ask.

3. What will this cost per task at our real volume, and what happens to that number when the model price changes?

The worrying answer is a project fee with no per-task figure, or a per-task figure with no assumptions behind it. The reassuring answer is a model of tokens per task, calls per task and retries, multiplied by current prices, with the sensitivity to a price move shown. Frontier prices have fallen all year and a vendor should be able to say how much of the saving reaches you. The budgeting piece sets out the maths a proposal should contain.

4. Which model will this run on, why, and what is the plan when that model is retired?

The worrying answer is a single model chosen because the vendor knows it, or a claim to be "model-agnostic" with no evidence of ever having switched. The reassuring answer names the model, gives the reason in terms of your task and your evaluation set, names the fallback that has been qualified against the same set, and describes the migration steps. Vendors retire models on twelve-month cycles; the deprecation playbook is the minimum a serious firm will have.

What do you ask about ownership and control?

5. Who owns the prompts, the evaluation set, the code and the data when the engagement ends?

The worrying answer is anything that keeps the prompts or the evaluation set with the vendor, or a licence to a platform of theirs that the system cannot run without. The reassuring answer is that all of it is yours, in your repository and your accounts, from the first week, and that the vendor's own tooling is either open or replaceable. The prompts and the evaluation set are the system; a vendor who keeps them has kept the system.

6. What will the agent be allowed to read, write and spend, and where is that enforced?

The worrying answer is a shrug, or "whatever it needs". The reassuring answer is a written scope per agent, a separate identity for each, and enforcement in a gateway and in the tools rather than in the prompt. The permissions piece is the checklist. A vendor who has not thought about this will build an agent with your admin credentials, because it is faster.

7. How will we see what it did, and for how long is that record kept?

The worrying answer is that the platform has logs. The reassuring answer is a run record with inputs, tool calls, decisions, approvals and cost, a dashboard your manager can read, alerts with an owner, and a retention period that your privacy obligations set. Observability is where the difference between a demo and a system becomes visible, and it is the first thing a vendor cuts when the budget is tight.

8. Where does our customers' personal data go, and what does the design do about it?

The worrying answer is a list of model providers and their privacy pages. The reassuring answer starts with redaction before the data leaves your boundary, names every processor and its location, sets retention on traces and indexes, and includes erasure that reaches the vector store and the memory. India's DPDP Rules become enforceable in May 2027 and the EU AI Act's transparency rules already apply; a vendor who has not read either should not be handling your customer data.

What do you ask about the vendor?

9. Who will do the work, and how much of it will be done by an AI?

The worrying answer is that the people in the pitch will not be the people on the project, or that the question about AI-written code causes discomfort. The reassuring answer names the engineers, says plainly that a large share of the code will be generated with tools like Claude Code and that every line is reviewed and tested by a named person, and explains what that does to the price. Any firm not using these tools in 2026 is charging you for typing. Any firm that will not say so has a review process it would rather you did not ask about.

10. What would you tell us not to build?

The worrying answer is silence, or enthusiasm for everything you mention. The reassuring answer is a specific item from your own list, with a reason: the volume is too low to pay back, the data does not exist, a rule-based system would do it, or an automation would beat an agent. A vendor who has never talked a client out of something has either had unusually good clients or no interest in the outcome.

11. What does your engagement look like at week two, and what do we hold in our hands then?

The worrying answer is a discovery phase with a deck at the end. The reassuring answer is a written scope with an evaluation set started, a ranked list of what to build first with payback estimates, and a decision point at which you can stop without owing more. Two weeks is enough to scope a deployment and a firm that needs longer to say what it will build is billing for its own learning.

12. What happens after launch, who answers at three in the morning, and what does that cost?

The worrying answer is a support email address. The reassuring answer is a named on-call arrangement, a monthly review of the four numbers on the dashboard, a change process for prompts and models that runs the evaluation set first, and a price for all of it stated separately from the build. Production AI is an operating cost, as we set out in February, and a vendor who prices only the build is pricing only the demo.

How do we answer these about ourselves?

Briefly, because a buyer's guide that ends in a pitch has wasted the reader's time. Our production systems and the people who run them are on the case studies page, and we will make introductions to the people who operate them on request. Every engagement starts with an evaluation set in your repository and it stays yours, with the prompts and the code. Proposals carry a per-task cost model with the price sensitivity shown. We name the model and the qualified fallback. Permissions are scoped and enforced in a gateway, run records and dashboards are in the first release, and redaction sits in front of every model call that carries personal data. Most of our code is written with AI tools and every line is reviewed by a named engineer. The AI strategy engagement is two weeks, ends with a ranked list and a stop point, and includes a list of what not to build.

And the cases where you should not hire us. If you want a chatbot on a website by Friday, a platform vendor's template will be cheaper and you should use it. If your volume is under a few hundred tasks a month, the maths rarely works for a custom system and we will say so in the first call. If you need a firm with a thousand engineers in one location, that is not us. And if what you want is the demo, every vendor can give you one, and you should ask them the other eleven questions first.

Frequently asked

How do you choose an AI development company? Ask about production rather than demos: systems still running a year after launch and the people who operate them, the evaluation set that defines success before go-live, the cost per task at your volume, and the plan for when the model is retired. Then ask about ownership of prompts, evaluation data and code, how agent permissions are enforced, what is logged and for how long, and where personal data goes. Finally ask who will do the work, what they would advise you not to build, what you hold at week two, and what post-launch support costs.

What should an AI vendor proposal include? A written scope with a named model and a qualified fallback, an evaluation set and the score the system must reach before launch, a per-task cost model with assumptions and price sensitivity, a permissions scope for every agent, a description of logging, dashboards and retention, a data protection design that names every processor and includes redaction and erasure, the names of the engineers, a two-week scoping phase with a stop point, and a separately priced operating arrangement for after launch.

What are warning signs when hiring an AI consultancy? A demo in place of an evaluation plan, a project fee with no per-task cost, a single model with no migration plan, prompts or evaluation data that stay with the vendor, agents built on shared or administrative credentials, logging described as a platform feature, no clear answer on where personal data goes, discomfort about AI-generated code, enthusiasm for everything on your list, a long discovery phase ending in slides, and support priced as an afterthought.

Related reading

• Why AI pilots fail: the production-readiness gap
• Evaluation is not a phase-two upgrade
• What production AI actually costs
• How to scope an AI deployment in two weeks
• Your vendor will retire the model you built on

Twelve questions take an hour to ask and will save most buyers a year. If you would like to put them to us, book a call and bring the list.