A vendor-neutral guide: ten evaluation criteria, six red flags and the questions to ask in the first meeting – useful even if you never work with us.
Last updated: July 2026
Judge an AI agency on four things: who actually builds your system, what working artifact you get and when, how pricing maps to outcomes, and who operates the system after go-live. Brand size predicts little – seniority in delivery predicts almost everything. The ten-point checklist below makes each of these testable in a first meeting.
The 10-point checklist for evaluating AI vendors
Every point can be verified in a first meeting. A good agency answers all ten without dodging.
The builders are in the room – not just the partners
Ask who writes the code – and whether that person joins the first meeting. If you only meet account managers before signing, that is also who you will meet after.
A working artifact within weeks, not months
A concrete plan within days, a runnable prototype within weeks. Month-long strategy phases without a running system are rarely justifiable on SME budgets.
References with verifiable numbers
Ask for reference projects with concrete, checkable metrics – automation rate, accuracy, time saved – instead of logo walls. Then ask how the numbers were measured.
Fixed scope or clear milestones
Price must map to outcomes: a fixed price per scope or clearly defined milestones. Pure time-and-materials with no milestones shifts the entire risk onto you.
GDPR and EU AI Act competence shown, not claimed
Have them explain the specifics: hosting location, access control, data flows, human oversight, audit logs. "We are compliant" without an architecture sketch is a marketing sentence.
They say no to at least one of your ideas
A serious partner has an opinion: at least one of your use cases is probably not worth building. A vendor that builds everything you order is optimizing revenue – not your outcome.
A post-launch operating plan exists
AI systems do not stay good on their own: data ages, models drift. Ask about monitoring, maintenance and further development – before signing, not after.
Tech-stack transparency and settled IP ownership
You should know which models and components are used, which of them are replaceable, and who owns the code and models at the end. Vendor lock-in hides in the fine print.
Knowledge transfer to your team is planned
Documentation, training, handover: your team must be able to understand and operate the system. An agency that hoards knowledge is selling you dependency as a service.
A small first engagement is offered
Serious vendors let you test them small: a workshop, a scoped pilot or a free feasibility assessment – instead of a six-month contract as the entry point.
What are the red flags with AI consultancies?
Six warning signs from real selection processes. None of them proves bad faith – but each deserves a follow-up question.
Guaranteed ROI promises
Nobody can guarantee the results of an AI project before seeing your data. Serious vendors work with projections and caveats – "guaranteed 40% savings" is a sales argument, not an analysis.
Model name-dropping instead of process questions
A vendor who talks about GPT, Claude or Llama before understanding your processes is selling technology, not solutions. Model choice is one of the last decisions in a project, not the first.
A strategy phase beyond eight weeks before anything runs
Analysis matters – but if nothing has been tested on real data after two months, you are financing slides. Feasibility shows up in a prototype, not in workshop minutes.
No engineer in the sales call
If nobody in the room could build the system, your technical questions go unanswered – and the estimates you receive come from people who will not have to deliver on them.
Time-and-materials with no milestones
Open-ended billing without defined interim results rewards duration instead of outcomes. Insist on clear milestones with acceptance criteria at minimum.
Demos that never touch your data
A polished demo on sample data proves nothing about your case. Insist on a test with a slice of your real data – however small.
To be fair: red flags are rarely signs of fraud – usually they are misaligned incentives, such as billing models that reward duration over outcomes. For the structural background on time-and-materials vs. fixed fees, and for normal market ranges, see: AI consulting vs. traditional consulting · What does AI cost? Typical market ranges
Which questions should we ask in the first meeting?
Seven questions to copy and paste. Pay less attention to the answers than to whether they are precise or evasive.
- Who exactly writes the code – and is that person in this meeting?
- Show us one project where you advised against AI.
- What working artifact will we see after four weeks?
- How was the headline metric of your best reference measured?
- What happens after go-live – who monitors, maintains and improves the system?
- Who owns the code, models and prompts at the end of the project?
- What is the smallest sensible first step – and what does it cost?
Want to test the questions live? We will answer all seven in a 30-minute call – no slides, no sales pitch. Book a conversation
Agency, freelancer, big consultancy or in-house – who fits when?
The honest answer depends on scope, maturity and politics. Four options compared – including the cases where AI Phase is deliberately not the right choice.
Freelancer
Fits narrow, well-specified single tasks with technical supervision already in-house. Risky as soon as architecture, compliance and operations enter the picture – everything then depends on one person.
AI Phase take: for a single, clearly specified script, a freelancer is often the cheaper choice than we are.
Big consultancy
Fits board-level transformations across many countries and corporate politics: when the organization itself has to move, the brand matters. Delivery, however, is often done by junior teams at senior day rates.
AI Phase take: for global multi-site rollouts with hundreds of stakeholders, larger partners fit better than a founder-led team.
Specialized AI agency
Fits when a system needs to be built, integrated and operated: build and operate from one team, senior delivery, short paths. This is exactly where AI Phase positions itself – from feasibility to production.
AI Phase take: this is our field – implementation and operations, integrated into systems such as M365, SAP or ticketing.
In-house team
Fits once AI is daily core business and several systems need permanent development. Building it earlier rarely pays: recruiting, utilization and retaining senior people are expensive.
AI Phase take: we build with knowledge transfer – the goal is that your team can take over when the time is right.
For companies with 50–500 employees, we have answered the build-or-buy question in more depth: AI for SMEs: use cases, readiness and limits · AI implementation by AI Phase · Who is behind AI Phase
How does AI Phase score on this checklist?
Transparency instead of neutrality theater: here is our own scoring – including the point where we are not the best choice.
The honest caveat: AI Phase is a founder-led team from the Munich TUM ecosystem, not a 500-person integrator. For global multi-site rollouts, or programs that mainly need political cover, larger partners are the better choice. For building, integrating and operating, we are. More about the team
Frequently asked questions about choosing an AI agency
Whether the people who scope your project also build it. Seniority in delivery predicts project outcomes better than brand name or company size.
Test us against the checklist
Send us your AI idea – you will get a scope sketch and an honest feasibility assessment within 24 hours. That is point 10, redeemed for free.