Two phases, five steps each, one deliverable per step.

I use the TRL scale, the nine technology readiness levels, to place a technology and avoid treating a research topic as an available product. Many projects fail because they were premature, not because they were badly run.

Three kinds of maturity are constantly conflated in procurement files, and they need separating: that of the underlying technology, that of the product wrapped around it, and that of your own organisation faced with this use. A mature technology inside a young product sold to an organisation with neither the data nor the roles to exploit it does not add up to a mature project.

This step is short. Its purpose is to rule out early whatever has no business being in the comparison, and to align everyone on the nature of what is being examined.

The set is built with your domain experts, from real cases and above all from the hard and ambiguous ones, never from the examples supplied by the vendor. This is the step that determines the value of everything else: whoever builds the set decides who wins.

A useful set mixes four kinds of case. Everyday cases, in the proportion in which they actually arrive, because a solution that excels on the exotic and is mediocre on the ordinary is a poor solution. Edge cases, sitting on the boundary between two rules. Trap cases, which look like an everyday case and are not. And out of scope cases, where the right answer is to say that the system does not know: a solution that never abstains is a risk, not a help.

Ground truth is established by your experts before they have seen a single answer from a single solution. It is a tedious constraint and it is not negotiable: a reference written after the fact always drifts, unconsciously, towards whichever answer sounded most convincing.

Same inputs, same criteria, several runs. The criteria cover answer quality, functional coverage, robustness on edge cases, cost in use and compliance, to which I add exit cost whenever the commitment is long.

Several runs, because a generative system is not deterministic: the same question asked twice does not produce exactly the same answer. A single run measures luck as much as quality. The spread between two runs of the same solution is itself valuable information: if that spread is of the same order as the gap between two solutions, then the comparison says nothing and the set needs widening.

Scoring is done blind wherever possible, meaning the assessor does not know which solution produced the answer being scored. It is a simple precaution, and it changes the results more often than people expect.

Every score points back to a specific case in the set. A score resting on no case is not a score, it is an opinion.

A single campaign measures a moment, on one version. Several campaigns measure a trajectory, and it is the trajectory that supports a multi-year commitment.

In practice, you rerun the same set at a set interval, on the successive versions of the shortlisted solutions. Three things then appear that a single campaign never shows. The direction and speed of progress: a vendor steadily gaining ground on your cases is often worth more than a vendor who led on the first pass and has stalled since. Regressions: a new version that improves the average may have broken precisely the edge case that matters to you. And the stability of the vendor itself, readable in how it handles the gaps you report.

On fast moving technologies, the gap between two campaigns a few months apart is frequently larger than the gap between two competitors at a given moment. That alone is reason enough never to decide on a snapshot.

A comparative scorecard, a reasoned recommendation and the conditions for success. Not a test report nobody reads.

The test of the document is simple: a committee must be able to decide on that document alone, with no preparatory meeting to decode it. So it states the recommendation first, then the conditions it rests on, and puts the detailed results in an annex for those who want to check.

It also states what was not measured. Every evaluation has blind spots. Naming them is better than letting the reader assume the scope was complete.

Three outcomes are possible at this point, and only one of them leads to the next phase: select a solution and put it to the test in house, widen the search because none is convincing, or walk away because the need does not justify the cost. Walking away here costs the price of an evaluation. Walking away after deployment costs something else entirely.

A short note setting out the stake, the scope, and the decision indicators fixed from the start. You move on when the stake is stated in a verifiable way, not merely an interesting one.

The difference fits in one sentence. "Improve case handling with AI" is interesting and cannot be verified. "Reduce first response time on category B cases, measured over the past month, without degrading the rework rate" can be verified, and makes it possible to say at the end whether this was won or lost.

Indicators are fixed now, not at the end. That is the only protection against the bias of discovering, after the fact, that the real value of the project lay elsewhere, precisely where the results happen to be good.

The framing also names what is out of scope. A pilot whose edges are not drawn grows until it can no longer demonstrate anything.

A shared assessment of the organisation, the data, the technology and the regulatory constraints, then the options and their trade-offs. You move on when the options are down to those that deserve a real test.

Four dimensions, and technology is rarely the blocker. The organisation: who does what today, who will have to work differently, and who has an interest in nothing changing. The data: does it exist, is it accessible, in what state, and at what cost to make it usable. The technology: what integration with the existing estate actually requires, and what it costs. Regulatory constraints: what the nature of the processing imposes, and what your sector adds on top.

The important word is shared. A diagnosis the business disputes is useless, even when it is correct. It is built with the people whose work it describes.

By the end, the options are few and each carries an explicit trade-off. An option with no downside is an option that has not been properly examined.

A precise hypothesis, a limited scope, a fixed deadline, an explicit test protocol. You move on when observable results exist, favourable or not.

Targeted is the word carrying the weight. A pilot set up to demonstrate that "it works" demonstrates nothing, because it has no failure condition. A pilot testing a hypothesis stated in advance, on a scope where measurement is possible, produces a result either way.

The deadline is fixed at the outset, and it is short. A pilot that drags on becomes a de facto service: users get used to it, stopping it becomes politically expensive, and the decision then makes itself without ever having been taken.

The protocol states who uses what, for how long, what is measured and how. It also states what happens in the event of an incident, because these are real users on real cases.

The scorecard is completed on feasibility, real business impact, maturity, regulatory and lock-in risk, then held against the framing criteria, without rewriting them after the fact.

Real business impact is what can be observed, not what can be inferred. Time saved per case only becomes a gain on two conditions: that the freed time goes to something else useful, and that the workload has not simply moved elsewhere. Both are checked with the people who do the work.

This is also where the internal cost of the pilot is measured, in team time and in management attention. That cost is real and it weighs on the industrialisation decision, since it will recur at greater scale.

Holding the results against the framing criteria is the moment of truth for the phase. You take the framing note, read it exactly as it was written, and answer point by point.

A reasoned note, worth just as much when it concludes that the work should stop. A documented stop is a management decision, not a failure to be hidden: that is what separates a consulting engagement from a showcase of successes.

The three outcomes carry equal weight and are stated as such from the framing. Industrialise: the note then says on what conditions, with what monitoring, and what will need rechecking in six months. Adjust the scope: the value is there, but not where it was being looked for, or not yet at this scale. Stop: the note says why, what the exercise taught, and on what conditions the subject would deserve reopening.

That last part is the one people skip, and it is the most useful. A technology ruled out today for lack of maturity may become relevant in eighteen months. Writing down the reopening conditions saves having to reinvestigate everything, and equally stops the subject coming back too soon.

Whatever the outcome, what has been built stays with you: the evaluation set, the scorecard, the pilot protocol and the indicators. That is the real asset of the exercise, the one that serves the next decision.