Evidence Before Enthusiasm: How to Run a Change Pilot That Survives Scale

Pilots occupy a comfortable place in change programmes. They are small enough to control, visible enough to celebrate and short enough to avoid some of the harder questions. That is precisely why they can mislead. A trial that succeeds because it has been sheltered from normal operational pressure is not proof that the model will work across an organisation; it is proof that the model can work when it is given exceptional conditions.

The distinction matters because pilot results are rarely treated as provisional. They are cited in board papers, used to unlock funding and presented as evidence that resistance has been overcome. If the pilot has not been designed to test the proposition, the organisation is not making a decision from evidence. It is making a decision from atmosphere.

The attention subsidy

Most pilots receive a subsidy that will not exist at scale. Senior managers attend the launch. Additional staff are assigned to resolve problems quickly. Reporting is more frequent. Exceptions are granted because the pilot is new. Teams tolerate inconvenience because they know the arrangement is temporary.

None of this is necessarily wrong. Early stages often need support. The error is failing to account for it. If the pilot succeeds only because a programme manager is manually clearing blockages each morning, the real operating model includes that programme manager. If users work around a missing integration because they know someone on the project team will fix the output, the process is not yet ready for routine operation.

A useful discipline is to list the subsidies at the start and then ask which will remain after rollout. Anything that disappears should be treated as a risk to the business case, not as a detail to be resolved later.

Begin with a hypothesis that can fail

A pilot should test a statement specific enough to be disproved. "The new process will improve service" is not a hypothesis; it is a hope with a project plan attached. A stronger formulation might be: "Within eight weeks, the revised triage process will reduce the average time from request to first substantive response by at least fifteen per cent, without increasing rework or reducing completion quality."

That statement identifies a measure, a time frame, a comparison and a set of guardrails. It also allows for an honest negative result. If the improvement does not appear, or appears only alongside a deterioration elsewhere, the organisation has learned something useful before committing to full implementation.

The hypothesis should be agreed with the people who will eventually make the scaling decision. A pilot designed by the delivery team and judged by a different group after the event invites disagreement about what the numbers mean. Pre-agreement does not guarantee consensus, but it narrows the scope for retrospective interpretation.

Choose a comparison that means something

Many pilots are assessed against their own starting position. That is a reasonable beginning, but it is rarely sufficient. Seasonal variation, staff turnover, a temporary lull in demand or a change in customer behaviour can all produce an apparent improvement that has little to do with the intervention.

Where practical, use a comparison group that continues under the existing process. It need not be a perfect scientific control, but it should be close enough in workload, geography or service type to make the comparison informative. If a comparison group is not feasible, use a longer baseline and more than one measure. Record what else changed during the trial. A pilot that cannot distinguish its own effect from a busy month is not a reliable basis for scale.

It is also worth being explicit about what is being compared. A pilot that uses new software, revised roles, additional training and a simplified form is not testing any one of those things. That may be acceptable if the question is whether the whole package works. It is not acceptable if the organisation later needs to know which element produced the result.

Agree the exit criteria before the launch

Pilots tend to acquire momentum. Once a team has invested effort and a senior sponsor has publicly supported the initiative, stopping or changing course becomes socially difficult. The remedy is to define exit criteria in advance and to state them in operational terms.

Exit criteria should cover three areas: performance, practicality and durability. Performance asks whether the expected benefit appeared. Practicality asks whether the process could be run by ordinary teams without exceptional support. Durability asks whether the change held when attention moved elsewhere, for example during a holiday period or a sudden increase in demand.

A pilot should also have a defined point at which it stops being a pilot. Open-ended trials blur into permanent exceptions. Set a review date, a decision owner and the options available: scale, scale with modification, extend with a specific new test, or stop. An extension should require a reason and a new question, not simply a preference to keep going.

Price the scale, not just the pilot

The pilot budget is rarely the rollout budget. At scale, training takes longer, support costs more, integrations are less forgiving and local variations multiply. A pilot that looks affordable because it used existing staff as a temporary support team may become unaffordable when those staff return to their substantive posts.

Before recommending scale, test the model against realistic conditions: normal staffing levels, standard service hours, ordinary approval routes and the actual volume of exceptions. Ask what happens when the most enthusiastic manager is not in the room. Ask how the process performs when demand is uneven rather than carefully managed for the trial.

Keep learning separate from lobbying

Those close to a pilot often become its advocates. That is human and not inherently problematic, but it creates a risk that interpretation outruns evidence. Separate the people accountable for delivery from those accountable for assessing the result, or at least make the distinction explicit in the review. Invite challenge from operators who were not involved in designing the trial. Publish the measures, including the inconvenient ones.

A pilot is a decision-support tool, not a ceremony. Its value lies in reducing uncertainty about whether a change will work under normal conditions. That requires the discipline to design a fair test, the honesty to record what supported it and the willingness to accept a result that does not match the original enthusiasm. Organisations that do this do not avoid failure; they avoid scaling it.