A pilot ends remarkably often with the sentence “it went fine”. Nobody quite knows what went fine, and nobody can say what would have had to happen for it not to.
That is rarely because the technology is unclear. It is because the pilot was set up as a demonstration rather than a trial. A demonstration exists to show that something can work, and it almost always manages that. A trial exists to find out whether it does – and it therefore has to be able to fail.
Two weeks is enough. The most common mistake is spending three months on something that was settled after nine working days. What follows is the order that turns those two weeks into a decision rather than a recommendation.
1. Write down the decision before you write the conversation
The first sentence of the pilot document should not be about the agent. It should be about what you are going to do on the fifteenth of next month.
“We will decide whether the agent takes every incoming appointment booking outside opening hours from 1 October” is a decision. “We will test voice AI” is not.
Then write down the harder half: which result means no. One number, agreed before you have seen any data. If nobody can articulate what would make you call it off, you are not going to call it off – whatever the two weeks show.
2. Pick one call type, not one department
“Customer service” is not a call type. It is twenty.
Pick one with high volume, a short life and a clear end point: confirming an appointment, giving the status of a case, taking a cancellation, logging a fault. Something you can count before and after.
The objection arrives immediately: surely the easy calls are not worth automating. It is wrong in two ways. The easy ones are most of them, and they are the only ones that give a clean answer in two weeks. A pilot placed on the hard calls does not measure the agent. It measures the escalation.
It is also why a pilot should rarely wait for the back-office system to be connected: the call types that are suitable first usually need no integration at all.
3. Measure the baseline for five days before the agent goes live
Almost nobody does this, and it is why almost no pilot proves anything.
Five ordinary working days, agent switched off. Count how many people called about the chosen call type, how many got through, how long they waited, how many hung up before anyone picked up – and how many called back about the same thing within a day.
That last number is the important one, and the only one that is slightly awkward to extract. Do that work now. In two weeks it is the only line you will be arguing about.
4. Agree the way out before you agree the way in
What decides whether a pilot feels successful is not the opening line. It is what happens when the agent cannot get any further.
Settle three things before the first call: what triggers a transfer, what the human is told when the call arrives, and what happens when nobody picks up. Escalation is the part that is most expensive to do halfway, and in a pilot it is also the part everybody notices.
Set the threshold low for the first few days. An agent that transfers too early hands you a list of what it could not do. An agent that hangs on too long hands you irritated customers and no list.
