How to read a top-performer study before you roll it out

"Seven of our eight best branches do this." So what?

Picture this. A slide lands in your inbox. A consultant, or head office, or your own analyst has studied your best branches and found that seven of the eight top performers send customers a reminder text the day before an appointment. The last line says: roll it out to all forty.

Do not say yes yet, and do not say no. The slide describes what the winners have in common. It does not say whether the practice made them win. Those are different claims, and acting on the first as if it were the second costs you months, goodwill and staff time.

This guide is for the person who has been handed that finding and has to decide. It works for branches, locations, reps, franchisees, agents and partners alike. You get a worksheet to fill in for each claimed practice, a way to read the answers, and a small test to run before anything becomes policy. If you are the one running the study, start with our guide to running a top-performer study and come back here when the results arrive.

What a top-performer study can and cannot give you

The method is sound for what it is built for. Positive-deviance research looks closely at the best performers to generate ideas about what they do differently, then tests those ideas in a wider sample.¹ The first half produces hypotheses. The second half is what turns a hypothesis into a decision.

Check whether the study in front of you stops after the first half. That is fine, as long as everyone treats its output as a list of things worth testing, not a list of causes.

Four things make "most of our best do X" weaker than it sounds.

The practice may be common everywhere. If nearly every branch sends reminder texts, finding them at the top tells you nothing. The study has to say how many ordinary branches do it too.

Only the winners were studied. A study of successes can't see the branches that did the same thing and did badly. Observations taken from the survivors of a selection process are typically a biased sample.² Choosing cases by their outcome biases the conclusions you draw.³

"Top" may mean "lucky this year". If the eight branches were picked on one period's results, some of them will tend to drift back towards average next period without changing anything. This is regression to the mean: unusually large or small measurements tend to be followed by measurements closer to the average.⁴ Whatever you roll out in the meantime will get the credit.

Something else may explain it. The best branches may sit in the busiest town centres, be older, serve a different customer mix, or have kept the same manager for ten years. A practice that travels with those features will look like a cause.

The worksheet: read the finding

Fill in one sheet for each practice the study claims matters. If the study cannot supply an answer, write "unknown". Unknown is not zero, and it is not a pass.

READ THE FINDING: one sheet per claimed practice

1. The claim
   Who does what, and when: [one sentence]

2. Top performers
   Studied: [n].  Doing it: [x] of [n].
   Picked on: [one period / several periods]
   Of those doing it, also in the top group the period
   before: [x] of [number doing it]

3. Typical or weaker performers
   Asked: [n].  Doing it: [x] of [n].
   (Nobody asked = unknown. Those who do it and do
   not shine are your "did the same and failed".)

4. Minority practice?
   Done by [x] of [n] top performers.
   Fewer than half: [yes / no]
   If yes: who runs it, and what do they share? [..]

5. Rival explanations
   Location, size, tenure, customer mix, timing, staffing.
   Which of these do the sites that do it share? [list]
   Rival explanation still standing: [yes / no]

6. Not asked
   Cost and time. What happens when it fails. Who tried
   it and dropped it, and why. [list]

7. Decision: ROLL OUT / PILOT / PARK
   Reason, in one sentence: [..]

If item 3 or item 5 says unknown, get the answer before you read the sheet. A rival explanation is still standing when the sites that do the practice share one of the features in item 5, and you have no evidence from sites that share that feature but do not do the practice.

Then read the sheet with these rules, applied in this order.

• Park the practice if it is about as common among typical performers as among the top group, or more common among typical performers. You lose nothing by parking it. If it comes back with better evidence, read it again.
• Roll out only when it is clearly more common among the top group than among typical performers, the top group was picked on several periods or most of those doing it were also in the top group the period before (item 2), no rival explanation is still standing, it is cheap, and you can undo it easily. A change to a form label may qualify. A new daily routine for front-desk staff does not.
• Pilot everything else. Expect this to be your usual answer. It is a good one.

Count both groups, not just the winners

Item 3 is the line to check first. A study might report that seven of eight top branches do something. Without the other number, you cannot tell a differentiator from a habit.

The ratio matters more than the headline. "Seven of eight top branches, against eight of ten typical branches" is a practice that nearly everyone has. "Two of eight, against none of ten" is rare, and rare is interesting.

If the study did not ask typical performers, you can fix that cheaply: send three questions to a sample of them. Do you do this? Since when? What happened the last time you did it, or chose not to? Ten short replies from ordinary branches are a better use of the next week than another round of interviews with the best ones. Ask about the last time, not about what people believe in general. For more on getting past the first answer, see how to ask questions that get past the first answer.

Do not throw out the minority practice

A practice that only some of the top group does can still be the one that matters. Perhaps it is hard to do, or needs a particular kind of person, so most branches don't bother and a few do. The study that says "most of our best do X" can bury exactly that.

So read item 4 with as much care as item 2. If the practice is a minority one, ask who runs it and what those people have in common. If the answer is "both are in the two busiest town centres and both have a full-time receptionist", you have found a rival explanation and a reason to pilot, not a reason to ignore it.

If the study found several different tactics at different sites, that is a separate problem: working out how they fit together. This guide covers one practice at a time.

Here is an invented example to show the method

Everything below is made up. A group of forty opticians' branches commissions a study. The outcome is the share of customers who rebook within a year. The study ranks all forty on the last twelve months and interviews the top eight. Its finding: seven of the eight send a reminder text the day before an appointment, and two of the eight phone any customer who misses an appointment within the hour. The recommendation: roll out the reminder text to all forty.

The owner reads it with the worksheet. The sheet below has the same seven items as the blank version.

The recommended rollout falls at the first rule. Eight of ten ordinary branches already send the text, so it is unlikely to be what makes the top eight different. The owner parks it and saves the effort.

The call is the interesting one. Two of eight is a minority, and none of the ten typical branches do it. But both branches that do sit in the two busiest town centres and have a full-time receptionist. Maybe the call helps. Maybe a receptionist with spare time is what helps, and the call is a symptom. Only a test separates the two. Also, both call branches were in the top group the year before. That gives some comfort that they are not just having a lucky year. It does not settle the question.

Pilot it against comparison branches

A pilot with comparison sites tests the practice. A before-and-after comparison on its own does not. Without a comparison group you cannot say what would have happened anyway.⁵ Regression to the mean can make a before-and-after change look like a result, and a control group avoids that problem.⁶ More broadly, a control group removes a whole host of biases that complicate the evaluation of any change.⁷ Seasons and promotions can move your numbers too. When both branches in a pair share them, they can move the comparison branch as well.

Here is the blank plan. Agree every line before the pilot starts, and write it down.

PILOT PLAN

1. Sites and pairs
   [n] branches in [n/2] pairs, paired on [item 5 features].
   In each pair, a coin flip picks who tries it.

2. Comparison sites
   [n] branches keep the current way. Tell them nothing.
   A local event at one branch of a pair makes that pair
   not ahead.

3. Measure and guardrails
   Measure: [what is counted, over which window]
   Guardrail 1: [name, what is recorded, breach level]
   Guardrail 2: [name, what is recorded, breach level]

4. Length and checks
   Length: [weeks]. Every week: [calls made, guardrails].
   Results read once: [when]

5. Stop rule
   A site leaves when: [..]. Effect on its pair: [..]

6. Decision rule
   A pair is ahead when: [..]. Tie or lost pair: [..]
   Roll out: [x or more pairs]. Another round: [range].
   Park: [range]. Second round, read once: [roll out / park]
   Round two may change: [..]. Written down before it starts.

Here it is filled in for the opticians' example.

1. Sites and pairs. Twelve typical branches, chosen from those whose managers confirm they do not already call. Pair them up by the rival explanations on your sheet: size, whether they are in a town centre, and front-desk staffing. That makes six pairs. In each pair, flip a coin to pick which branch phones missed appointments. The other keeps its current way.
2. Comparison sites. The other six branches keep the current way. Do not tell them anything. Record any local event that hits only one branch of a pair, such as a road closure or a local promotion. A pair with such an event counts as not ahead in the decision rule.
3. Measure and guardrails. The measure is the missed-appointment rebooking share: missed appointments rebooked within fourteen days, divided by all missed appointments, counted the same way at all twelve branches. Count only misses from weeks one to ten, so that every one has its full fourteen days by the end of week twelve. The study's own outcome, rebooking within a year, cannot be read in twelve weeks, so the pilot measures the thing the call is meant to change. There are two guardrails. Desk minutes are the minutes the front desk spends on the calls each week, recorded at the six calling branches. Desk-wait complaints are the complaints logged about waiting at the front desk each week, recorded at all twelve. A guardrail is breached at a calling branch when its desk minutes top 180 in a week, two weeks running, or when it logs more desk-wait complaints than its pair, two weeks running.
4. Length and checks. Twelve weeks. Every week, check that the calls were made and look at both guardrails, because the stop rule works on two weeks running. Do not read the rebooking shares until the end of week twelve.
5. Stop rule. A branch leaves the trial if its calls stop for two weeks running, since there is then nothing left to test. A calling branch also leaves if a guardrail is breached there. A pair that loses either branch counts as not ahead.
6. Decision rule. At the end of week twelve, a pair is ahead when the calling branch has a higher missed-appointment rebooking share than its comparison branch. A tie is not ahead. Count the pairs that are ahead, out of six. Five or six: roll out in waves. Three or four: run one more twelve-week round, using the same pairs and the same rules, and read it once. Round two may change only the timing of the call or who makes it, and you write the change down before round two starts. A pair that lost a branch in the first round stays out and still counts as not ahead. In the second round, five or six pairs ahead means roll out, and anything else means park. Two or fewer in the first round: park it.

The rule is a way to commit before you see the data. It is not a statistical test. Its job is to stop you from talking yourself into a result afterwards.

What this will not tell you

A pilot that clears your decision rule gives you evidence that a practice is worth adopting. It does not give you a measured effect size you can put in a forecast. A pair can go the wrong way by chance, and twelve branches are not forty.

The top group's interviews are accounts of work, not observations of it. A manager can describe a routine the branch rarely follows. If the decision is expensive, send someone to watch before you pilot.

The worksheet also cannot rescue a bad selection. If the top group was chosen on records you do not trust, or on a measure that rewards gaming, no amount of careful reading will fix it. Ask how "top" was defined before you read anything else. For how to judge whether a finding rests on solid evidence, see what makes a customer insight evidence-backed.

Last, a practice that works in a pilot may need a particular person, tool or budget. Write down the conditions, not just the result.

Your next step

Take the next top-performer finding that reaches your desk and fill in one sheet. If item 3 or item 5 says unknown, ask the branches yourself, or send the study back with those questions, before you let anyone talk about rollout. If you are planning your own study, there is a use-case page on top-performer analysis.