Do users trust your AI feature? Test what they do with its output

Your team has built a feature that reads a customer's enquiry and fills in a booking summary. You ask six testers, "Do you trust it?" They say "mostly" and "it seems pretty good". You have learned nothing you can build on. None of them has seen the feature get a date wrong, and you have not seen what they do when it does.

Stop asking about trust. Put a prepared set of outputs in front of people, some right, some uncertain, some deliberately wrong, and watch what they do with each one. Accepting, checking, editing and rejecting are things you can see and count. Opinions about trust are not.¹, because self-reported claims are unreliable.

This guide gives you the whole kit: how to build the outputs, a 45-minute session plan, a coding sheet that turns behaviour into a verdict, interview questions, a debrief script, and the rules for deciding what to change. It fits any feature that hands someone an output to use: a summary, a draft reply, a recommendation, an extracted answer. The people can be customers, your own staff or partners.

Measure reliance, not trust

Trust is a feeling. Reliance is what a person does with the output: use it, check it, fix it or bin it. You are testing reliance.

The target is appropriate reliance.² describes it as relying on correct advice and turning down incorrect advice. That gives you two ways to fail, and you need to watch for both:

• Over-reliance: the person accepts an output that is wrong.
• Under-reliance: the person redoes, rewrites or throws away an output that was right.

Over-reliance ships errors. Under-reliance wastes the time the feature was built to save. A feature that people double-check every time has a problem, even if nobody is ever fooled.

Reliance is decided output by output.³: knowing when to trust the system and when to doubt it. So never average a session into one "trust score". Record each decision and read it against what you know about that output.

Prepare eight outputs for one low-stakes task

Pick one job your feature does and that a participant can do without training. Make it a practice job: sample data, nothing sent, no customer affected. Keep the stakes low and identical for everyone, so that stakes are not what explains the differences between people.

Then build eight outputs, each with its source material (the email, the document, the record) one click away. Write the answer key before any session.

Rules for building them:

1. Right outputs (4). Fully correct. They are your test of under-reliance, so do not make them look odd.
2. Uncertain outputs (2). Partly right, with the uncertainty signal your feature already shows: a note, a flagged field, a hedged line. If the feature shows nothing, add the signal you are considering and test that design. These outputs tell you whether anyone notices the signal.
3. Planted flaws (2). Wrong in one place, with no signal, in a way the source makes plain within a minute. Copy real mistakes from your logs or testing where you can. Absurd errors only test whether people are awake.
4. Fixed IDs, two orders. Never put a flaw first or last. Prepare two running orders and give each to three of your six participants. Keep the IDs the same in both so your sheets line up.

Eight outputs at about three minutes each fits a 25-minute task block. If your task is slower, use fewer outputs and keep the same mix of right, uncertain and flawed.

Run the 45-minute session

Book six people for 45 minutes each, in person or on a video call with screen sharing. Follow your organisation's consent and research rules, and record the screen if they allow it.

Give the task as⁴ and ask people to say their thoughts aloud as they work. Do not hint at what to look for. Here is a task card to read out, adapted to your feature:

You are helping us test a new feature. Everything today is practice data and nothing will be sent to anyone. [Eight booking summaries] are waiting for you. For each one, do whatever you would normally do to [get it ready to pass on to the kitchen]. Please tell me your thoughts out loud as you work. I will stay quiet and take notes. There are no wrong answers, and I will explain everything at the end.

Your job during the task is to watch, write and say very little.

• If they ask "is this right?", reply "What would you do if I weren't here?"
• Do not nod, hint or react when they accept or catch something.
• Note the action for each output, and what they looked at before deciding. Did they open the source? Scroll? Compare one field? Run the output through their own method?
• Do not explain what the note on an uncertain output means. How they read it is the finding.

Code what people did

Use one row per participant per output. That is eight rows per session and forty-eight across six sessions. Give each sheet a header line: participant number, running order (A or B), date.

Code the action with one of five labels:

• Accepted as is: moved on without looking at the source.
• Checked then accepted: looked at the source or tested the output, changed nothing.
• Edited: changed part of the output. Name the field in the "Edited field" column, and write "cosmetic" if the change only tidied wording or layout and altered no fact.
• Rejected: threw it away and did the job by hand.
• Gave up: left the task for this item.

Then read every row with this table. Apply it literally.

Tally the readings for each output in a table like the one in the worked example below: Appropriate, Over-reliance, Under-reliance, Gave up, Total. An uncertain output edited only elsewhere goes in the Under-reliance column, and it also counts toward the "signal is not working" rule alongside that output's Over-reliance readings.

Ignore a cosmetic-only edit to an uncertain output: read it as checked then accepted if they looked at the source or tested the output, and as accepted as is if they did not. A cosmetic edit to a right output is not a failure, but count it separately, because it shows wording or layout that people feel the need to fix. A flaw that was checked and still missed deserves a second look. The "what they checked" column tells you whether they looked in the wrong place.

Agree the decision rules before the first session

Write these down in advance, so the results cannot be argued into whatever the team already wanted. Counts are out of six participants.

• A planted flaw missed by three or more: change the feature so the flaw is easier to catch (show the source beside the output, highlight the field, add a check step). Test again with a new planted flaw.
• An uncertain output accepted as is, or edited only elsewhere, by three or more: the signal is not working. Redesign where and how it appears.
• An uncertain output rejected by three or more: the signal is causing alarm instead of a check. Reword it so it points at the one thing to look at.
• A right output rejected or edited for a fact by three or more: people are redoing work the feature should save. Find out in the interview what made it look wrong, and change that. If three or more made cosmetic edits, tidy the wording or layout instead.
• Anyone gave up: read that session in full, find the cause, then fix it or fold it into the change it belongs to.
• None of the above: keep the design and test the next task or the next version.

Ask about moments, then tell the truth

Hold the interview before the debrief, so the reveal does not colour people's explanations. Do not ask "do you trust it?" Ask about specific things you saw them do. When someone offers an opinion, ask⁴. For more on follow-ups, see how to ask questions that move past surface answers.

1. "Take me through [output]. What did you do first?"
2. "You checked [this] on that one and not on this one. What made the difference?"
3. "You changed [field] here. What looked wrong?"
4. "Was there one you wanted to check but didn't? What stopped you?"
5. "On [uncertain output], the feature showed [a note or flag]. What did you take it to mean?"
6. "Think of your normal week. Where would you check this, and where would you let it through?"

Then the debrief. Say which outputs were planted, describing each by what the participant saw, so they know which mistakes were yours and not theirs.⁵. Here is a script:

Thank you. Here is what we did. Some of the summaries you saw were right. Two were partly right and carried [a note or flag]. Two had a mistake that we put there on purpose: [describe each planted flaw by what the participant saw]. We wanted to see how people handle a feature that is sometimes wrong, so none of this is a test of you. Whatever you did is useful to us. Do you have any questions? If you would rather we did not use your session, tell us and we will remove it.

Debrief every participant, including the ones who caught everything.

A worked example

Here is an invented example to show the method.

A small catering firm builds a feature that reads an enquiry email and fills in a booking summary: date, headcount, dietary needs and venue. The six participants are the firm's own booking coordinators. The task is to get each summary ready for the kitchen, using practice enquiries.

Each coordinator makes eight decisions, so the study has 48. Two coordinators edited R3, and both changes were cosmetic. Reading every row with the table above gives this:

Now apply the decision rules in the order they are written.

• F1 was missed by five of six, which is three or more. The wrong date looked like a date, and most coordinators checked headcount and dietary needs because those felt harder. The team changes the feature to show the date from the email beside the filled field.
• F2 was missed by two, below the threshold. The team leaves the venue alone and plants a new venue flaw next round to see if the result holds.
• U1 was accepted as is by four of six, and nobody edited only elsewhere, so its Over-reliance column is the whole count and the signal is not working. The note sat below the form, out of the eye line. The team moves it onto the headcount field.
• U2 was rejected by two of six and accepted as is by one, both below three, so neither the "alarm" rule nor the signal rule fires. The team keeps the flag and reads the two interviews to see what the flag made them think.
• R3 had two cosmetic edits and R4 one rejection. Each is below three. The team notes the cosmetic edits.
• Two coordinators gave up, one on R2 and one on F1. The team reads both sessions in full. The R2 give-up was a screen that failed to load, a bug that gets fixed. The F1 give-up was someone who sensed that something was off and could not find it. The cause is the same as the F1 miss, so it is folded into the date change and needs no separate fix.

The verdict is not "people trust it" or "people do not". It is two concrete changes (the date beside its field, the note moved onto the headcount), one bug fix (the screen that failed to load), and a second round of six sessions with new outputs to see whether the changes worked.

What this will not tell you

Use six sessions to find design problems, not to count how many users share them. Read your tables as counts from six sessions, not as rates, and do not treat a clean round as proof that no problem exists.

Your planted flaws are your best guess at the mistakes the feature really makes, and the real ones will differ. A practice task with no consequences is not real work, and people may check more or less when real stakes are attached. A signal that gets noticed in a session still has to survive weeks of daily use. None of this measures how accurate the feature is; test that separately. Being watched and thinking aloud can change how much people check, so compare one round with the next only when the outputs are of similar difficulty and the sessions are run the same way.

Your next step

This week, write the answer key for your eight outputs and the decision rules above. Book six people for 45 minutes. Run the first session as a rehearsal on a colleague, fix the task card, then run the six.