Moderated vs Unmoderated Usability Testing: Choose and Plan the Right Study

Choose around the uncertainty you need to resolve

Moderated testing puts a researcher in the session with the participant. Unmoderated testing gives the participant prepared instructions to follow without a researcher attending live. Either format can produce observations and quantitative measures; the presence of a moderator does not by itself decide whether a study is qualitative or quantitative.

The useful decision is whether your question can survive a session in which nobody can clarify a misunderstanding, investigate an unexpected action, or recover a broken prototype. NN/g's unmoderated-testing guide stresses advance preparation and piloting. GOV.UK's moderated-testing guide explains how tasks, observation, and follow-up questions work together. Neither makes one format the universal choice.

Consider a fictional community class website. People have already booked a pottery class and now need to change the date. The team wants to learn whether they understand a pending change request and whether they can complete the rescheduling flow. The same topic contains two different uncertainties: interpretation of the status, and completion of a well-defined action. The example below shows how those uncertainties lead to different study designs.

Compare the same task, not two different levels of ambition

Start by fixing the audience, device, starting state, and intended outcome. Otherwise, a rich moderated investigation can look slower than an unmoderated study simply because it tries to answer more questions. A comparison of ten-minute recordings with hour-long interviews is not a clean comparison of facilitation alone.

For the pottery example, participants are adults who have booked a paid community class online within the past year. Use a test account with a Tuesday booking and a Thursday place available. The target is a confirmed Thursday booking, with no remaining Tuesday booking. A request marked Pending does not meet that target. This operational definition prevents a platform's generic 'task completed' button from becoming the only evidence.

There is also a separate status question: after submitting a change request that needs staff approval, what does the participant believe they are booked into? That branch ends at Pending by design. Do not score it as failure to reach a confirmation that the test system cannot provide. The two branches need different outcomes even though both concern changing a class.

For the researcher, the fictional pending-approval rule is explicit: the Tuesday booking remains valid until staff approve the change; Thursday is not yet confirmed. Approval replaces Tuesday with Thursday, while refusal leaves Tuesday unchanged. The status-interpretation task succeeds when the participant explains the current valid booking, the lack of a confirmed Thursday place, and the need to await the staff decision and check the resulting status. Equivalent explanations count; exact wording is not required. Keep this answer key out of the participant-facing task prompt.

Build a moderated session that does not become a demonstration

The following discussion guide is a complete small session for the pending-approval branch. It includes an introduction, a neutral task, observation, follow-up, and closure. Arrange consent and recording choices before the task; the sample introduction is not a substitute for the information your actual study requires.

Allow the participant to interpret the interface before asking about the status. Asking 'Do you understand that this is still pending?' supplies the intended answer. Asking 'What would you expect to happen next?' lets their explanation emerge. If you later need to teach the state so you can inspect another part of the flow, record where that intervention occurred.

Assistance can be useful for learning about later screens. It changes what you can conclude about independent completion. Keep the unassisted observation and the assisted exploration separate in notes rather than treating the whole session as either a clean success or useless data.

Make the unmoderated task stand on its own

Use the available-place branch for the unmoderated example. The test account must start with a Tuesday booking every time, and submitting a change must lead to a real test confirmation. If several participants share a mutable account, the first person can change the starting state for everyone else. Reset or isolate accounts before launch.

A task that says 'change your booking' leaves important context unstated. The complete instructions below specify the goal, safe test conditions, stopping point, and an option to stop without pretending to succeed. They do not reveal the names or order of the interface controls.

Review the recording or event evidence against the final booking state. A participant may click the study platform's completion button after reaching a request screen. Conversely, someone may finish correctly and forget to press that button. A self-report is useful evidence of what they believed; it is not interchangeable with confirmation of the system outcome.

Pilot the whole research path, including the result you receive

A pilot is not complete when the link opens. Run the exact invitation-to-recording path on a participant-like device, then inspect the evidence the researcher receives. A readable screen in the participant's browser can become an unreadable cropped recording. A task can reset on desktop and remain stale on a phone.

For this example, the team should deliberately try a successful change, an abandoned attempt, and a broken-account condition. These are checks of the research setup, not extra participant results. Confirm that each can be recognized in the exported material and that the study does not force every attempt into a success category.

The table is a fictional pilot log. Its value is the connection between a defect and a concrete repair. Do not discard inconvenient participant behavior merely because it differs from the path you expected. A valid attempt that fails belongs in the research findings; a recording that cannot establish what happened belongs in an explicitly reported evidence gap.

Budget for preparation and analysis, not just attendance

Unmoderated sessions remove the need to attend every session live. They do not remove recruitment, task design, recording review, or synthesis. NN/g's cost comparison is useful for identifying categories of work, but its 2020 price examples are not current quotes. Obtain actual tool, recruitment, and incentive costs for your own study.

Here is a small planning calculation using invented researcher-hour estimates. It compares six attempts at the same scoped task, not a standard sample-size recommendation. Moderated planning assumes six 30-minute sessions and 15 minutes of notes after each. Unmoderated planning assumes six 20-minute recordings reviewed at normal speed, with 10 minutes of notes per recording. Recruitment administration and participant costs are held outside this illustration, so the totals are not complete project budgets.

Under these assumptions, preparation of 3 hours plus 3 hours of live sessions, 1.5 hours of notes, and 2 hours of synthesis gives 9.5 moderated hours. Unmoderated preparation of 4 hours plus 2 hours of viewing, 1 hour of notes, and 2 hours of synthesis gives 9 hours. The half-hour difference is a consequence of the stated assumptions, not evidence that the formats generally cost the same. Change the task, recording length, or recruitment difficulty and the result changes.

Write the decision and the limit of the conclusion

For the fictional pottery site, start with moderated sessions if the central uncertainty is what people think Pending means and what they expect the service to do. Once that interpretation is understood and the confirmed-change branch is stable, an unmoderated study can examine that narrower task without requiring live attendance. The second study answers its declared question; it does not retroactively validate every observation from the first.

A usable decision note would read: 'We will run moderated sessions on the pending-approval branch because the next design decision depends on participants’ interpretation of the status. We will record the first explanation before probing and label any assisted exploration. We will not use these sessions to estimate population completion rates. A later unmoderated study may assess the confirmed-change branch after account reset, stopping options, and recording quality pass the pilot.'

For your own note, fill in the research question, target participants, exact task state, evidence needed, chosen format, assistance rules, pilot checks, and conclusion you will not draw. If those fields remain vague, choosing a platform will not make the study more precise. A clear format decision follows from the work you need the evidence to do.

• Question: what decision will this study change?
• Participants and task: who is doing what, from which starting state?
• Evidence: what distinguishes completion, assistance, stopping, and an unknown outcome?
• Method: why does live follow-up matter here, or why can the task stand alone?
• Pilot and reporting: what must work before launch, and what will remain outside the conclusion?

References

• NN/g: Unmoderated User Tests
• GOV.UK: Using moderated usability testing
• NN/g: Remote usability-testing costs