Wizard of Oz testing simulates part of a system with a human operator before that capability is built. The useful output is evidence about the proposed interaction: what people ask, where clarification is needed, and whether the response supports their next action. It does not establish that an automated system can produce the same result.
Choose one job for the simulated assistant
Use a library assistant that helps a member choose a book available for Saturday collection. The interface accepts a short request, offers one suitable title from a fixed catalog, and links to its pickup options. The study question is whether members can move from an everyday request to a book they understand and a pickup they can use.
Keep the scope narrow. The operator can select from the three catalog entries below and ask one clarification when needed. They cannot invent stock, reserve a book, promise a new pickup time, or search the web for a more impressive recommendation. Otherwise each participant encounters a different service and the team cannot tell which interaction it tested.
| ID and title | Suitable request | Pickup availability |
|---|---|---|
| B1 — The Secret Garden | A classic story; no short-book preference | Central: no Saturday slots. Riverside: Saturday 10:00 |
| B2 — The Wind in the Willows | A classic animal story | Central: Saturday 11:00 |
| B3 — The Railway Children | A classic family adventure | Riverside: Saturday 10:30 |
Decide what participants know about the operator
The traditional Wizard of Oz setup described by TU Darmstadt keeps human operation hidden during the interaction. For a transparent early exercise, tell participants that a person may help produce the responses: We are testing a prototype, and a researcher may select its replies. The catalog and pickup times are sample data; no reservation will be made.
That disclosed setup is useful for testing request wording, clarification, and the handoff to pickup selection. It does not answer whether people would trust or use an apparently autonomous assistant in the same way. If perceived automation is the research question, design an appropriate consent and debrief protocol with your research team rather than quietly changing the disclosure midway through a study.
Give the operator a finite response playbook
Prepare the response text before the session. Let the operator choose the relevant response and substitute only the allowed catalog fields. Log deviations instead of hiding them in polished conversation. If the request falls outside the catalog, say so and give the participant a way to browse the available titles.
A clarification should resolve information needed for the task. Asking for a favorite color would be conversational decoration. Asking whether another branch is possible directly changes which pickup options can satisfy the request.
| Input condition | Operator action | Response |
|---|---|---|
| Member names B1 and wants Saturday at Central | Use B1 availability; ask about branch flexibility | Central has no Saturday pickup times for this option. Could you collect at Riverside at 10:00? |
| Member wants an animal story and can use Central | Select B2 | The Wind in the Willows is a classic animal story. Central has a Saturday 11:00 pickup option. View pickup details to continue. |
| Member asks for a family adventure at Riverside | Select B3 | The Railway Children is a classic family adventure. Riverside has Saturday 10:30 available. View pickup details to continue. |
| Member asks for something good, without another preference | Ask one useful clarification | Would you prefer an animal story, a family adventure, or The Secret Garden? |
| Request cannot be satisfied from these entries | Use fallback; do not invent a result | These sample titles do not cover that request. You can browse this catalog or change your preference. |
| Member asks Have you booked it? | Clarify the handoff | No reservation has been made. Open pickup details to review the branch and time before confirming. |
Connect the visible interface to the backstage work
The participant sees a request field, a response area, a title card, and View pickup details. The operator sees the incoming message, catalog, response playbook, and timestamped log. The researcher should be able to reconstruct which rule produced the visible response.
Do not add an artificial instant reply if a person actually takes time to decide. Record both request and response time. If you intentionally simulate a delay, record that separately. Observed operator speed is a property of this study setup, not a forecast of model latency.
Run a task, then inspect the handoff
Participant task: You want a classic animal story for the weekend. Central is the easiest branch for you, and you can collect on Saturday. Use this assistant to find an option and tell us what you would do next. Do not include the expected title or View pickup details label in the task.
Before involving participants, rehearse three requests: a supported request, an ambiguous request, and an unsupported request. Check that the operator can locate a rule without help, that each title card opens matching pickup details, and that the prototype never displays a reservation confirmation after a recommendation alone.
Observe whether the member knows why the title was suggested, notices the branch and time, and understands that another action is required. An enthusiastic response to the book recommendation does not establish successful pickup or willingness to return.
| Event | Visible behavior | Operator record | Design implication |
|---|---|---|---|
| Request at 00:12 | Member asks for an animal story at Central on Saturday | Rule B2 selected; reply sent at 00:20 | Eight-second response in this example; do not extrapolate automation speed |
| After recommendation | Member asks whether the book is already reserved | Used the prepared no-reservation reply | The card may make recommendation and reservation look equivalent |
| Pickup handoff | Member opens details and correctly names Central, Saturday 11:00 | No operator improvisation | Keep branch/time visible at the handoff |
| Next version | Card adds Not reserved yet above View pickup details | One wording change recorded | Check whether a new participant still assumes a reservation exists |
Decide what to build from the result
The example log identifies a concrete next change: separate a recommendation from a reservation in the title card. Test that revised handoff before building a broad conversational engine. Keep a request inventory too; repeated unsupported requests may mean the catalog or product scope is wrong, not that the operator needs more improvisation.
Use three decision buckets. Interface repair: revise wording, hierarchy, or the next action. Service scope: add a supported request only after deciding its data and policy requirements. Technical feasibility: separately test whether the intended automated component can select valid catalog entries, preserve constraints, and refuse unavailable results.
Copy this run sheet for your own study: user job; fixed data; allowed operator actions; response rules; unsupported-request fallback; participant disclosure; task prompt; timing log; observed misunderstanding; proposed repair; next test. A good handoff includes the playbook and evidence behind the change, not just a transcript that makes the assistant sound capable.