The short answer
A prototype can pass usability testing by users moving through the journey, completing the task, or giving positive feedback, and yet still fail as a service once it goes live. This is not because the testing was poorly designed, but because most test scenarios validate one thing: progression. Checking users can move forward under ideal conditions. What they do not test is resilience: whether the service holds up when conditions are not ideal. Progression and resilience are different qualities, and testing only one while deploying both is a structural gap in most UX practice.
What usability testing typically validates
Standard usability testing is built around task completion. A participant is given a scenario, asked to complete a task using the prototype, and observed while doing so. The researcher notes where they hesitate, where they make errors, where they express confusion. The prototype is revised. The test is run again.
This is a valuable process. It surfaces interface problems, identifies unclear labelling, reveals navigation failures, and produces qualitative insight that no other method provides at the same granularity.
But this is also a process that is structurally biased toward the ideal path. Participants are given a scenario designed to take them through the primary journey. They are starting fresh, with no prior history in the service. They are completing the task in a single session. And the prototype they are using typically doesn’t fully simulate the conditions of real use like the delays, the ambiguous states, the moments of genuine uncertainty about whether something worked.
The result is that usability testing tells you whether a path is usable, but it does not go far enough to tell you whether the service is robust.
What resilience requires that progression testing does not
A service is resilient when it behaves predictably under variation — when users deviate from the ideal path and the service responds in a way that still makes sense to them.
Real user behaviour introduces variation that most test scenarios don’t include:
- Answers need to change midway through: a user decides that information they entered three steps ago needs to be different
- Users return after a delay: a task started one day is continued the next, with the user needing to re-establish where they are
- Doubt appears after submission: the user completes a task but isn’t confident it worked, and returns to check
- Actions are repeated because the outcome wasn’t clear: the user submits something twice because the first attempt wasn’t confirmed clearly enough
- Trust in the outcome wavers: the task appears complete but the user seeks confirmation elsewhere before accepting it
None of these are exceptional scenarios. They are normal user behaviour at scale. In complex services — long applications, multi-stage processes, document-heavy transactions — they represent a significant proportion of total interactions.
A prototype tested for progression only has been validated against a version of user behaviour that accounts for a minority of real use.
The illusion of structural confidence
The gap between progression and resilience produces a state that is easy to mistake for readiness. The prototype works, the completion rates in testing look strong, and user feedback is positive. There is no obvious reason to doubt the design.
But what has not been tested are the moments where real use diverges from ideal conditions: where commitment increases, where certainty drops, where a user’s trust in the service is genuinely at stake. These are the moments where services either hold up or begin to fracture. And they are precisely the moments that a test scenario built around a single forward path under ideal assumptions will systematically miss.
A simple way to review a prototype before treating it as ready is to ask a set of deliberately disruptive questions:
- What happens if someone changes an earlier answer late in the journey?
- What happens if they leave before committing and return the next day?
- What happens if they are not sure the outcome was successful?
- What happens if they need to correct something after submission?
If those conditions have not been explored, the testing may be validating appearance rather than structure. The prototype may look like a service, but until those transitions have been tested, it is not yet known whether it will behave like one.
Frequently asked questions
Why do some prototypes pass usability testing but fail in production?
Because most usability testing validates progression — whether users can complete a task under controlled, ideal conditions — rather than resilience: whether the service behaves predictably when real-world variation occurs. Prototypes that pass testing have typically been validated against a single forward path, with participants starting fresh and working in a single session under conditions that don’t reflect the full range of real use.
What is the difference between usability and resilience in UX design?
Usability describes how effectively users can interact with an interface under standard conditions. Resilience describes how well the service responds when those conditions vary — when users change their minds, return to incomplete tasks, receive unclear outcomes, or encounter states the design didn’t anticipate. A service can score well on usability and poorly on resilience if it has been designed and tested primarily for the ideal path.
What should usability testing include to surface resilience problems?
Resilience testing requires introducing variation into test scenarios: asking participants to change an answer they gave earlier, to leave and return to a task mid-way, to express doubt after completing a submission, or to navigate an error state. These conditions reveal whether the service’s structure — its states and transitions — is robust enough to handle the full range of real user behaviour, not just forward progression under ideal assumptions.
How do you know if a prototype has been tested for resilience?
Ask whether the test scenarios included variation from the ideal path. If all scenarios were built around a single forward journey from start to completion, with no interruptions, corrections, or returns, the prototype has been tested for progression only. A resilience-tested prototype will have specific evidence of how it behaves when users deviate — what states are entered, what feedback is provided, and whether recovery is clear and achievable.