Usability Evaluation Protocol
Summary
The Usability Evaluation Protocol provides specific testing procedures for conducting summative usability evaluation of your medical device. This document translates your usability evaluation plan into detailed test scenarios, participant requirements, and data collection methods to systematically validate that users can safely operate your device without use errors that could cause harm.
Why is Usability Evaluation Protocol important?
The protocol serves as your systematic testing blueprint that transforms use-related hazard scenarios into measurable test tasks. Regulators require detailed protocols to ensure usability testing is conducted consistently and comprehensively. This document demonstrates your methodical approach to human factors validation by specifying exactly how you will test critical user interactions. Without a detailed protocol, your usability testing may miss critical scenarios or lack the rigor needed to satisfy regulatory requirements for summative usability evaluation.
Regulatory Context
Under FDA Human Factors Guidance and 21 CFR Part 820.30:
- Protocol must detail specific test scenarios covering all hazard-related use scenarios
- Must specify participant selection criteria representing intended user population
- Critical tasks must be clearly identified and tested with appropriate rigor
- Data collection methods must capture use errors, difficulties, and task completion
- Protocol must support validation requirements for design controls
Special attention required for:
- High-risk devices requiring formal validation protocols
- Software devices with complex user interactions
- Testing environments that simulate actual use conditions
- Integration with risk management and design control processes
Under FDA Human Factors Guidance and 21 CFR Part 820.30:
- Protocol must detail specific test scenarios covering all hazard-related use scenarios
- Must specify participant selection criteria representing intended user population
- Critical tasks must be clearly identified and tested with appropriate rigor
- Data collection methods must capture use errors, difficulties, and task completion
- Protocol must support validation requirements for design controls
Special attention required for:
- High-risk devices requiring formal validation protocols
- Software devices with complex user interactions
- Testing environments that simulate actual use conditions
- Integration with risk management and design control processes
Under EU MDR 2017/745 and IEC 62366-1:2015:
- Protocol must comply with IEC 62366-1:2015 testing requirements
- Must cover all hazard-related use scenarios identified in risk assessment
- Summative evaluation protocol required for usability engineering file
- Testing must provide evidence for Annex I GSPR compliance
- Results must support technical documentation requirements
Special attention required for:
- IEC 62366-1 Section 4.3 tailoring requirements
- Integration with clinical evaluation when applicable
- Post-market surveillance planning for use errors
- Notified body review of usability evidence
Guide
Your usability evaluation protocol converts your high-level usability plan into specific, executable testing procedures that generate regulatory-compliant evidence.
Protocol Foundation
Begin by documenting your user interface description including platform details, interaction methods, authentication requirements, and workflow integration. Specify training requirements for participants including any materials that must be reviewed or skills that must be demonstrated before testing begins.
Document materials provided to users during testing including instructions for use, training materials, and any supporting documentation that accompanies your device in real-world use. These materials must match what users will receive with the commercial product.
Hazard-Related Use Scenarios and Tasks
Translate each hazard-related use scenario from your risk assessment into specific, testable tasks. Each task should represent a realistic user interaction that could lead to the identified hazardous situation. Clearly identify which tasks are critical tasks where user errors could cause significant harm.
Structure tasks to flow logically through typical user workflows while ensuring all hazard-related scenarios are covered. Include both normal use tasks and reasonably foreseeable misuse scenarios that could occur in real-world environments.
Participant Selection and Testing Environment
Define participant criteria that represent your intended user population including demographic characteristics, medical conditions, technology experience, and any special requirements. Specify minimum sample sizes based on device complexity and regulatory requirements, typically five or more users per user group.
Design testing environments that simulate actual use conditions including physical settings, lighting, noise levels, time pressures, and potential distractions. Consider whether testing should occur in clinical settings, home environments, or specialized facilities based on your intended use.
Data Collection Framework
Establish systematic data collection methods that capture task completion status, use errors, use difficulties, and close calls. Design observation forms that allow testers to document not only what users do but also how they think through tasks and where confusion occurs.
Plan user feedback collection through structured interviews or questionnaires that gather insights about interface clarity, task difficulty, and improvement suggestions. Design questions that probe both objective performance and subjective user experience.
Test Execution Procedures
Document step-by-step procedures for conducting each test session including participant briefing, task administration, data recording, and session conclusion. Specify how to handle unexpected situations, user questions, and technical difficulties during testing.
Include safety considerations for managing situations where users commit errors during testing, especially for critical tasks. Define stopping criteria for tests that might put participants at risk or generate invalid data.
Results Analysis and Integration
Specify how test results will be analyzed including criteria for determining task success, failure, and partial completion. Define how use errors and difficulties will be categorized and how findings will feed back into your risk management and design control processes.
Plan for iterative testing if initial results reveal significant usability issues requiring design modifications and retesting.
Example
Scenario
You are creating a detailed protocol for testing your mobile ECG monitoring app that will validate whether patients can safely record heart rhythms at home and respond appropriately to emergency alerts. Your protocol must systematically test all hazard-related scenarios identified in your risk assessment through specific user tasks.
Example Usability Evaluation Protocol
ID: UEP-001-ECG-Monitor
Scope: This protocol defines summative usability evaluation procedures for the Mobile ECG Monitor System to validate user interface safety and effectiveness for hazard-related use scenarios.
User Information:
Intended Users: Adults aged 45-80 with suspected cardiac arrhythmias, representing both high and low technology comfort levels. Participants must have basic smartphone experience and be able to read English instructions.
Use Environment: Home environment with typical household distractions including television, family members, and varying lighting conditions. Testing simulates real-world scenarios including evening use and time-pressure situations.
Device Information:
User Interface: Mobile application accessed via iOS or Android smartphone with touch screen interaction, voice prompts for guidance, and Bluetooth connectivity to ECG sensor hardware. Authentication requires PIN or biometric login.
Training Required: 15-minute introduction video covering electrode placement, app navigation, and emergency response procedures. No specialized medical training required.
Materials Provided: Instructions for Use booklet, quick reference card for electrode placement, emergency contact information card, and device carrying case.
Hazard-Related Use Scenarios:
| Risk ID | Risk Type | Hazard | Hazardous Situation | Harm |
|---|---|---|---|---|
| R-015 | Use Error | Poor electrode contact | Undetected arrhythmia | Missed diagnosis leading to cardiac event |
| R-023 | Use Error | Ignoring emergency alert | Delay in seeking emergency care | Serious cardiac complications |
| R-031 | Use Error | Incomplete symptom reporting | Misinterpretation by clinician | Inappropriate treatment decisions |
Testing Tasks:
| Task ID | Task | Hazard-Related Use Scenario | Risk ID | Critical Task |
|---|---|---|---|---|
| T-001 | Complete initial app setup and account creation | Poor electrode contact leading to poor recordings | R-015 | No |
| T-002 | Attach electrodes following app guidance and achieve >80% signal quality | Poor electrode contact leading to undetected arrhythmia | R-015 | Yes |
| T-003 | Record 30-second ECG during simulated chest pain episode | Failure to capture arrhythmia during symptoms | R-023 | Yes |
| T-004 | Respond appropriately to high-priority emergency alert within 2 minutes | Ignoring emergency alert leading to delayed care | R-023 | Yes |
| T-005 | Complete symptom diary entry and share with provider | Incomplete reporting leading to misdiagnosis | R-031 | No |
User Testing Data Collection:
For each participant (UD1-UD15), collect:
- Demographic information (age, gender, technology experience)
- Task completion status (pass/fail) for each task
- Time to complete each task
- Number of attempts required for successful completion
- Use errors observed during testing
- Use difficulties and close calls
- Critical task performance (must achieve 100% success rate)
- Post-test feedback on interface clarity and difficulty
Test Session Procedure:
- Welcome participant and obtain informed consent
- Provide 15-minute training session with materials
- Allow 10-minute practice period with app
- Conduct tasks T-001 through T-005 in sequence
- Document all observations in real-time
- Conduct post-test interview (15 minutes)
- Review safety considerations and emergency procedures
User Feedback Questions:
- How would you describe the ease of using this application?
- What aspects of electrode placement were most challenging?
- How clear were the emergency alert messages and required actions?
- What improvements would make the app safer or easier to use?
- How does this compare to other health apps you’ve used?
Impact Assessment: Results will be evaluated for impact on:
- Risk management file updates for newly identified use-related risks
- Software requirements modifications for user interface improvements
- User interface design changes to address usability issues
- Instructions for use clarifications based on user confusion
- Training material enhancements to reduce use errors
Acceptance Criteria:
- 100% of participants must complete critical tasks T-002, T-003, and T-004 successfully
- No more than 20% of participants should require multiple attempts for critical tasks
- Use difficulties must be documented and assessed for risk impact
- Any use errors leading to potential harm must trigger design review