Usability Evaluation Report
Summary
The Usability Evaluation Report documents the results and analysis of your usability testing activities, providing evidence that users can safely operate your medical device. This report analyzes both formative evaluation conducted during development and summative evaluation performed on the final device, demonstrating compliance with human factors engineering requirements and supporting risk management decisions.
Why is Usability Evaluation Report important?
This report serves as regulatory evidence that your device user interface is safe and effective for its intended users. Regulators require documented proof that you systematically evaluated user interactions and addressed any use-related risks before market release. The report demonstrates that your usability engineering process is complete and provides objective data supporting your device’s safety profile. Without this comprehensive documentation, you cannot demonstrate compliance with human factors requirements or justify that your device is safe for user operation.
Regulatory Context
Under FDA Human Factors Guidance and 21 CFR Part 820.30:
- Report must provide objective evidence of user interface safety validation
- Must document all use errors, difficulties, and task failures observed during testing
- Critical task performance must be thoroughly analyzed and justified
- Results must inform risk management and design control processes
- Report supports validation requirements for design controls
Special attention required for:
- High-risk devices requiring formal validation evidence
- Documentation of any critical task failures and corrective actions
- Integration with 510(k) submissions or PMA applications
- Post-market surveillance planning based on usability findings
Under FDA Human Factors Guidance and 21 CFR Part 820.30:
- Report must provide objective evidence of user interface safety validation
- Must document all use errors, difficulties, and task failures observed during testing
- Critical task performance must be thoroughly analyzed and justified
- Results must inform risk management and design control processes
- Report supports validation requirements for design controls
Special attention required for:
- High-risk devices requiring formal validation evidence
- Documentation of any critical task failures and corrective actions
- Integration with 510(k) submissions or PMA applications
- Post-market surveillance planning based on usability findings
Under EU MDR 2017/745 and IEC 62366-1:2015:
- Report must comply with IEC 62366-1:2015 documentation requirements
- Must demonstrate summative evaluation completion for usability engineering file
- Results must support Annex I GSPR compliance claims
- Report required for technical documentation per Annex II
- Must address use-related hazard scenarios from risk assessment
Special attention required for:
- IEC 62366-1 Section 4.2 usability engineering file requirements
- Integration with clinical evaluation when use errors affect clinical outcomes
- Notified body review of usability evidence and conclusions
- Post-market surveillance system for ongoing use error monitoring
Guide
Your usability evaluation report transforms raw testing data into regulatory evidence that demonstrates user interface safety and supports risk management decisions.
Report Structure and Analysis Framework
Begin with comprehensive documentation of your formative evaluation activities including expert reviews, prototype testing, and iterative design improvements. Document specific feedback received and how it influenced your final user interface design, demonstrating your systematic approach to human factors engineering.
For summative evaluation, provide detailed analysis of all testing data including task completion rates, use errors, use difficulties, and user feedback. Analyze patterns across participants to identify systematic usability issues versus individual user variations.
Formative Evaluation Documentation
Document all formative activities conducted during development including stakeholder consultations, expert reviews, prototype testing, and design iterations. Describe specific suggestions received from medical experts, user representatives, and usability specialists and how these informed your design decisions.
Demonstrate the evolution of your user interface by documenting key design changes made in response to formative feedback. This shows regulators that you proactively addressed usability concerns before final testing, reducing the likelihood of summative evaluation failures.
Summative Evaluation Results Analysis
Present comprehensive test results including participant demographics, task completion data, time measurements, error rates, and qualitative feedback. Organize results by hazard-related use scenario to demonstrate systematic coverage of safety-critical interactions.
Analyze use errors and difficulties by categorizing them by severity, frequency, and potential impact on safety. Distinguish between errors that could lead to harm versus those that only affect user satisfaction or efficiency. Document any close calls where users nearly made errors but recovered.
Critical Task Performance Assessment
Provide detailed analysis of critical task performance since these represent scenarios where user errors could cause significant harm. For any critical task failures, document the specific failure mode, contributing factors, and immediate corrective actions taken.
Justify acceptance of any critical task issues through risk-benefit analysis, demonstrating that residual risks are acceptable when weighed against device benefits and available risk controls.
Risk Management Integration
Document how testing results influenced your risk management file including newly identified risks, updated risk control measures, and changes to design requirements. Demonstrate that all use-related risks are properly controlled through design features, protective measures, or information for safety.
Specify post-market surveillance plans for monitoring use errors and difficulties that may emerge during real-world use, showing your commitment to ongoing safety monitoring.
Impact Assessment and Conclusions
Analyze testing impact on software requirements, user interface design, use specifications, and risk management processes. Document any design changes triggered by testing results and verification that changes address identified issues.
Provide clear conclusions about user interface safety and readiness for market release, supported by objective data from your testing activities.
Example
Scenario
You have completed usability testing of your mobile ECG monitoring app with 15 patients and need to document the results to demonstrate that users can safely operate the device at home. Your testing revealed one critical task failure and several use difficulties that required analysis and response.
Example Usability Evaluation Report
ID: UER-001-ECG-Monitor
Scope: This report summarizes formative and summative usability evaluation results for the Mobile ECG Monitor System, providing evidence of user interface safety and effectiveness.
Formative Evaluations:
During development, we conducted design reviews with three cardiologists and two emergency medicine physicians to gather feedback on clinical workflow integration. Five patient advisory group sessions with cardiac patients reviewed prototype interfaces focusing on instruction clarity and symptom reporting mechanisms.
Formative Evaluation Feedback: Expert cardiologists recommended simplified electrode placement guidance after observing confusion during prototype demonstrations. Emergency physicians suggested more prominent emergency alert indicators and clearer action instructions. Patient advisors requested larger text for critical instructions and simplified symptom terminology.
Design Improvements from Formative Evaluation: Based on expert feedback, we implemented animated electrode placement guides with step-by-step verification, enhanced emergency alert visual design with red backgrounds and flashing indicators, increased font sizes for safety-critical instructions, and simplified symptom reporting with plain language descriptions and visual icons.
Summative Evaluation Results:
Participant Demographics: Fifteen participants (8 male, 7 female) aged 47-78 years with varying technology experience levels. All participants had basic smartphone experience and suspected cardiac arrhythmias. Technology comfort levels: 6 high, 5 moderate, 4 low.
Task Performance Summary:
| Task | Participants | Success Rate | Average Time | Use Errors | Critical Task |
|---|---|---|---|---|---|
| T-001: App setup | 15 | 100% | 3.2 min | 0 | No |
| T-002: Electrode placement | 15 | 93% (14/15) | 4.8 min | 1 | Yes |
| T-003: ECG recording | 15 | 100% | 2.1 min | 0 | Yes |
| T-004: Emergency response | 15 | 100% | 1.4 min | 0 | Yes |
| T-005: Symptom reporting | 15 | 87% (13/15) | 5.3 min | 2 | No |
Critical Task Analysis: One participant (UD-012, age 74, low technology experience) initially placed electrodes incorrectly during Task T-002, achieving only 65% signal quality. However, the app’s real-time feedback prompted electrode repositioning, ultimately achieving 85% signal quality on the second attempt. This use error did not result in a failed recording but demonstrated the importance of real-time feedback systems.
Use Difficulties Identified:
- Three participants initially struggled with Bluetooth pairing but succeeded with app guidance
- Two participants missed symptom severity indicators, leading to incomplete reporting
- Four participants requested confirmation dialogs for emergency actions
- One participant suggested adding a practice mode for first-time users
User Feedback Summary: Participants rated the app as “easy to use” (average 4.2/5.0) with particular praise for electrode placement guidance and clear emergency instructions. Suggested improvements included practice mode availability and larger buttons for emergency functions.
Risk Management Impact: Testing identified one new risk (R-045): “User confusion during initial Bluetooth setup could delay emergency recordings.” This risk was assessed as low probability, moderate severity and controlled through enhanced pairing instructions and automatic retry mechanisms.
Updated risk controls for existing risks:
- R-015: Real-time electrode feedback system validated as effective
- R-023: Emergency alert system achieved 100% appropriate response rate
- R-031: Symptom reporting interface required minor improvements for clarity
Design Changes Implemented: Based on testing results, we implemented enhanced Bluetooth pairing instructions with visual indicators, confirmation dialogs for emergency actions, optional practice mode for new users, and improved symptom severity indicators with color coding.
Post-Market Surveillance Plan: We will monitor customer support inquiries for usability issues, track emergency response effectiveness through clinical partners, and conduct annual user satisfaction surveys to identify emerging usability concerns.
Conclusions: Summative usability evaluation demonstrates that users can safely operate the Mobile ECG Monitor System for its intended use. All critical tasks achieved acceptable performance levels with no failures that could lead to patient harm. The one critical task difficulty was successfully mitigated by existing design controls. The device user interface is validated for safe use by the intended user population.