Are you working on the development of a health technology? An important prerequisite is that the design of the technology fits well with the target audience. Is the product user-friendly? Does the target group understand how it works, even without an instruction manual? And are they satisfied with your technology? And how are these factors measured?
Conducting usability testing is a crucial activity within the design and development of a health technology. It can improve the quality of an innovation by uncovering issues that hinder use, user experience and impact, which can then be resolved in a redesign. This saves your organization time, money and effort by making your technology more responsive to end-user needs.
In addition, conducting a usabiliy test is a prerequisite for assessing the safety of a health technology, a standardized component in CE marking within the Medical Device Regulation (MDR) for a medical device.
Our Method
RRD has extensive experience in usability testing, user experience and usability benchmarking of your prototype or already implemented technology. Depending on the maturity of your technology (TRL level) and your desire, we select the most appropriate usability test and test your health technology with potential end users in the appropriate context.
We would like to relieve you of all your worries if we offer you the total solution below. Of course we can determine whether RRD performs all steps for you or whether your organization takes care of step 1, 2 and/or 3 itself.
- Test preparation (setting goals, selecting critical performance indicators, and writing the test protocol);
- Ethical approval (writing the documentation for obtaining medical-ethical consent, or an exemption and handling the entire ethical process);
- Recruitment of participants (including patient groups, older adults and healthcare professionals);
- Implementation of the test (in our own usability lab or at your location);
- Analyze and report the science-based results, which you can then use to complete your own CE documentation (usability engineering file).
Usability testing
The "classic" usability test
A group of end users interact with your technology while thinking aloud, we observe, and we inventory and prioritize usability issues, with the goal of making recommendations to optimize your health technology;
Peer discovery
Is your technology being developed for users with limited (digital) skills? If so, it may be wise to have pairs of end users interact with your technology for a richer evaluation. We analyze their shared interaction and identify and prioritize usability issues;
Expert reviews
Do you need a specialized evaluation of your health technology, for example, in terms of accessibility, use by low-literate people, or to assess potential ethical issues? If so, one of our experts will be happy to conduct a thorough assessment of your technology.
User experience
You want your health technology to work, but also to deliver a positive experience? We can assess the user experience of your technology for factors you consider relevant (e.g., engagement, trust, control, enjoyment).
Usability benchmarking
We can benchmark the usability of your health technology using the System Usability Scale (SUS) or the special eHealth UsaBility Benchmarking instrument (HUBBI)., which factors scores as task technology fit and understandable information and terminology.
Not sure what the best test is for your situation? We would be happy to analyze your situation together and come up with a customized solution for you.
Practical example of a "classic" usability test
To gain insight into the usability and user acceptance of a hand exoskeleton prototype (GRIP; Figure 1), which aims to support lateral grasping in paraplegic patients, RRD conducted a classic usability test. The most important functionalities of the prototype were tested among potential end users, consisting of spinal cord injury (n=5) and CVA patients (n=2). They were asked to perform seven tasks using the aloud thinking method while we observed them. The outcomes focused on identifying and prioritizing usability issues, measuring quantitative task metrics and some questionnaires measuring benchmarking and acceptance of the technology.
Figure 1. GRIP hand exoskeleton
Task metrics
Figure 2 shows radar charts of the task metrics, where a fully filled chart indicates the best possible performance on all outcomes. Task 3 was the best performed by participants, followed by Task 2 and Task 7. The performance of task 4 was very poor.
Figure 2. Radiagrams of task metrics.
Benchmarking questionnaires
To benchmark the GRIP hand exoskeleton, the System Usability Scale (SUS)1 and the Usefulness, Satisfaction, and Ease of Use Questionnaire (USE)2 questionnaire were administered. The SUS consists of 10 statements, where participants were asked to score each statement on a 5-point Likert scale. The USE measures the subjective usefulness of a product or service. Through the completion of 30 items, four dimensions of usability were examined: usefulness, ease of use, ease of learning and satisfaction. The questionnaire uses a 7-point Likert scale, which focuses on experience with and feedback on the GRIP hand exoskeleton.
Two participants scored a below-average SUS score (score < 68; Figure 3). The lowest and highest SUS scores were 55 (participant 3) and 97.5 (participant 2), respectively. The mean SUS score was 79.3 (sd 14.1), which represents good usability.
Figure 3. SUS scores
The scores on the USE are summarized in Table 1. The lowest mean score (4.5) was found for usability. In particular, both CVA patients (participant 5 and 7) scored low on this aspect of usability. The highest mean score (6.46) was found for ease of learning. Participants also scored above average on the aspects of usability and satisfaction.
Acceptance
Acceptance of the GRIP hand exoskeleton was evaluated using the Technology Acceptance Model (TAM)3. TAM states that perceived usefulness, perceived ease of use and intention to use have the most influence on the adoption rate of a new product. Perceived usefulness is the extent to which a user sees the added value or utility of the product and perceived ease of use is the perceived ease of use. The TAM questionnaire uses a 7-point Likert scale and consists of 12 items.
Table 2 shows the mean results on all items. The mean scores for perceived usefulness, perceived ease of use and intention to use were 4.3 (sd 1.9), 6.2 (sd 0.8) and 4.7 (sd 2.0), respectively. The lower scores on perceived usefulness and intention to use were again due to the low scores given by both CVA patients (participant 5 and 7). The paraplegic population scored well on all aspects of technology acceptance.
Conclusion
The GRIP hand exoskeleton appears to be a suitable tool for the paraplegic population in particular. The current prototype scores well on usability and acceptance, and participants were above average in their satisfaction with the technology. However, modifications are desirable to the control of the current prototype, which is currently done via software on a smartphone or smartwatch. Especially putting on (task 4) and controlling via smartwatch (tasks 5 and 6) showed many usability issues and reduced performance on task metrics. Control via voice could provide a suitable solution direction.
References
1 Brooke, J. (1996). SUS - A quick and dirty usability scale. In and I. L. M. (Eds. . P. W. Jordan, B. Thomas, B. A. Weerdmeester (Ed.), Usability evaluation in industry (pp. 189-194). Taylor & Francis.
2 Lund, A. M. (2001). Measuring usability with the USE questionnaire. Usability Interface, 8(2), 3-6.
3 Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. Mass Q. 1989;13:319.