By Marijke Platenkamp-Broekhuis
When I started my PhD on usability benchmarking of eHealth applications, I noticed a certain level of scepticism. There was this notion that usability was ‘figured out’, that there was nothing new to discover. In this blog post I will argue why the concept of usability is still worthy for further exploration, especially in the field of eHealth.
The general definition of usability has not changed since the ‘90s. It is described as: ‘The extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use’. On the one hand, this definition is clear: the user needs to be able to use the system effectively, efficiently and satisfactory. However, on the other hand the definition is fuzzy as it does not specify the type of system, users, the goals and context-of-use. You, as a researcher of usability expert, need to fill this in and decide what effective or satisfactory use means for your product. That needs to be taken into account during the evaluation of the application’s usability. The funny thing is, that when we evaluate usability of systems and applications, the same instruments are used for all different kinds of (digital) applications. Research showed that usability questionnaires are the most popular means to evaluate an application’s usability. These questionnaires, of which the System Usability Scale (SUS) is most frequently used, are all general in the sense that they do not consider specific product, user, goal or contextual characteristics that may affect the user’s perception of the usability. In my opinion however, we need to reverse this process: by starting to define usability from these characteristics and then select or build a suitable instrument to evaluate the usability of this application. In other words, to define and evaluate usability based on the system domain. In my research this has been the field of eHealth.
For eHealth applications, it is especially important to consider these product, user, goal or contextual characteristics for a couple of reasons:
Product: The SUS has been used for a wide variety of products, like microwaves, eLearning platforms, eHealth applications and computer programs. However, a microwave is not even remotely similar to an eHealth application. So it does not make slightest sense to use the same questionnaire to evaluate the usability of both systems. Now I guess you are with me on the whole ‘eHealth is not a microwave’ -argument, but I could imagine you think that for other digital applications, may it be eHealth, eLearning or eCommerce, usability involves similar aspects. This is true to a certain degree. However, eHealth applications includes often medical terminology, are connected to other health applications or built in a certain way to accommodate for visual, cognitive or physical health impairments of the intended target audience. Furthermore, user problems could lead to hazardous situations. For example, I once found the following usability issue in a dataset of an online application for people with diabetes type 2: The user does not understand the word ‘hypoglycemia’; it is not clear if this indicates a high or low blood sugar level. This is an example of a potentially life-threatening situation when the user does not understand signals from the eHealth application. It is therefore not sufficient to ask if the application is easy to use; you want to know if the user understands the medical terms, feedback and signals of the application. These are factors that are not relevant for webshops or eLearning platforms.
User: For eHealth applications, the end-users are often (1) people with a certain health condition, (2) (a subset of) the general population or (3) health professionals. It could also be that an eHealth application is used by both patients and health professionals. This is often the case if an application is used within treatment programs. For example, the patient uses the eHealth application to receive information or do exercises at home while the health professional monitors the progress and sends his or her patient the exercises via the application. If the user is a patient, the eHealth application needs to make sure that the terminology and wording fit with the knowledge the user has about his or her health condition. Also, it could be that the user has a visual or physical health impairment that could hinder user-system interaction (like when having small buttons on a phone for people with hand muscle or joint problems). Likewise, if the eHealth application is primarily used by health professionals and it does not fit within their work flow or support their tasks, the application will not be used.
Goal: eHealth applications are designed with a specific health goal in mind: to prevent, inform about, diagnose, treat or monitor a health condition. Users need to be aware of the health goals the application can provide. If the users like using an eHealth application but they do not see how it can support their or their patient’s health condition, again, you end up with a smooth working application that few will actually use.
Contextual: The eHealth application is often embedded within a medical institute or treatment program. While you can use an app on your smartphone anywhere and anytime you want, eHealth applications can be confined to specific training rooms within a medical centre (like a VR system in the training room of physiotherapy practice) or dates and times (where the user needs to fill in a health questionnaire at certain time intervals). It is important to make sure that the system is not only user-friendly, but that it is also suitable for the given context in which it is used. If the VR system takes up too much time setting up during a training session, the health professional will probably skip this system and move to other fitness equipment that is easier to start.
How suitable is a general usability evaluation instrument to evaluate the usability of eHealth applications?
Taking this all in mind, I wanted to put it to the test: how suitable is a general usability evaluation instrument to evaluate the usability of eHealth applications? To find the answer, I conducted usability evaluation studies with three different eHealth applications. I compared the System Usability Scale with task performance data and the number of minor (e.g. the user does not like the music), serious (e.g users with colour blindness have difficulty distinguishing elements in the interface) and critical (e.g. the user does not know how to schedule an exercise for the patient) usability issues. These usability issues were derived from a think aloud test. This list of usability issues based on a qualitative data collection method is considered to be the best indicator of a system’s usability (however, it is not the most efficient way to measure the usability of an application as it takes up much time and effort from both researcher and participant, hence the preference for questionnaires). If there are few serious or critical issues, the usability can be considered quite good. I was curious to see if a low (or high) SUS score would result in more (or less) serious or critical usability issues.
Our results indicated that actually task completion, the number of tasks users were able to complete, had a better correlation with the number of serious and critical usability issues and the SUS. This indicates that the SUS is not sufficient for usability evaluations for eHealth applications.
Now that we know that usability has to be interpreted differently for eHealth applications and that a general instrument like the SUS is not good enough, the next step is to conceptualize usability for the eHealth domain. In my next blog, I will continue with exploring the concept ‘eHealth usability’.
If you want to read more about the study I described in this blog, here below you find the information: