Wendy Grevink, auteur op RRD

7 lessons for designing virtual agents for eHealth

By Lex van Velsen

In the past years, I (or rather, RRD) have participated in numerous projects in which we developed virtual agents for eHealth. Virtual agents that supported healthy eating, cognitive health, or supporting older adults in the mourning process after losing their spouse. In all of these projects, we have learned valuable lessons in the design, implementation and evaluation phase. In this article, I would like to share 7 lessons with you for designing virtual agents for eHealth.

The more advanced its functionalities, the more human-like the appearance of the virtual agent should be. This is in line with the expectations of end-users, where the level of simplicity of the agent appearance should match what it does.
Include humour in the dialogues, but not too much. A discussion with a virtual agent should be engaging. Humour can certainly help here, but too much humour will have a detrimental effect on the interaction. So joke with caution, and test the end result with potential end-users.
Make sure that the most important UX aspects for virtual agents for health -‘usefulness’ and ‘enjoyability’- are taken care of. Virtual agents for health should do two things, be useful and engaging. This way, their effectiveness and efficiency are optimized, while end-users keep on using the service. Be sure to have a keen eye on usefulness and enjoyability during the design and testing process.
Be cautious with making the virtual agent look like a peer, it induces bias. It is tempting to make the virtual agent look like a peer of the end-user. You can imagine it will instil feelings of trust and relatedness. However, for the case of older adults, we found out that this introduces ageism. Societal prejudices towards older adults were embodied in the virtual agent and not appreciated by test users.
First impressions last. The first impressions that end-users have of a virtual agent will last months, and will thus affect both the short and long term interaction.
First impressions of a virtual agent are shaped by two factors. The presence of positivity and attentiveness are the factors that, in first instance, predominantly make up whether or not an end-user takes a liking towards a virtual agent for the health context.
More realistic virtual agents lead to more compliance. End-user are more willing to comply with advice given by a realistic agent than with advice given by, let’s say, a cartoonish agent.

Credit due where credit is due. Most of these lessons were the result of the hard work of some of our junior researchers. I would especially like to mention Silke ter Stal and Leonie Kramer here.

Did we inspire you to embed a virtual agent in your own eHealth service? Or do you want to improve your current virtual agent? Drop us a note, we would love to talk shop.

04/14/2022

SmartWork: Smart Age-friendly Living and Working Environment

SmartWork is a project funded under the Horizon 2020 research and innovation action programme (grant agreement No 826343), which started in January 2019 and ends in March 2022. The main aim of the SmartWork project was to build a system that supports older adults staying actively working as long as desired (also called work ability sustainability).

As one of nine research partners, RRD has developed several services and algorithms which were showcased in a demo at the 2nd Workshop on Smart, Personalized and Age-Friendly Working Environments. This workshop was held in conjunction with the 13th International Joint Conference on Computational Intelligence (IJCCI 2021) in October 2021.

This demo video, that can be found below, shows the services that RRD developed as part of the H2020 SmartWork project:

the modules of the healthyMe smartphone application;
the iCare portal;
the Interventions Manager Service (IMS).

healthyMe smartphone application

The healthyMe smartphone application is the main mobile entry point for the users to collect and visualise physiological, activity and lifestyle data. It is available on Android and iOS in three languages (English, Danish, Portuguese). Each module (steps, sleep, heart rate, food diary, weight, exercises) has its own widget, presenting the collected data in daily, weekly and monthly overviews. These collected data are automatically measured through:

an activity tracker to measure physical activity, sleep and heart rate (Fitbit Charge 3); and
a smart scale to measure body weight (Withings Body).

The food diary in the application allows users to manually track their food intake, which raises their awareness of the total amount of energy consumed. The office-friendly exercise widget presents a library of video-guided exercises that have been recorded in collaboration with healthcare professionals. The exercise videos allow users to safely perform physical exercises at home or at work at the time of their best convenience. The integrated filter allows the user to select exercises by body parts (shoulders, neck, back, arms, legs).

The virtual coach “Amelia” guides users through the application, starting with an intake dialogue through which users can set their activity goals. Depending on their actual level of physical activity that is tracked later on, the goal is automatically adjusted. If a person is less active, the step goal will be adjusted and increased if a person reached their step goals. To prevent demotivation, the automatically adjusted goal is always slightly higher than was reached in the previous week and hence likely to be achievable for the person.

iCare portal

The iCare portal is a service that allows formal and informal caregivers to support the older office worker reaching their health goals. Strong focus is placed on privacy and control in that the office worker can configure within the healthyMe service which data they want to share, from which period of time and with whom. After configuration, summaries of health-related information collected within the healthyMe service are visualised in a web-based portal. This way, the caregiver can monitor the health status of the office worker and provide support for the self-management of health conditions.

Interventions Manager Services (IMS)

The Interventions Manager Services (IMS) is a centralised component within the SmartWork platform that acts as a smart message hub for triggered interventions. From the back-end service side, the IMS can be called if any of the smart services developed within SmartWork decides that some intervention should be triggered. From the client side, the IMS lets the SmartWork client applications register themselves to be notified of triggered interventions. Through the IMS, all smart services have a single entry-point for delivering intervention triggers, and all client applications have a single entry point for registering to receive triggers. Another motivation for the single entry-point was to avoid overloading the user with multiple notifications of triggered interventions at the same time. Currently, only one intervention is delivered at a given time, and in the future more sophisticated intervention prioritisation mechanisms can be implemented.

After a bit over 3 years, the SmartWork projects is coming to an end this month. It was a great collaboration with research partners from Greece, Switzerland, Portugal, Sweden, Denmark, United Kingdom, Ireland and The Netherlands. We enjoyed working together with the partners and hope we can collaborate in future projects.

03/28/2022

eHealth is not a microwave: so why use the same usability evaluation instrument?

By Marijke Platenkamp-Broekhuis

When I started my PhD on usability benchmarking of eHealth applications, I noticed a certain level of scepticism. There was this notion that usability was ‘figured out’, that there was nothing new to discover. In this blog post I will argue why the concept of usability is still worthy for further exploration, especially in the field of eHealth.

The general definition of usability has not changed since the ‘90s. It is described as: ‘The extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use’. On the one hand, this definition is clear: the user needs to be able to use the system effectively, efficiently and satisfactory. However, on the other hand the definition is fuzzy as it does not specify the type of system, users, the goals and context-of-use. You, as a researcher of usability expert, need to fill this in and decide what effective or satisfactory use means for your product. That needs to be taken into account during the evaluation of the application’s usability. The funny thing is, that when we evaluate usability of systems and applications, the same instruments are used for all different kinds of (digital) applications. Research showed that usability questionnaires are the most popular means to evaluate an application’s usability. These questionnaires, of which the System Usability Scale (SUS) is most frequently used, are all general in the sense that they do not consider specific product, user, goal or contextual characteristics that may affect the user’s perception of the usability. In my opinion however, we need to reverse this process: by starting to define usability from these characteristics and then select or build a suitable instrument to evaluate the usability of this application. In other words, to define and evaluate usability based on the system domain. In my research this has been the field of eHealth.

For eHealth applications, it is especially important to consider these product, user, goal or contextual characteristics for a couple of reasons:

Product: The SUS has been used for a wide variety of products, like microwaves, eLearning platforms, eHealth applications and computer programs. However, a microwave is not even remotely similar to an eHealth application. So it does not make slightest sense to use the same questionnaire to evaluate the usability of both systems. Now I guess you are with me on the whole ‘eHealth is not a microwave’ -argument, but I could imagine you think that for other digital applications, may it be eHealth, eLearning or eCommerce, usability involves similar aspects. This is true to a certain degree. However, eHealth applications includes often medical terminology, are connected to other health applications or built in a certain way to accommodate for visual, cognitive or physical health impairments of the intended target audience. Furthermore, user problems could lead to hazardous situations. For example, I once found the following usability issue in a dataset of an online application for people with diabetes type 2: The user does not understand the word ‘hypoglycemia’; it is not clear if this indicates a high or low blood sugar level. This is an example of a potentially life-threatening situation when the user does not understand signals from the eHealth application. It is therefore not sufficient to ask if the application is easy to use; you want to know if the user understands the medical terms, feedback and signals of the application. These are factors that are not relevant for webshops or eLearning platforms.

User: For eHealth applications, the end-users are often (1) people with a certain health condition, (2) (a subset of) the general population or (3) health professionals. It could also be that an eHealth application is used by both patients and health professionals. This is often the case if an application is used within treatment programs. For example, the patient uses the eHealth application to receive information or do exercises at home while the health professional monitors the progress and sends his or her patient the exercises via the application. If the user is a patient, the eHealth application needs to make sure that the terminology and wording fit with the knowledge the user has about his or her health condition. Also, it could be that the user has a visual or physical health impairment that could hinder user-system interaction (like when having small buttons on a phone for people with hand muscle or joint problems). Likewise, if the eHealth application is primarily used by health professionals and it does not fit within their work flow or support their tasks, the application will not be used.

Goal: eHealth applications are designed with a specific health goal in mind: to prevent, inform about, diagnose, treat or monitor a health condition. Users need to be aware of the health goals the application can provide. If the users like using an eHealth application but they do not see how it can support their or their patient’s health condition, again, you end up with a smooth working application that few will actually use.

Contextual: The eHealth application is often embedded within a medical institute or treatment program. While you can use an app on your smartphone anywhere and anytime you want, eHealth applications can be confined to specific training rooms within a medical centre (like a VR system in the training room of physiotherapy practice) or dates and times (where the user needs to fill in a health questionnaire at certain time intervals). It is important to make sure that the system is not only user-friendly, but that it is also suitable for the given context in which it is used. If the VR system takes up too much time setting up during a training session, the health professional will probably skip this system and move to other fitness equipment that is easier to start.

How suitable is a general usability evaluation instrument to evaluate the usability of eHealth applications?

Taking this all in mind, I wanted to put it to the test: how suitable is a general usability evaluation instrument to evaluate the usability of eHealth applications? To find the answer, I conducted usability evaluation studies with three different eHealth applications. I compared the System Usability Scale with task performance data and the number of minor (e.g. the user does not like the music), serious (e.g users with colour blindness have difficulty distinguishing elements in the interface) and critical (e.g. the user does not know how to schedule an exercise for the patient) usability issues. These usability issues were derived from a think aloud test. This list of usability issues based on a qualitative data collection method is considered to be the best indicator of a system’s usability (however, it is not the most efficient way to measure the usability of an application as it takes up much time and effort from both researcher and participant, hence the preference for questionnaires). If there are few serious or critical issues, the usability can be considered quite good. I was curious to see if a low (or high) SUS score would result in more (or less) serious or critical usability issues.

Our results indicated that actually task completion, the number of tasks users were able to complete, had a better correlation with the number of serious and critical usability issues and the SUS. This indicates that the SUS is not sufficient for usability evaluations for eHealth applications.

Now that we know that usability has to be interpreted differently for eHealth applications and that a general instrument like the SUS is not good enough, the next step is to conceptualize usability for the eHealth domain. In my next blog, I will continue with exploring the concept ‘eHealth usability’.

If you want to read more about the study I described in this blog, here below you find the information:

Broekhuis, M., van Velsen, L., & Hermens, H. (2019). Assessing usability of eHealth technology: A comparison of usability benchmarking instruments. International Journal of Medical Informatics, 128(January), 24–31. https://doi.org/10.1016/j.ijmedinf.2019.05.001

03/01/2022

HOLOBALANCE: HOLOgrams for personalised virtual coaching and motivation in an ageing population with BALANCE disorders

Feasibility and effectiveness of the HOLOBALANCE telerehabilitation platform

88% of participants in the HOLOBALANCE interventions achieved at least the minimal clinically significant difference in the Functional Gait Assessment (which is used to assess postural stability and an individual’s ability to perform multiple tasks while walking) scores vs. 55% for participants in the control group. Moreover, 82% of participants in the HOLOBALANCE interventions achieved at least the minimal clinically significant difference in Mini-BESTest (which is a standard balance measure) scores vs. 52% for participants in the control group.

Balance disorders are very common in older adults and have wide ranging detrimental physical, cognitive/psychological and life quality sequelae. Early detailed individualized assessment and treatment interventions for older adults with balance disorders at risk of falls is recommended by several guidelines but is often not implemented into clinical practice due to lack of resources and specialist knowledge.

The HOLOBALANCE platform (https://holobalance.eu/ ) provides an advanced telerehabilitation solution that addresses the critical health needs of older adults with balance disorders. HOLOBALANCE features evidence-based exercises, cognitive and other gamified training, physical activity planning, and components that aim to improve motivation and compliance with the activities.

The proof-of-concept study, conducted from May 2019 to September 2021, with about 80 patients using HOLOBALANCE and 80 patients as controls, aimed to explore an advanced and more holistic approach to standard care. HOLOBALANCE is a tele-rehabilitation system which provides an individualised, prescribed program to be performed at home. The program has been originally designed by a specialist balance physiotherapist and is intended to be prescribed and regularly reviewed by a non-expert clinician. The system supports this by remotely monitoring task performance and providing the outcomes to the treating physiotherapist to review. The system comprises of a set of CE marked wearables (i.e., accelerometers, sensorized soles, smart bracelet), ambient sensors (motion capture sensor) and a head worn augmented reality display, which provide detailed movement and physiological data for the remote assessment of task performance (prescribed exercises, auditory tasks, and cognitive games).

The study recruited 145 older adults with falls/at risk of falls and middle aged or older individuals with a confirmed diagnosis of a vestibular disorder. Participants were randomised into one of two intervention groups. One group received the HOLOBALANCE/HOLOBOX intervention, i.e. the home-based or the clinic-based intervention, and the other received the standard clinical treatment i.e., the OTAGO Home Exercise Program (HEP) for 8 weeks. All participants were required to undertake a daily exercise program intended to improve their balance, with the program overseen by a physiotherapist.

Despite the huge challenges since the study was performed during the COVID-19 global pandemic, recruitment and retention rates were good, dropout rates were low, and the estimated sample size was achieved. The COVID-19 global pandemic inevitably led to continuous recruitment and intervention uncertainty due to the various government restrictions imposed across the three study sites of Athens, Freiburg, and London and imposed challenges for participants, researchers, and treating therapists. We anticipate that these measures of feasibility would be better under less adverse circumstances. Importantly, the absence of any adverse events during the study related to participant interventions or associated with functioning of the HOLOBALANCE system suggest that the program is safe to use in future studies and clinical applications.

The high rate of exercise compliance (83%) observed across all study sites, and the improved balance perceived by 73% of these older adults, and those with vestibular disorders adds further support for the feasibility of the HOLOBALANCE programme implemented in this research. Furthermore, both the HOLOBOX and Home-based delivery of the systems appear to be feasible interventions, regardless of the age, sex, or education level of this study population.

Regarding effectiveness, for average baseline scores near normal or normal, 88% of all participants in the HOLOBALANCE interventions vs. 55% for participants in the control group improved their postural stability (measured with the Functional Gait Assessment scale - FGA). 82% of all participants in the HOLOBALANCE interventions vs. 52% for participants in the control group improved their balance (evaluated with the Mini-BESTest). Importantly, an average improvement of 25% for FGA and of 36% for Mini-BESTest for those with abnormal scores at baseline and eventually trained with HOLOBOX. It is anticipated that in a client group with greater disability and lower scores, the pre-post treatment change would have been even greater.

A high dosage (>50 hours) of challenging balance exercises has been associated with a successful reduction in fall rates in older adults. Our findings though, indicate that a moderate dosage multisensory and multifactorial balance rehabilitation program may also have a significant effect on falls risk and intervention cost. The potential benefit for these multiple factors is therefore immense and must be confirmed in a full randomised controlled trial.

An additional beneficial finding was noted for social well-being. In all groups a decrease, in the average total number of days during which their balance difficulties affected their ability to perform daily activities, was observed. Participants in the HOLOBALANCE interventions combined, 87.5% achieved at least a 25% improvement in social wellbeing. Moreover, a decrease of 14.66 days per month during which the participants’ balance impacted their lives in a negative way was demonstrated.

Further exciting findings were observed for cognitive function scores with significant pre-post improvements (8%) noted for visual memory and new learning and visual pattern recognition memory. It is well documented that falls risk is greater for those with cognitive impairment whereby all aspects of balance control deteriorate with increasing severity of cognitive impairment. This finding has significant implications for further studies in this area, particularly in individuals with cognitive impairment, and subsequently for clinical practice. Positive improvements in cognition, as those observed, may also contribute to the delay of onset for cognitive impairment and consequently prolong the active life of these older adults, with better quality of life, improved social wellbeing and reduced economic costs for their care.

The project involves 13 partners from across Europe and is coordinated by Prof. Dimitrios I. Fotiadis, who is Professor of Biomedical Engineering at the University of Ioannina.

If you would like more information about the project visit https://holobalance.eu/.

HOLOBALANCE has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 769574.

02/10/2022