The Use of Eye-Tracking in Usability Testing of Medical Devices
Thomas Koester, Jesper E. Brøsted, Jeanette J. Jakobsen, Heike P. Malmros and Niels K. Andreasen; FORCE Technology
Objectives
Method
Results Conclusions
Scope of work
Two scenarios
Insights, findings and classification of results
Evaluation and Conclusions
The study presented here is made in collaboration with a medical device manufacturer. Normally, the
device manufacturer uses verbal self-reporting protocols, interviews and observations in their formative usability tests during product development. The objective of this study is to investigate whether
the use of eye-tracking technology (Tobii Pro Glasses 2 shown in figure 1 and iMotions 6.2.5 recording
software) can contribute to the data collection and bring new data and knowledge into the product development. It is the assumption that the use of eye-tracking will provide an insight into the visual perception of the user that cannot be obtained from verbal self-reporting, interviews or observations. In
this context, eye-tracking is seen as a supplement to the ethnographic data collection (interview and
observation), giving further insight into the visual perception and attention of the user, and not as a
standalone tool. The observation of user actions and use errors is still the core point of the formative
usability test, but the use of both traditional ethnographic methods and eye-tracking in combination
could give the development team a much deeper understanding of the reasons and explanations behind the observed user actions and use errors. Furthermore, this could provide better insights into root
causes in use errors as expected by the FDA and other regulatory authorities.
The study focuses on two scenarios or use cases with a medical device: (1) how the user correctly
identifies the “device status”, and (2) how the user handles replacement of consumables in the maintenance of the device. The identification of the device status includes: (a) to be able to identify if the
device is ready for use or not and (b) to be able to identify if the device needs maintenance or problems
need to be solved before use (and which problems to solve, if any).
The analysis made by the team of psychologists showed seven main insights and findings:
Evaluation The seven insights and findings described under “Results” was presented to the manufacturer’s team for comments and evaluations concerning three issues:
1. It was evaluated if each insight and finding also had been identified by the manufacturer’s team in their
own analysis or if it was a results of the use of eye-tracking as additional method apart from the traditio
nal ethnographic methods.
2. It was evaluated how significant, valuable or relevant each of the seven insights and findings were according to the manufacturer’s evaluation in relation to the experimental design of the usability test, the
practical administration of the test, the scenography, instructions to participants etc.
3. It was evaluated how significant, valuable or relevant each of the seven insights and findings was according to the manufacturer’s evaluation in relation to the design process, prevention of use errors etc.
1. What are the pros and cons of the use of eye-tracking in usability tests?
2. How can eye-tracking used in combination with conventional methods such as observations, interviews and verbal self-reporting (composite methods) support the understanding of the user’s visual
attention and visual distraction during use?
3. Will eye-tracking used in combination with conventional methods (composite methods) provide faster and more efficient usability tests with more details, more insights and better understanding of for
example root causes of use errors?
The scenarios described above were tested as part of a formative usability test with ten participants.
The test took place in February 2017 according to normal schedule for the product development process. A small pilot study with two participants was carried out prior to the test and these results are
included in the study.
The medical device is intended for use by health care professionals. Twelve participants were recruited
from other departments in the manufacturer’s organization and five of the participants had prior experience with similar devices.
Manufacturer’s team
The test was conducted according to normal practice for the medical device manufacturer and with
use of the ethnographic methods normally used, including observation, verbal self-reporting and interviews. This process included three observers from the manufacturer (manufacturer’s team). The
manufacturer’s team carried out an analysis based on their data collection.
Work in progress
The study is presented as work in progress and is conducted by Department of Applied Psychology at
FORCE Technology in cooperation with the medical device manufacturer.
Figure 1: Psychologist Jeanette Jakobsen wearing Tobii Pro Glasses 2
Confidentiality
The name of the manufacturer and the product in development cannot be mentioned in this abstract
due to confidentiality reasons during the product development process.
Composite methods and The PCA Model
The idea about composite methods including both ethnographic methods and data collection based
on visual perception (eye-tracking) is based on The PCA Model (perception, cognition, action) of user
behavior. Some methods such as observation of behavior can give us information about “action”.
Other methods such as eye-tracking can give us information about “perception”. And “cognition”
can be accessed through the use of verbal self-reporting (e.g. “thinking-aloud” reflections) and
interviews. However, in the regular design of formative usability studies, the medical device manufacturer now uses only observations (including video observations), self-reporting and interviews
to gain access to the perception of the user, e.g. what the user reports she or he has perceived. We
know that this can be heavily influenced by socially desirable responses (giving answers perceived
as appropriate) and memory-related issues (what is remembered, e.g. influenced by short-term
memory span width, primacy-recency effect or false memories) as well as the fact that very short
perception episodes and perception during automated actions might happen at a non-conscious
level. Furthermore, participants are not asked about their experience or cause of difficulty in using
the device between tasks but rather in a post-test interview. In relation to this, it is the assumption
that the use of eye-tracking will provide data about very short visual perception episodes and/or
episodes that are part of automated behavior and therefore difficult for the user to report during
self-reporting or interviews.
Classification of results
The participants started looking at the device already when they
received instructions for the usability test
Better understanding of visual distractions during use
None of the participants looked at an important button (fixation
point 5 in figure 2) when first visiting a particular screen
Additional information
Usability test
Participants
Research questions
Insights and findings
Team of psychologists
The test was also observed by a team of psychologists from FORCE Technology who used eye-tracking
technology (Tobii Pro Glasses 2 and iMotions 6.2.5 software) to investigate and record details about
the participant’s visual perception and attention during the use scenarios. The team of psychologists
performed an additional analysis based on the eye-tracking recordings.
Comparison of results
The results from the analysis made by the manufacturer’s team and the results of the analysis made
by the team of psychologists using eye-tracking were compared and evaluated to identify additional
information and results gained from the use of eye-tracking.
Evaluation parameters
The five most important parameters for evaluation (criteria for success) are:
A. FASTER RESULTS: Eye-tracking will support faster results (e.g. insights available at a very early stage
during the usability test, and comments from the participant can be obtained in an interview right after the
test).
B. LEVEL OF DETAIL: Eye-tracking provides deeper details of the perception and attention of the user compared to e.g. the user’s self-reporting.
C. ADDITIONAL INFORMATION: Eye-tracking provides additional information not obtainable by means of
traditional methods, e.g. additional information about root cause of use errors and additional information
about the user’s attention to important indications and call to actions from the device.
D. BETTER UNDERSTANDING OF VISUAL DISTRACTIONS DURING USE: Eye-tracking provides a more valid
representation of visual distractions than self-reporting of distractions which is typically “filtered” through
for example social desirability (it is not socially acceptable to be easily distracted during work) and impression management (it is considered more professional to be non-distractible).
E. TO WHAT EXTENT ARE USERS INFLUENCED BY THE EYE-TRACKING TECHNOLOGY ITSELF, e.g. change
of focus, enhanced attention, enhanced awareness, difference in content of self-reports or interviews?
Figure 2: Gaze map aggregating four participants’ use of eye-tracking
FORCE Technology - Department of Applied Psychology - Hjortekaersvej 99 - 2800 Kongens Lyngby - Denmark - Corresponding author: Thomas Koester, [email protected]
Participants with prior experience from other similar devices were
influenced by this experience and thereby showed reduced performance compared to inexperienced participants
Faster results
Additional information
Some participants did not notice an important indication of failure
condition, but after the test they reported that this indication must
have been present and they acted accordingly
Level of detail
Participants with experience did not read instructions on the screen
or skipped the last lines of the instructions
One participant looked only on a product number and ignored an
important product identification symbol
Some participants tried to match the product number shown on the
screen with the product number on the packaging of consumables.
Furthermore, they spent much time on this, and in their effort to do
so, they missed the correct match of product identification symbol
(different packaging share almost the same product number and the
product discrimination should be done on basis of the product
identification symbol. There is a risk that users select a wrong box
with consumables and fail to notice the small difference between
product names)
Faster results
Level of detail
Additional information
Faster results
Additional information
Level of detail
Better understanding of visual distractions during use
Table 1: The insights and findings corresponds to the five most important parameters for evaluation (criteria for success) as
identified under “Method”. Parameters A-D are indicated in the table and parameter E was evaluated by means of interviews
after each session, and the results are described below.
Influence from eye-tracking
Nine of the twelve participants were interviewed after their sessions. They were asked how they experienced the use of eye-tracking and how the technology affected them in the situation. The interviews
were based on three questions:
1. WAS IT UNCOMFORTABLE WEARING THE GLASSES? The majority of participants (7) reported
no discomfort, one participant reported some discomfort and a single participant reported that the
glasses were ill-fitting.
2. DID YOU NOTICE THE PRESENCE OF THE GLASSES DURING THE TEST? Five participants reported that
they did not notice the glasses during test and four participants reported that they noticed the glasses
e.g. “In the beginning you can feel they are there” and “The edge is a bit annoying”.
3. DID THE GLASSES AFFECT THE WAY YOU INTERACTED WITH THE DEVICE? Almost all participants
(8) reported that the glasses did not affect their interaction with the device. A single participant reported “Maybe a little because I was thinking about where I was looking”.
Significance, value or relevance
Five (marked *) of the seven insights and findings were not identified in the manufacturer’s analysis
based on traditional ethnographic methods. They can therefore be attributed to the use of the eye-tracking method (evaluation issue 1). The manufacturer’s evaluation of the significance, value or relevance
is presented in the table below:
Insights and findings
Issue 2
Issue 3
* The participants started looking at the device already
when they received instructions for the usability test
Very low
Very low
* None of the participants looked at an important button (fixation point 5 in figure 2) when first visiting a particular screen
Very low
High
Participants with prior experience from other similar devices were influenced by this experience and thereby showed
reduced performance compared to inexperienced participants
Low
Very low
* Some participants did not notice an important indication of
failure condition, but after the test they reported that this indication must have been present and they acted accordingly
None
Low
* Participants with experience did not read instructions
on the screen or skipped the last lines of the instructions
Low
Very high
* One participant looked only on a product name and
ignored an important product identification symbol
Very low
Some
Some participants tried to match the product name shown
on the screen with the product name on the packaging of
consumables. Furthermore, they spent much time on this,
and in their effort to do so, they missed the correct match
of product identification symbol
Very low
Very low
Table 2: Manufacturer’s evaluation
Conclusions
The use of eye-tracking provided five unique insights and findings. Although the evaluated significance
of them varied, a couple of findings stands out as important. This implies that the use of eye-trackers
can indeed contribute positively to the results obtained from a usability test based on traditional
ethnographic methods. It should be noted that use of eye-tracking requires additional time,
resources and technical skills including optimal light conditions. However, with the promising perspectives in mind, eye-tracking is recommendable as an additional tool for usability studies.