Experiencing interactive healthcare technologies:
embracing “the wild” on its own terms
ANN BLANDFORD, UCL
ERIK BERNDT, UCL
KEN CATCHPOLE, Cedars-Sinai Medical Centre
DOMINIC FURNISS, UCL
ASTRID MAYER, Royal Free Hospital
HELENA MENTIS, Microsoft Research
AISLING ANN O’KANE, UCL
TOM OWEN, Swansea University
ATISH RAJKOMAR, UCL
REBECCA RANDELL, University of Leeds
We highlight challenges of studying the use of interactive medical devices in “the wild”, in hospitals and
homes: gaining access; being outsiders in intimate spaces; focusing attention on technologies that are of
limited interest to their users; and being with people whose lives and health depend on successful
technology use. Drawing on case studies across hospitals and homes, we present a repertoire of ways of
engaging with the “actors” in these settings, to understand their interactions with devices, their
experiences of device use, and the contextual factors that shape those experiences. These include working
with clinicians as researchers, engaging patients effectively, taking an iterative approach to data
gathering, and planning the study design carefully and assertively. Our intention is that this paper will be
a valuable resource for researchers embarking on studies in healthcare, and will convey some of the
benefits of working in this setting.
Categories and Subject Descriptors: H.1.2 [User/Machine Systems]: Human Factors; H.5.2 [User Interfaces]: Evaluation
/ Methodology
General Terms: Design, Human Factors
Additional Key Words and Phrases: in the wild, situated studies, healthcare, evaluation
ACM Reference Format:
TBC
1.
INTRODUCTION
Healthcare is evolving in many ways: it is becoming increasingly reliant on interactive
technologies; more healthcare is being delivered outside clinical settings; and people are being
expected to take more responsibility for their own health. While it is possible to do usability
studies of healthcare technologies under controlled conditions, these studies give little
understanding of the experience of using such technologies in their normal contexts of use: often
under pressure of work (for clinical staff) or in stressful situations (for patients). Typically,
The work of Blandford, Berndt, Furniss, Mayer, O’Kane, Owen and Rajkomar is supported by EPSRC EP/G059063/1
(CHI+MED). Catchpole is supported by Department of Defense Grant W81XWH-10-1-1039.
Author’s addresses: Ann Blandford, Erik Berndt, Dominic Furniss, Aisling Ann O’Kane and Atish Rajkomar, UCL
Interaction Centre, UCL, Malet Place Engineering Building, Gower Street, London WC1E 6BT, U.K.; Ken Catchpole,
Department of Surgery, Cedars-Sinai Medical Centre, Los Angeles, California, USA. 90048; Astrid Mayer, Department of
Oncology, Royal Free Hospital, Pond Street, London NW3 2QG, UK; Helena Mentis, Socio-digital Systems, Microsoft
Research, Cambridge CB5 8HT, UK; Tom Owen, Future Interaction Technology Laboratory, Swansea University,
Swansea, SA2 8PP, UK; Rebecca Randell, School of Healthcare, Baines Wing, University of Leeds, Leeds LS2 9JT, UK
Permission to make digital or hardcopies of part or all of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the
first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others
than ACM must be honored. Abstracting with credits permitted. To copy otherwise, to republish, to post on servers, to
redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee.
Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701
USA, fax +1 (212) 869-0481, or [email protected].
@2010 ACM 1539-9087/2010/03-ART39 $15.00
DOI10.1145/0000000.0000000 http://doi.acm.org/10.1145/0000000.0000000
1
clinicians, patients and carers will all have some responsibility for, and modes of engagement
with, interactive health technologies. It is essential to design novel interactive technologies for
healthcare with a good understanding of the users and the contexts of use, and with some
understanding of how the proposed technology is likely to be used, and how it will change the
context as it is appropriated [Carroll et al, 2002].
Because of the complexity and safety-critical nature of healthcare, it is often not possible to
deploy early prototypes without going through a time-consuming validation process. So for many
interactive healthcare technologies, “in the wild” has to mean studying existing use, rather than the
use of new technologies, but in the spirit of engaging with the actors and their rich, situated use of
the technologies that are essential to their work and their wellbeing. This is consistent with the
early use of the term “in the wild”, such as the work of Hutchins [1995]. It is, at least superficially,
less consistent with the use of the term as applied in the call for this special issue, which focuses
on “creating and evaluating new technologies in situ”. Rogers [2012, p.73] discusses how “in-thewild studies show how people come to understand and appropriate technologies in their own terms
and for their own situated purposes”. The studies on which this paper is based are consistent with
this spirit: understanding not just how clinicians work and patients live their lives, but how they
adopt, adapt and appropriate the technologies they engage with. It is in this sense that these studies
are “in the wild”. As will emerge below, it is also “wild” in the sense that it presents many
challenges that are not faced by researchers studying environments that have clearer structures, or
where participants are well and not almost constantly multitasking.
In this paper, we aim to:
• sketch the space of current technology studies in healthcare;
• highlight challenges of conducting “in the wild” studies in high risk and sensitive
environments;
• share experience and best practice of doing situated research in healthcare; and
• facilitate future HCI research in healthcare.
2.
BACKGROUND
Many earlier situated studies have reported on methodology. For example, in some of our earlier
work [Blandford and Wong, 2004; Furniss and Blandford, 2006], we reported on the use of
Contextual Inquiry [Beyer and Holtzblatt, 1998] and the Critical Decision Method [Hoffman et al,
1998] in the study of ambulance control. However, such study contexts are less wild than the
healthcare contexts we report on and reflect on here: there is a spectrum of wildness, in terms of
variability and challenges of working in the setting. Here, we briefly summarise previous reports
of methodologies used at the wilder end of the study spectrum, focusing specifically on studies of
technology use. We also discuss the challenges and possibilities of introducing novel prototypes in
healthcare.
2.1 Methodologies reported from previous “wild” studies
Previous “wild” studies in clinical settings report using ethnographic methodologies, with a focus
on observations and interviews. The major strengths of an observational methodology are the
ability to collect data on the tasks actually carried out as opposed to prescribed procedures and
manuals, and the generation of rich, detailed data [Carayon et al., 2005; Randell, 2003; Seagull
and Sanderson, 2001]. These studies used observational methods such as time study and flow
process charting [Carayon et al., 2005], process-tracing, to construct behavioural protocols [Cook
and Woods, 1996; Seagull and Sanderson, 2001], and cognitive task analysis [Cook and Woods,
1996]. Observation sheets were used to record notes in all of these studies, and later transcribed.
For most of the studies, special observation sheets were developed during the first few
observations. Carayon et al. [2005] report that the observers in their study were “complete
observers” who did not participate in any way in the process being observed, and Randell [2003]
reports that her observations were unobtrusive. Besides observations, researchers conducted
interviews with participants to: improve their understanding of particular events in a complex
setting; supplement observation data; ask for clarifications on actions performed by participants or
devices that appeared out of the ordinary interaction; and validate observation data. These
2
exchanges with participants happened either when auditory alerts indicated that an error had
occurred, or during a particular phase of an observed intervention, or informally during coffee
breaks and quiet moments. Researchers also took other measures to deepen their understanding of
the context, such as attending training sessions on the devices and attending meetings of nurses
and doctors. To help make sense of observation data, they also consulted system manuals [Cook
and Woods, 1996] and other medical documents related to the procedures being observed [Seagull
and Sanderson, 2001]. Seagull and Sanderson [2001] reflect that, although their data collection
mechanism provided a comprehensive perspective on the procedures observed, audio-visual
recording would have improved the richness of data. They were limited to focusing on events
denoted by auditory alarms; audio-visual recording would have allowed them to capture events
denoted by visual alarms as well. Audio-visual recording has been useful in the capture of
complex, dynamic safety-critical healthcare work where risks can quickly escalate and no other
records exist [Catchpole et al. 2006; Catchpole 2011].
For conducting studies in the home, Blandford et al. [2009] reflect on the lack of welldeveloped methods for data gathering. They emphasise that observational work in homes presents
a special research challenge in terms of the efficiency, effectiveness, privacy and ethical issues of
data gathering and analysis. Previous studies of home healthcare technologies have combined and
triangulated several sources of data. Lehoux [2004] had three sources of data: interviews with
patients, interviews with carers, and direct observations of nursing visits. Obradovich and Woods
[1996] conducted three kinds of investigations: interviews with nurses about how nurses and
patients used the device; bench tests that explored device behaviour, representations of states and
activities, and control sequences; and observations of nurses programming the device. Obradovich
and Woods [1996] conducted the investigations in an iterative and intermixed fashion, with one
type of investigation informing or setting the stage for another. Lehoux [2004] and Obradovich
and Woods [1996] used interviews differently. The former used biographical interviews to
examine coping strategies, and to elicit how patients and carers perceived the technology and how
their lives were transformed because of technology use. The latter used interviews with nurses to
specifically examine how users learnt to train, inform, and proceduralise tasks so that the infusion
device could be used despite its HCI deficiencies. Obradovich and Woods [1996] point out that
due to lack of organizational support, their ability to collect and report more kinds of data, e.g. the
analysis of actual incidents and the observation of patients during training and device use, was
limited. In another study, Kaufman et al. [2003] evaluated the usability of a telemedicine system
in patients’ homes by recording video data of the participants and of the system’s screen displays.
They analysed the video data at different levels of granularity to understand participants’
interactions with the system.
While these studies report on methods, they do not dwell particularly on challenges and
strategies for overcoming them. Siek and Connelly [2006] is unusual in reporting lessons learnt
while working with haemodialysis patients in a hospital; as well as noting the need to check
regulations (e.g. about data recording) and practicalities (e.g. how to organise research instruments
within the study space), they also report on the challenge of ensuring that paper prototypes
conformed to infection control procedures (being laminated and routinely disinfected); they note
the need to be sensitive to patients’ needs, to dress appropriately, and to be flexible and adaptable
in data gathering. We expand on many of their themes below. However, many of our studies have
been on the use of monitoring and drug administration technologies, and the safety-critical nature
of these devices affects the practicality of deploying early prototypes in healthcare.
2.2 Designing and deploying novel healthcare technologies: the practicalities
Within healthcare, there is a paradox: that devices that are formally classified as medical devices
have to go through a rigorous approval process before being deployed, whereas other devices (e.g.
calculators and networking technology) can be used in the medical context, for purposes that are
safety critical, without going through such accreditation processes. Within Europe1, the definition
of a medical device includes that it is “intended by its manufacturer to be used specifically for
1
http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1993L0042:20071011:en:PDF
3
diagnostic and/or therapeutic purposes” (p.5). In the USA, the Food and Drug Administration
(FDA) definition2 includes the statement that a medical device is one that is “intended for use in
the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of
disease, in man or other animals”. The details of pre-market approvals3 (in the USA), conformance
with the Medical Devices Directives (Europe), or the equivalent in any other jurisdiction, are
outside the scope of this paper. However, it is important to recognise that technologies that are
deployed “in the wild” in healthcare vary in terms of the validation they must be subjected to
before they can be deployed. The pre-market approval processes that are mandated for the infusion
devices, glucometers and haemodialysis machines that have been the focus of many of our studies
mean that they cannot be developed and tested “in the wild”, in the sense outlined in the call for
papers for this special issue. Even minor amendments to hardware or software of established
products are subject to scrutiny through certification processes that vary from country to country.
In the UK, there is an important exception to this situation: a health institution, such as a NHS
trust, is permitted to design and use a device in-house on their patients without requiring medical
device approval4. Thus, if you partner with a hospital’s research team, you may be able to design,
develop, and test a system in a hospital without medical device approval first as we did with the
development and field trial of a touchless display for surgeons.
Because of the challenges in deploying novel healthcare technologies, evaluations of new
technologies that arise out of “in the wild” studies tend to be conducted in simulated environments
[Favela et al., 2010; Dahl et al., 2010]. Where technologies have been introduced and
subsequently evaluated “in the wild”, they are typically lightweight technologies that do not
anticipate serious threat of disruption to the ongoing work. For example, Wilson et al. [2010]
describe the introduction of technology to support real-time information sharing between a
paediatric ambulance service and paediatric Intensive Care Unit (ICU) that supplemented existing
practices for information sharing rather than replacing those existing practices. “In the wild”
research has been combined with participatory design (PD) for the introduction of technologies to
support coordination of surgical operations [Bardram, 1998; Bossen and Jensen, 2008]. Drawing
on the lessons of PD, researchers have used the approach of co-realisation for introducing
technologies into medical settings. The emphasis in co-realisation projects is on “simple”
technology, in the sense that the system should only provide the necessary functionality and that,
where possible, use should be made of technologies that are already available; the intention is not
to provide “bells and whistles” but to support work. An important aspect of the co-realisation
approach is having an IT facilitator who not only installs the system and provides training but
continues to visit the site [Hartswood et al., 2002]. As new requirements emerge, the facilitator
can customise the system to incorporate them. For example, in a three year project in a toxicology
ward, use of an off-the-shelf speech recognition system was explored for producing discharge and
transfer letters. The technology was found to be difficult to use and was eventually abandoned.
However, the ward staff’s move to typing was not treated as a failure but as “a practical, situated,
members’ choice of a work-affording artefact” [Hartswood et al., 2002, p.285]. What such studies
highlight is that what we as HCI researchers typically consider to be straightforward technologies
may present challenges for healthcare professionals integrating these technologies into their work.
3.
METHOD
This paper is not the outcome of a single study, but of the authors reflecting on experiences of
conducting studies “in the wild” in healthcare. As well as our own experiences, we also report on
studies conducted by students under our supervision. Consequently the form of this paper is
similar to a confessional ethnography: where researchers reflect on their relationship with people
in practice, how they conducted themselves in practice, and how this affected data gathering and
2
http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/Overview/ClassifyYourDevice/ucm051512.htm
3
http://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/DeviceApprovalsandClearances/PMAApprovals/
default.htm
4
http://www.mhra.gov.uk/Howweregulate/Devices/Inhousemanufacture/index.htm
4
analysis, to demystify the fieldwork process [Rode, 2011]. Below, we provide a brief summary of
the studies on which our findings are based. Where relevant, we include initials to indicate which
author(s) were involved in a particular study (e.g. TO=Tom Owen; AOK=Aisling O’Kane).
Themes for this paper were initially identified through discussion between three of the authors
(AB, DF, AR), including both broad themes (e.g. challenges of access, respecting privacy, and
managing potential patient and clinician concerns) and vignettes (stories of incidents that
illustrated or teased out points that we identified as being important given the aims of this paper).
As we recognised important gaps in experience, we recruited further authors, who validated the
narrative to date and extended it.
Themes were further developed through brainstorming sessions between the authors. It became
apparent that there were important differences between studies that involved clinical professionals,
where patients were involved by virtue of their treatment, and studies where the main engagement
was with people with long-term conditions (such as diabetes or kidney disease) and their lay carers
(typically family members). At the time of writing, this is a good reflection of practice: most
technology use in hospitals is by clinicians and most technology use in other settings is by lay
people (quadrants HC and NP in Figure 1). However, the boundaries between these kinds of use
are becoming less well defined, and we anticipate greater use of technology in quadrants HP and
NC in the future.
Situations
Users
[P]
Patients
Lay carers
[C]
Clinicians
[H]
Hospital
HP
HC
[N]
Non-clinical
(Home, Mobile)
NP
NC
Fig. 1. Situations and users.
We present the planning stages of studies in both clinical and non-clinical settings together and
then report findings of more detailed engagement between these two settings separately.
3.1 The studies on which this paper is based
The authors of this paper bring a range of experiences of studies in healthcare, including:
• patient safety and human factors assessment without a focus on any particular
technology;
• understanding particular clinical procedures and designing technologies to better fit
those procedures;
• the design and use of particular technologies and how they are adopted and adapted
for use in context (without deploying novel prototypes, for reasons presented
above); and
• designing, deploying and testing novel technologies with clinicians.
Focusing on human factors assessment, KC has conducted studies in surgery. Here, we have
focused on the close relationship between task, team and technology [Catchpole 2011]; the
frequently disruptive and occasionally risk-inducing events that signify a mismatch between
components of the work environment [Catchpole et al. 2006]; the different technological
5
requirements and risks for different types of surgery [Catchpole et al. 2007] ; and the use of nontechnical skills training as a method for addressing these risks in lieu of better systems
development [Catchpole et al. 2010].
We also draw on studies we have undertaken of existing work practice: in a hospital
histopathology department to inform the design of a virtual reality microscope [Randell et al.,
2012]; in Medical Equipment Libraries across different hospitals to inform best practice [Werth
and Furniss, 2012]; and across diverse hospital settings, including a general medical ward,
emergency assessment unit, paediatric surgical ward, and paediatric ambulance service, to inform
the design of technologies to support clinical handover [Randell et al., 2011a; 2011b].
In our technology-focused studies, the two main technologies are infusion devices and
haemodialysis machines, but our studies have extended to other devices (such as oximeters and
blood gases analysers) within hospitals and homes. We have restricted our focus to interactive
medical devices, and considered other devices (e.g. phones or washing machines) only insofar as
their use influences the use of medical devices.
Infusion devices are programmable pumps and syringe drivers that deliver liquids (e.g.
chemotherapy drugs or analgesics) to patients at a controlled rate. We have studied their use in an
ICU [Rajkomar and Blandford, 2012; Randell, 2003]; an Oncology Day Care Unit [Furniss et al
2011a; 2011c; 2011d]; a Haematology Ward [Gant, 2011]; an oncology ward; and accident and
emergency (studies ongoing). In some of these studies, we have taken a particular theoretical
perspective; for example, we have made use of Distributed Cognition [Hollan, Hutchins and
Kirsh, 2000] as a theoretical framework to guide both data gathering and analysis. In other studies,
we have avoided taking such a perspective, but have followed an approach more akin to Grounded
Theory [Charmaz 2006], with data gathering and analysis guided by questions such as how the
technology is used, what errors people make, and what strategies they have developed to maximise
their effectiveness.
In a study of blood gases analysers, O’Connor [2010], supervised by AB, observed and
interviewed users of blood gases analysers in Accident and Emergency and in an ICU with a
particular focus on the work-arounds (i.e. non-standard practices) that staff had developed, and the
causes and consequences of those work-arounds.
Home haemodialysis machines are used to clean patients’ blood and to remove excess fluid
from their bodies. Our interest has been in how people shape their lives around the technology
and, conversely, how other contextual factors shape technology use.
We are currently conducting investigations into the use of patient-centric mobile medical
technologies such as glucometers, and how these can be studied in context in order to gain insight
into patient experience. To gain insight in the everyday practices of users, interviews with a wide
range of patients and diabetes specialists have been completed. These interviews have probed
patients’ everyday lives and experiences of caring for the condition, and their information needs
[O’Kane and Mentis, 2012].
Finally, we have conducted studies on the design and use of devices that anaesthetists use
(ongoing), and the design, deployment and assessment of a touchless display for surgeons [Mentis
et al. 2012].
We have structured the paper according to key themes that have emerged: planning studies and
negotiating access; strategies for studying technology use in hospitals; strategies for studying the
use of healthcare technologies outside formal healthcare settings; and the experience of studying
technology use in healthcare.
4.
PLANNING STUDIES AND NEGOTIATING ACCESS TO WORK IN HEALTHCARE
There are many triggers for HCI research projects in healthcare settings. We have summarised the
kinds of studies of which we have experience above. These projects have had different paths in
planning the research and negotiating access. Some have taken many months, whereas others have
taken only a few weeks to negotiate.
There are many variables in planning and conducting studies. These include variability in the
motivations for conducting a study, the expertise of the team, the familiarity of the health service
organisations involved with research and ethics procedures, the clinical setting and the
technologies involved.
6
4.1 Different starting points: the coupling between the researcher and the context
For clinicians studying technology use in their own environment, the very first steps – of
identifying the study environment(s), the focus of the study, and the key healthcare professionals
to work with – may be very straightforward. Human Factors specialists face a choice: whether to
remain as outsiders to the health domain or to embed themselves within the healthcare context and
work more closely with clinicians. Two of the authors of this paper (KC and RR) have chosen the
latter, while others have chosen to keep their base in HCI. Those who are not based in the setting’s
administrative structure have more barriers to building links into the healthcare context; but
conversely those outside have more freedom to select study settings for addressing research
questions.
In the following sections, we structure the discussion of how to plan a study in terms of PRET
A Rapporter, a framework for considering the purpose, situational factors, ethical concerns and
methods to be applied in conducting and reporting a study [Blandford et al, 2008].
4.2 P: the Purpose of a study
In many cases, the first step is to define the purpose of a study; we have outlined many examples
of purposes in the background and methods sections above.
For HCI researchers outside a particular health context, there can be an element of luck in
choosing settings and foci for “in the wild” studies. There is a risk of choosing to focus attention
on something that actually doesn’t happen very often, making the study time-consuming or
yielding limited data. For example, O’Connor [2010] was particularly interested in work-arounds,
and was keen to conduct a study in Accident and Emergency (AandE). Because his was a short (3
month) project, we needed a clearer focus for his study. We worked with the Matron in AandE in
one of the local hospitals to identify the focus, keen to identify a project that would be of interest
to the staff there as well as to ourselves. At the time, they were in the process of replacing the
blood gases analyser; the previous model had several recognised deficiencies, and she wanted to
know whether the replacement model was easier for staff to use. In practice, the new model was
found to be so much easier to use that staff did not develop many work-arounds with it,
minimising the data available through the study. For this reason, a comparative study in the ICU
was added.
There can be great value in having multiple study settings. This helps to highlight aspects of
design or use that might otherwise go unnoticed. For example, by studying the use of identical
machines in two different environments (AandE and ICU), O’Connor [2010] identified features of
the environment of use (how easy it was for unauthorised people to physically gain access to the
machine) and of the management of the machine (in terms of how closely its use was monitored
by a responsible individual) that had a large bearing on how the machines were used in practice,
and hence of how well they were designed to fit these different contexts of use. Such benefits are
also found when studying current work practices in order to inform the design of novel
technologies. For example, studying handover practice across a variety of clinical settings revealed
not only that the term “handover” captures a variety of collaborative practices that vary in both
form and content, but also how the level of heterogeneity amongst those participating in the
handover impacts how the handover takes place [Randell et al., 2011c].
While HCI research in hospital settings has to some extent neglected the role of the patient,
patient and public involvement (PPI) is gaining increased prominence in health service research
and is a requirement for some research funding. PPI involves those with experience of being a
patient or carer in designing or undertaking research in such settings. PPI can be really useful for
the HCI researcher wanting to undertake research on medical technologies. Patients know these
settings really well, so can advise on aspects of undertaking the research, particularly about how
and when to approach patients for consent. They can give feedback on the clarity of patient
information sheets, assist in writing a ‘lay summary’ for research funding proposals, and help to
prepare summaries of the research findings for non-academic audiences.5
5
Advice on involving patients and the public in health research is available from http://www.invo.org.uk/.
7
4.3 R: Resources and constraints
Any study has to be designed to work with the available resources and to fit within practical
constraints. In healthcare, constraints typically include the time available to complete a study;
getting institutional buy-in to conduct it; and availability of particular expertise.
4.3.1
Time
Studies vary in the time taken to prepare for healthcare research and the time taken to do them. In
healthcare, time for preparation is most obviously coupled with complications in negotiating
access and gaining ethical approval. However, time can also be positively used in parallel to
prepare the researcher for observations. For example, KC spent many months learning the
intricacies of a cardiovascular operation before observations in surgery, and HM spent about a
year as a volunteer in a hospital to get acquainted with the environment and to feel comfortable in
it before data gathering. However, timescales and deadlines mean that extended periods of
preparation are not always possible. For example, UK MSc projects (e.g. [O’Connor 2010; Gant
2011]) have to be completed within three months, and other studies have pressures to deliver
results on a shorter time frame that will be familiar to many academics, and practitioners in
industry more so. The study’s focus and methodology depend on the time available. For example,
investigating how implicit and subtle emotional interaction is used in an emergency room’s work
[Mentis, Reddy and Rosson 2010] has quite different research requirements compared to studying
how infusion devices are used in an ICU [Rajkomar and Blandford 2012]. The methodology and
time frame should suit the study’s purpose.
4.3.2
Institutional buy-in
Studies initiated by a hospital insider typically face fewer challenges in getting institutional buy-in
than those initiated by outsiders. As an outsider, if a study involves formal healthcare settings
(either for the study, or for recruiting participants), you need to identify what access is needed and
who manages those settings. Exploring your current contacts to make new relationships is a
promising place to start. In addition, presenting your ideas to a number of people to see who
would most be interested may yield a very strong champion. For instance, for the touchless
project [Mentis et al, 2012], at one of the hospitals we subsequently worked with, we presented at
a meeting attended by a number of surgical departments which led to our working relationship
with the neurosurgery department.
Having a clinician on the project team, or in an advisory role, can be invaluable at this stage to
help identify appropriate settings for the research questions you wish to tackle and to identify who
to talk to. If it is a senior clinician they will carry political weight within the organisation for
making things happen and be able to open doors for the research team. Once contact has been
made with the site manager then one should have a meeting with them to discuss the proposed
study and meet staff and take a tour of the setting to see what it’s like on the ground, to plan how
the research will fit the context. For example, studying medical devices that are used in theatre
where patients are unconscious poses different challenges compared to studying patients on wards
who are conscious; whether wards are open or patients have single rooms will affect how the
researcher engages with them.
It is important to recognise and respect participants’ sensitivities. For example, studies of
human error may be regarded with suspicion, as individuals may feel threatened by this focus, and
may be concerned that a greater reporting of minor incidents will be perceived negatively by
hospital management. It is important to anticipate and allay concerns with assurances of
confidentiality, making it very clear what the research purpose of the study is, and building rapport
and trust with current and future participants at different levels of the organisation.
Occasionally, we have found that potential clinical champions lose interest in the project, or
get swamped by the local administrative processes required to get study approval. In one particular
case, we had made considerable progress in making arrangements with some senior healthcare
practitioners to study how palliative care nurses interact with ambulatory infusion pumps during
visits to patients’ homes. However, all of a sudden we stopped hearing from them, seemingly due
to organisational changes. Until engagement is confirmed, it is a good idea to have a backup plan.
8
Nevertheless, on the whole, we have found health service practitioners to be highly supportive
and enthusiastic to work with us, and the mutual benefits have been enormous. This is a theme we
return to in the discussion.
4.3.3
Expertise: clinical, HCI or both?
As noted above, researchers are often already in place (or you are the researcher yourself), and the
challenge is to define the study and the setting. Alternatively, it may be necessary to recruit
observers to a defined study. The background of the observer influences what is possible, so
recruitment and training plays an important part of the developing methodology. The clearest
delineation is whether to select medically trained or non medically trained observers. The former
are more able to identify the clinical implications of the behaviours, and thus suit clinical
outcomes. The latter are better disposed to understand the systems context and a broader range of
influences on behaviour, and so may understand technology use better. Although what constitutes
sufficient expertise, training and requirements for observation is poorly defined, there is an
emerging literature in this area (e.g. [Russ et al. 2012]).
A risk with clinician-observers, especially where staffing is short, is that they become drawn
into the delivery of the clinical work rather than the observation, and are rarely truly independent.
Conversely, non-clinicians have to maintain distance from the patients, and are at liberty to ask
questions that might appear naive if a clinician were to ask them. There is very little formal
documentation or methodological study in these areas [Catchpole 2009].
As noted elsewhere, healthcare is a highly complex system, characterized by various
interrelated parts, high dynamics and a high degree of unpredictability. Entering a complex system
can be overwhelming at first [Wong and Blandford 2003]. Whereas experts might be able to see a
clearly defined structure, lay analysts often start with a chaotic image [Norman, 2010].
Consequently, an observer won’t be able to develop a coherent understanding of what he observes,
not to mention revealing the underlying patterns. Even worse he may direct his attention towards
less relevant aspects of the environment. Analysing an aspect of the healthcare domain therefore
asks for substantial preparation in advance and reflection during observation to produce useful
results (see section 4.3.1). A preliminary visit of the site helps to develop a broad understanding of
the setting and provides some insights to the medical paradigm with its special features and
language. Likewise, it facilitates an appropriate choice of research focus [Jirotka and Wallen,
2000], and the decision of a suitable data capturing method [Fisher and Sanderson, 1996]. As it is
impractical to study every aspect of a complex domain, the literature often advises us to
modularize complexity, breaking it down into smaller units of analysis which are easier to handle
[Norman, 2010]. Modularization could be achieved in different ways. One method we have
developed and used is the DiCoT framework which has proved successful in handling complexity
in the healthcare domain [Furniss, 2004; Rajkomar, 2010].
4.4 E: Ethics
In most countries, there are processes in place to affirm that research studies meet a high standard
of ethical due diligence, primarily to ensure that patients, staff and researchers are protected. There
are also organisational considerations to be managed such as disturbance to practice, the use of
resources and potential benefits of the study. The processes for formal ethical approval in a
medical or health setting vary worldwide – some countries are very stringent and some countries
have no formal procedures over and above that of a non-healthcare study. For reference, we
include an informal summary of the process in the UK as Appendix A.
Given that HCI “in the wild” studies generally have few ethical implications compared to
clinical trials, the most challenging aspect of obtaining ethical clearance is generally dealing with
informed consent. The default position for getting consent is to get full written informed consent
from everyone who is involved in the study or whose data is engaged with. This includes an
information sheet and consent form, with an extensive briefing to make sure the participant
understands what the study is about, how their data is going to be handled, and know their rights
to opt-out of the study. Best practice for research governance is to give the information to the
participant days before being asked to participate properly so that they have time to think about it
and do not feel pressured. This model of consent is based on intrusive clinical studies that have
9
high consequences for the patient. Through experience we have learnt that as a model for HCI
studies ‘in the wild’ this is often disproportionate and unfeasible. We discuss the realities of
informed consent in hospitals in section 5.1.1.
With the benefit of hindsight and the added confidence that experience brings we believe one
should be clear and assertive about feasible and fitting research processes. The dilemma is that
sometimes we only learn what is feasible and fitting with experience. In this case, an experienced
advisor to the project can be a good source of support, and some form of pilot study could assist
the learning process. As a community it begs the question whether there could be some collective
pooling of best practice to assist better research governance of HCI “in the wild” in healthcare.
Processes should push for gathering appropriate data, while ensuring that patient confidentiality is
not compromised. For example, in a recent project we gained ethics approval to audio record and
video record Multi-disciplinary Teams (MDTs) without consent from the patients who would be
discussed. The research ethics committee (REC) accepted this but key was explaining why it was
not feasible to get consent from patients and what we would do with the data, i.e. explaining that
we were not interested in the details of individual patients and so no patient identifiable data
would be transcribed. Again, experience tells us that ethics committees can differ so this might not
be approved by all; however, again, perhaps a central directory of HCI “in the wild” healthcare
study best practices would provide methodological weight to future studies and even support
RECs who could more easily see that “studies like this” had been previously approved and were
not as novel or risky as they seem. HM has even experienced the benefits of working with an REC
that only deals with non-treatment studies. Working with an REC like this or one who even
specializes in observational studies can greatly improve one’s ability to negotiate a suitable
informed consent process. Our subjective view across the health services, is that ethics processes
are getting more proportionate; for example, it is now widely agreed that full ethics clearance is
not needed for studies that involve only staff and not patients.
Finally, it is important to understand the relevant local procedures and requirements for what is
typically called Research and Development (R&D). In many organizations, there are local experts
in R&D, who can help with navigating the procedures, and who have the authority to determine
the best route through ethics and access. The UCL team have benefited enormously from such
guidance from a key individual in R&D who listens, understands research pressures and helps, and
with whom we now have an established working relationship based on mutual respect. Just as we
remarked that a clinical champion can greatly assist a study, so can an R&D champion.
While the requirements for ethical approval vary between countries, a general recommendation
is to be clear about why you want to use particular data collection methods and the consequences
of those for the robustness of the findings. RECs want to make sure that good research gets done
so if, for example, video recording is the only way that you will gather the necessary detail, make
that argument clearly. At the same time, you need to show that you have thought about the
consequences of your data collection methods for the people in the setting, what their concerns
may be, and what you will do to overcome those concerns. Similarly, it is important to be clear
about the consequences of your approaches for gaining consent, both for the people whose consent
is being sought and the researcher. Gaining written consent from all patients who you hear being
discussed may be infeasible, require so much time that data collection is negatively impacted,
and/or mean approaching patients at what is already a distressing time. In such cases, you can
make an argument for not gaining written consent but you need to be clear about how you will
protect the privacy of the patients being discussed, such as ensuring patient identifiable data is not
transcribed and deleting the sound from video recordings after a certain time period. In planning
your methods of data collection and obtaining consent, talk to those in the setting about what they
consider to be feasible and get advice from PPI groups about how and when to approach patients
for consent. Because of differences between ethics committees, in the UK at least, it is worth
talking to other researchers in your institution about their experience of different ethics
committees.
4.5 T: Techniques for data gathering
Techniques for data gathering (e.g. interviews, observation, autoethnography) are discussed in the
following sections, focusing separately on studies in hospitals and outside clinical settings.
10
4.6 A: Analysis techniques
Techniques for data analysis are not significantly different in healthcare from other contexts, so
are not discussed in this paper.
4.7 Rapporter: reporting the findings
In addition to the usual challenge of reporting on “in the wild” studies, there are particular
challenges to reporting on healthcare studies. One is having due regard for the readership (which
might be, for example, HCI, medical informatics, or clinicians); another is having due regard for
participants.
To maximise impact, it is important to report findings to clinical audiences as well as
technology / HCI audiences. Clinicians’ interest in evidence-based medicine, and a focus on
outcomes-related research, makes reporting to the clinical audience challenging. The idea that
process and outcome may not be closely related, when evidence-based medicine relies on
understanding those differences, may be difficult to impart and a failure to include “p-values”
makes some clinicans consider HCI research to be less credible than quantitative, outcomes-based
research. Among clinicians, positivism is still a dominant philosophy. It is important to justify the
value of the questions being addressed and the legitimacy of the approach being taken.
We also need to be aware of the professional sensitivities we unearth when we begin to
expound the relationship between humans and systems to an audience who have been taught to
believe that only bad doctors or nurses make mistakes. The idea that equipment, processes, teams,
systems of work and organizations can have effects on performance that are beyond conscious
control can be disturbing for professionals whose confidence is paramount to their work, and who
believe it all comes down to what they do. Realising that they do not have control over things that
might make them have an adverse event is personally and professionally challenging. For
example, in one conference presentation, one of us was heckled by a clinician with a selfproclaimed expertise in Human Factors who, when presented with a photograph of an errorinducing design proclaimed that they “had never made that mistake in 20 years” even though it
had been brought to the attention of the researcher by clinical colleagues who had made exactly
that mistake.
We should therefore be aware not only that our presence as individuals in hospitals may be
alien; but our very philosophies are challenging the history, current teaching, and perceived
professional status of many professionals. In this sense, learning to communicate our findings
across this divide - and ultimately publish our research in respected clinical journals – is essential
to bring our worlds together and have long-term impact on practice.
5.
HEALTHCARE IN THE CLINICAL SETTING: CHALLENGES AND STRATEGIES FOR
CONDUCTING STUDIES IN HOSPITALS
We have identified two core themes that make “in the wild” studies of healthcare technologies
challenging: the first is engaging with the people who are in that context; the second is around the
pragmatics of gathering data.
5.1 Engaging with patients and professionals in the context
The fact that people in hospitals become participants by virtue of their roles (and so effectively opt
out if not happy to participate) means that they are less “voluntary” than participants in many other
settings. There is consequently great variability in the level of engagement of participants. This
puts an onus on the researcher to ensure that informed consent is respected, although this may
deviate in details from the planned approach to gathering informed consent as anticipated at the
outset of the study.
5.1.1
Informed consent in practice
The principles of gathering informed consent as part of the ethical clearance process have been
discussed above (section 4.4). In principle, informed consent should be obtained from every
participant in a study. In practice, we have found ourselves in situations where we have 3 or 4
minutes to introduce ourselves to the whole team, and individuals did not see the point in reading
11
information sheets or signing official consent forms. O’Connor’s [2010] experience was fairly
typical: he was introduced to all the staff who regularly interacted with the blood gases analyser
that was the focus of his study prior to the study beginning. He then sought permission from each
member of staff to observe each interaction with the analyser. All gave permission; some then
ignored him; some carefully talked him through what they were doing with the machine; and
others chatted about unrelated topics.
Obtaining informed consent from patients in the hospital setting can prove more challenging.
For example, Gant [2011] conducted studies during the night as well as the day; nurses were
concerned that sleeping patients could not give informed consent to be observed, so it was agreed
that consent would be obtained before patients went to sleep to enable her to observe nurses
interacting with infusion devices in their rooms during the night. In an earlier study in the
operating theatre, KC was required to consent the parents of the paediatric patients being studied
for permission to audio record and video record their operation. Due to challenges in finding the
parents (who could be anywhere in the hospital), and the sensitivities required toward their
extremely sick children, this process could take several hours on the night before the operation,
and might be entirely invalidated if one of the surgical team did not give their consent the
following morning, which sometimes happened. In the same study, another set of patients were
identified and consented at pre-operative clinic several weeks before their operation, yet many
were cancelled, or rescheduled and their cases were not observed. As a result approximately 50%
of patients consented, at considerable time and expense, did not eventually contribute to the study
[Catchpole et al. 2005].
We previously alluded to our well intended approaches to obtaining consent from staff and
patients, and how we found them disproportionate and infeasible in practice. We also found that a
particular difficulty is in settings where there is frequent movement of patients. In the handover
study, when observing in the medical assessment unit where patients typically stay for less than 24
hours, the researchers had to spend a lot of their time seeking consent from patients, to the extent
that it distracted from the data collection. In the same study, when observing the work of the
paediatric ambulance service, the nature of the work meant that it was necessary to seek consent
from parents to observe handovers concerning their child when they had just learnt that their child
was critically ill and needed to be transferred to a paediatric ICU. No parent objected to being
approached for consent and all were happy to be involved in the research, but certainly it raised
questions for the researcher about the ethics of seeking consent in such a situation.
Some patients do not speak English and so even a small introduction can be a huge source of
confusion for them as they look at you worried and bewildered wondering what this person who
looks like a doctor wants. Even though our study did not focus on patients but the nurses’ use of
devices, and our ethics form said we would not include patients who did not speak English, we
found ourselves in this situation as we shadowed the nurses. It only became apparent that the
patient did not speak English when we tried to explain who we were and what we were doing.
In practice, informed consent is much easier to obtain and retain if the researcher builds good
rapport with both staff and patients, which we discuss below. However, we would also argue,
based on our experience, that informed consent processes need to be proportionate: respecting the
rights of all participants, and ensuring that they are appropriately informed about the aims of the
research and have the opportunity to participate in ways that they choose, without the formal
processes becoming a barrier to or distraction from the research.
5.1.2
Building rapport with staff
As noted above, staff may have different degrees of interest in and engagement with the research.
In our experience, nearly all clinicians are cooperative and loosely supportive, but there are
nevertheless challenges, and strategies for overcoming them, that we have identified. These
pertain particularly to HCI researchers without a clinical background.
It is important to recognise from the outset that, like any other qualitative study, conducting
research in healthcare is not just about managing information (i.e. data gathering and analysis): it’s
also about working with people. The effectiveness of this depends on the researcher’s personality,
their experience and the context, e.g. being assertive might work in places where being reserved
does not and vice versa.
12
Building rapport with professionals in hospital can be difficult as they are often too busy to
talk, non-clinical researchers often cannot help with tasks because they are technical, sensitive or
for infection control purposes. You want to help but sometimes it is difficult to identify how, and
you don’t want to be a burden. Over time you may find ways in which you can be of assistance.
For example, when RR was undertaking research in an ICU, the nurses began to ask her to assist
in small ways, such as getting a chair for a relative. When observing in a paediatric ward, the
nurses seemed to appreciate the fact that RR took the time to talk to the children, particularly those
whose parents were not able to visit often. On wards, DF has also assisted in small tasks after
being present for a few weeks. These included moving chairs, getting water for a patient, making
an internal call and getting a replacement for a faulty infusion pump. In the operating theatre, KC
has been asked to help move patients, obtain equipment, answer pagers, assist in gowning, and
various other tasks. Extreme care has to be taken to balance this willingness to help (and
subsequent rapport building) with the effects on the observations themselves. On more than one
occasion, surgical observers in theatres were asked to “scrub in” to cover missing team members;
this change of role made the delivery of care smoother, but also masked the systemic challenges
faced in the integration of team and technology on that day.
Contextual Inquiry [Beyer and Holtzblatt, 1998] advises researchers to take an apprenticeship
stance. We have found it beneficial to make it clear that we are there to learn from nurses rather
than observe them. The former is much more collaborate and non-threatening. Even nuances in the
words you use can affect how people receive you in context. However, there are limits to being an
apprentice. DF has experienced needing a more assertive stance to convey the importance of the
study and HCI in healthcare more generally: a nurse said she had tried everything to silence an
alarm for a critically ill patient; it took a chain of increasingly assertive interactions to make her
aware of further actions that she had not and should have tried. Of course, being assertive is not
the same as being rude. We made a point of criticizing the usability of the device rather than her
competence and made a joke about the situation in the staff common room as the TV’s volume
was so much easier to control.
Small talk is often effective in building rapport, but can occasionally back-fire. DF once asked
a nurse where they go on their break; she smiled and said that they didn’t get breaks, when he
laughed to reflect her mood she turned serious and said it wasn’t funny that they don’t get breaks.
He apologised and empathized with her seemingly changed mood. In other studies we often try to
talk about things other than work, to build rapport and encourage people to be more open. This
also helps when staff introduce you to others as you are no longer just a researcher but a person
that they relate to. The efforts to engage in small talk with strangers, whilst trying to create a good
impression, think about your work and how you fit into this strange new context can be tiring.
Spending more dedicated down time with people can be very beneficial.
Knowing the benefits of being with staff in common rooms during lunch and breaks, DF once
asked for permission to access these areas. The manager refused and said that staff breaks need to
be protected. In contrast, on a ward study he was treated more like a nurse would be and invited to
put his bag and coat in the staff common room and have breaks at the same time as the nurses did.
This allowed time for informal chats, small talk, discussions other than work, and further
opportunity to explain the study and ask research questions. This greatly facilitates research and
rapport, but you need to be invited in.
Regardless of the extent to which you develop rapport with healthcare professionals in the
setting, you will always in some senses be an outsider. While suspicion of the researcher is typical
on first entering the field, with staff concerned that you are actually observing on behalf of the
hospital management, events outside of your control can heighten suspicion. For example, when
RR was observing in an ICU, a patient who was a heroin user was admitted with a rare form of
botulism and it was thought that the patient had contracted the disease through contaminated
heroin. This story was leaked to the press and the researcher was treated with increased suspicion.
Some people will be willing to help you more than others. Such research champions go out of
their way to help with your work and are of great value. This could be by introducing you to
others, being an ambassador for your work, and making data available to you.
There will be better and worse times for people to help with the study. For example, nurses
have said that they didn’t want to participate, citing reasons of fatigue. This shows that the consent
13
process is working. There was an occasion when a whole ward had had a bad weekend and the
vibe on Monday morning was stressful, much busier than usual and quite uncomfortable. In this
case DF decided to abandon that day. Even though the times when fatigue and stress are
heightened could be the most interesting for observational data and understanding work,
sensitivity is needed.
As noted above, in some of our studies we have been particularly interested in errors and
unremarkable disturbances, but how do you explore these tactfully? A naïve view is that you think
you see someone do something wrong, so you approach them and interview them about it as soon
as possible after so it is fresh in their mind. This approach doesn’t win you any friends because
mistakes are embarrassing. Furniss et al. [2011c] report nurses making mistakes with their
calculations and their safety checks. These were often brought up after the nurses had recovered
from the situation and long after the incident had passed. However, there were other mistakes like
a nurse filling out a new prescription chart because they had made a mistake, and blood samples
nearly getting mixed up that were not reported or discussed with the nurses. These mistakes were
outside the scope of the study and we deemed that engaging with them would not have been
constructive. In another example, Catchpole [2011] describes a situation that was perceived as
risky, but not by the team in question, and it was important to learn as much as possible about why
they did not perceive risk, rather than aggressively attempt to address the problem. Later
discussions revealed considerable political sensitivities towards the unit in question.
5.1.3
Engaging with patients and visitors in hospital settings
HCI studies in hospitals most commonly focus on the work of healthcare professionals. However,
building rapport with patients is also something that the researcher needs to do, in order to obtain
the patient’s consent to observe activities that take place around them and to make the patient feel
comfortable with the researcher’s presence. The perspective of patients on the work that goes on
around them can also be a valuable source of information. It is when seeking the patient’s consent
that engagement with patients typically begins. In doing this, it is advisable to ask the nursing staff
which patients it is okay to approach. For example, if a patient is suffering from confusion, they
are not going to be able to provide informed consent. Our general experience is that patients are
typically happy to take part in the research and we have found that some patients appreciate being
able to talk to the researcher and like the fact that you are someone they can talk to who is not a
healthcare professional. When a patient declines to take part, that is a sign that the consent process
works; that those who do not want to take part feel free to say so. Occasionally, you may find that
patients treat you with suspicion. With the patient who had botulism described above, as the
patient got better the researcher began to speak to him and he said that he had seen her observing
and thought that she was spying on him. Whatever patients’ reaction to your presence, you are
interacting with patients and their relatives at what is typically a difficult time and sensitivity is
needed.
In fact, it is this engagement with patients and their stories that can be one of the most
challenging aspects of working in the hospital setting, even if your engagement with patients is
limited. For example, one student told us of a handover meeting at which the nurses were told that
a patient had been given days to live. This saddened the whole room of people, including the
researcher and the awareness of someone’s imminent death weighed on her thoughts for a while.
Certainly, when you spend long hours on hospital wards, it can push your thoughts in a particular
direction, to reflect on your own mortality and to consider how you would respond if you or
someone close to you was in such a situation. RR remembers crying in a hospital car park after
learning that a child on a paediatric ward, who she had played with and developed a fondness for,
had died from a hospital-acquired infection. While the nurses had their own practices for
responding to these events, as a researcher she was excluded from these practices. KC observed a
paramedic crew attempting to resuscitate and de-fibrillate an extremely sick patient in the back of
an ambulance while he sat in the cab in full view of the family, looking anxiously to him for
hopeful clues in a situation that did not yield a good outcome. However, in other settings, we have
experienced the concern and support of the healthcare professionals when faced with such
situations and while the emotional aspects of the work can be challenging, they can also be
humbling and enlightening. For example, in studying the work of histopathologists, RR observed a
14
“cut-up” session, where the histopathologist takes samples from a specimen so that glass slides
with slices of the tissue can be produced for the histopathologist to view under the microscope. A
cancerous tumour had been removed from a patient’s liver and it was this that had been sent from
the operating theatre to the histopathology department. While the histopathologist was concerned
that the researcher may feel strange to observe the slicing of a human liver, in fact it was
interesting to see a tumour, when cancer is an illness we so often think of it as being invisible, and
a reminder of medical progress to see this tumour that had been removed.
5.1.4
Being an outsider in an intimate space
HCI researchers are not medically trained and so are outsiders. We do not have the experience or
sense of authority to be in this special place. There are times when you want to be unobtrusive, but
somehow it often feels like you’re trespassing. The work tempo doesn’t fit, there is often nowhere
convenient to sit or stand without getting in someone’s way, it is hard to help out in such
specialised work, patient spaces can be very intimate and circumstance can change quickly, e.g. a
nurse setting up an infusion pump might also change a patient’s incontinence pads or need to
undress them to check intravenous access points.
We often feel comfortable in an environment when we have implicitly learnt the unwritten
rules, social norms and cultural values. In everyday life we move into different spaces that have
different social norms and unwritten rules about how we act. For example, asking someone
directions in the street or for the time at a bus stop is an expected form of behaviour; however,
asking someone the time in a swimming pool changing room seems more strange and could be
interpreted as intrusive. So, how do we behave in different areas of the hospital? Where are we
allowed to go? What are the unwritten rules that staff take for granted and patients accept?
Sometimes these issues can be very functional, e.g. you have to learn when it is acceptable to
interrupt a nurse to ask a question by understanding their work to some degree. At other times
rules can be less apparent, and the researcher will experience a growing sense of uneasiness or an
acute sense of being uncomfortable. Different hospital contexts, e.g. surgery, wards and outpatient areas, have different activities and atmospheres. Even within a context the atmosphere can
change quickly, e.g. by a change of patient on an open ward or when someone has received bad
news. To give a sense of what one might have to adapt to in hospitals we draw on analogies from
everyday experience, but in a hospital this change of “place” rather than “space” [Harrison and
Dourish, 1996] can happen unexpectedly: e.g. places in hospital can be like entering a patient’s
private bedroom where they are sleeping; it can be like a swimming pool changing room in terms
of the potential for nudity; it can be like a restaurant kitchen where everyone is busy, task focused,
and you feel like you’re getting in the way; and it can be like a church or a contemplative place
where people share quiet, intimate moments. On top of this there are seriously ill patients which is
emotionally demanding, with scenes that can disturb the squeamish. Any one of these everyday
contexts would prove challenging, but it seems even more challenging when they are lumped
together. In practice one must learn quickly; be cheerful, polite and empathetic; and follow the
lead of clinicians who are used to working in these challenging environments.
5.2 Gathering data
Hospital settings offer extremely complex and unpredictable environments that require the
expertise of the observer and their interpretations of those observations within a frame of reference
to produce even the most well defined measures. As discussed above, there are various dimensions
on which studies vary, including:
• The nature of the study question (e.g. whether it is understanding work processes,
evaluating technology use, or developing a theoretical perspective);
• The expertise of the researchers involved in the study; and
• The degree to which data gathering is informed by a pre-existing theoretical
framework (such as Distributed Cognition or Resilience Engineering [Furniss et al.
2011b]); this is likely to depend on both the study question and the expertise of the
researchers involved.
15
In this section, we discuss the practicalities of gathering data in terms of methods and tools for
data gathering; engagement with participants; being in the right place at the right time; and
working with the constraints of the environment.
5.2.1
Methods of data gathering
The main techniques that we have used for data gathering in hospitals are observation; ad hoc
informal interviews; and more formal semi-structured interviews. The balance between
observations and interviews depends on the questions being addressed through the research and
the degree to which the researcher wishes to engage with participants – a topic to which we return
later. In this section, we briefly summarise our experiences of observation and interviews, with a
particular emphasis on the challenges of working in a hospital; in the next section we discuss the
benefits and limitations of different means of recording observations and interview notes.
Observations are well suited to descriptive studies for understanding work and technology use
[Jorgensen, 1989]. On the one hand, they give direct access to the work and interactions of
participants (without self-reports, which can be notoriously unreliable); on the other hand, they
place the onus on the researcher to note all the important information about the situation, and to
interpret that information, both of which can be highly challenging in such a complex and fastchanging environment in which the participants are experts. For example, in our study of infusion
pump use in an oncology day-care unit [Furniss, Blandford and Mayer, 2011], a nurse could
usually programme a pump in about 6 seconds, and it was often difficult for the researcher to
place himself unobtrusively to observe the interaction clearly.
We have found ad hoc informal interviews to be the best way to seek clarification on what was
observed – ideally, by engaging in conversation as soon as a question arises. In a hospital setting,
this is often not possible or appropriate – either because the member of staff involved is too busy,
or because, as discussed above (5.1.2) drawing attention to an error from which the member of
staff has recovered can be embarrassing.
To address research questions more fully, and to validate findings, it is usually necessary to
organise more structured interviews with members of staff. In our experience, nearly all members
of staff are willing to participate in interviews, but scheduling them can be very challenging, as
many clinicians are over-worked, with little time to spare for other activities. And interviews may
be cancelled at the last minute due to pressures of work. It is not usually a good idea to ask staff to
be interviewed during a break time: when they have a break, they need a rest, not to talk more
about work!
To cope with this, Rajkomar and Blandford [2012] conducted ad-hoc and intermittent
interviews, asking nurses a few questions at the bedside whenever possible. These interviews were
different from conventional interviews in that there was not sustained attention from the
participants, and they were different from contextual inquiries in that the questions could not be
asked during the activity; rather, questions had to be noted down to be asked during opportune
moments, which could be minutes or hours later. To make the most of these small pockets of time
that nurses would allocate to the researcher, in terms of getting as many questions answered as
possible, Rajkomar and Blandford [2012] maintained a spreadsheet to keep track of all questions,
and selected questions from it to ask.
In hospitals, more than any other study setting we have ever worked in, there can be a tension
between a relaxed “being there” approach and a more formal questioning approach. The latter is
more “scientific”, and easier to describe in papers; it is easier to gather evidence and demonstrate
that the work has been conducted systematically and objectively to give confidence in the validity
of the findings. However, you might not get to the heart of the matter to the same degree as “being
there”: the more relaxed and informal interactions with nurses and patients are, the more natural
they are, and hence the more open and honest they are likely to be, but the data is more ephemeral
and more difficult to report clearly.
5.2.2
Tools for recording
All the issues that apply in any qualitative study apply in hospitals, but perhaps more extremely.
Others report on the use of video [Wilcox, 2012] and even eye tracking technology [Grundgeiger
et al. 2010] in hospitals, but within the current UK ethical framework it is very difficult to justify
16
the use of such intrusive data gathering tools in wards. As discussed above, these are intimate
places, and privacy must be respected. In addition, in most hospital settings it would be difficult to
place video cameras effectively to capture the data of interest reliably and without disruption.
However, we have found benefit in using video in specific settings, as in studying MDT meetings
(RR) and surgical work (HM).
It is often possible to take still photographs, provided that they do not feature people or personrelated data. (People can feature if they give permission, and if the uses to which the photographs
will be put are made clear [Lipson 1997]).
Audio recording may be possible during more formal interviews (when participants can give
informed consent), but care needs to be taken that security and confidentiality are maintained
during storage and processing. While these challenges are faced in some other settings, the
demands of confidentiality in healthcare make safekeeping particularly onerous. However, as with
video, audio recording can be beneficial for gathering data in specific contexts. For example, in
studying handover, where we had consent from patients and clinicians we audio recorded
handovers, which allowed us to focus on recording the non-verbal interaction in our notes.
Similarly, when studying the work of histopathologists, we audio recorded our sessions with them,
in order to gather their descriptions of what they were doing.
The main form of data recording that we have made use of is field notes. These can vary in
their degree of structure. For example, O’Connor [2010] adopted a notation scheme:
nd
the following entry “m, doc, 26, 2 2day, chatting with col, ‘back again, you must
know this machine better than I do’, scan, enter false, bin & wait” meant that a male
th
doctor, on the 26 of the month used the machine for the second time that day, he
was chatting with his colleague throughout, while making a comment to the observer,
scanned his own ID, entered a false hospital number and left the sample on the
sharps bin until the results had printed.
The researcher has to make situated decisions about the most appropriate moment to record
data, and the most important moment to observe, since it is possible to lose data through
inadequate recording or during the recording of observations (too much “head down” time).
Furthermore, the significance of observations may only emerge later, which sometimes requires
adaptation or re-interpretation of notes.
The method of capturing notes can be intriguingly influential; the use of a clipboard is likely to
instil suspicion, while note-taking in a small book is less overt, but is still likely to generate
unease. A smart-phone or other PDA may be much less threatening as it is common to see this in a
wide variety of settings. The more overt the data recording is, the more likely the observer is to
become engaged in conversations with clinicians interested to know more, distracting the observer
from their data collection and the clinician from their work but, conversely, being a great spur for
engagement, leading to valuable discussions.
5.2.3
Engagement with participants
Sensitivity to those being observed is a key skill that is difficult to define. If data recording (going
from “head up” to “head down” – i.e. from watching to writing notes) is directly related to
particular actions, communications or errors, those being observed may become highly sensitised
to the behaviour of the observers and adapt their behaviour accordingly.
In the early phase of her study, Gant [2011] took extensive notes, as she developed her focus,
but soon realised that this was perceived as being threatening by some members of staff:
I became aware that some nurses were concerned I was noting information they
considered sensitive. To begin with I took notes of many things that have not
informed the analysis directly, but my approach was to gather details, even if they
seemed irrelevant, in order to assess if other data indicated they were important.
Senior doctors may be more comfortable with being observed than junior staff or nurses, as
they are frequently observed by medical students. Others are often more concerned that they may
be observed to make a mistake.
It is difficult to assess to what extent the sense of being observed influences people’s
behaviour. O’Connor [2010] noted one occasion when…
a participant used her operator ID for a colleague who did not have a barcode for the
machine; she then commented jokingly “I’m going to get struck off for that one!”
17
He also noted three occasions when participants left the blood gases analyser after inserting the
blood sample in order to retrieve the patient’s hospital ID; we can only speculate that participants
might have circumvented this step had they not been being observed.
As noted earlier, subject to informed consent, participants take part in a hospital study by
virtue of their role within the organisation, and some engage more enthusiastically with the
research than others. This has inevitable consequences in terms of the overall representativeness of
the data: it is easier to gather data with people who are more engaged and less time-pressured, and
in situations that are less hectic, and where patient privacy is not compromised. These aspects of
hospital studies are unavoidable, and need to be recognised and accounted for as far as possible in
the approach to data collection and analysis.
5.2.4
Being in the right place at the right time
One challenge, as an individual researcher in a hospital, is knowing where to be, what to observe,
and who to work with at any given moment.
It is typically important to agree times for the study with key members of staff so that they
know when to expect you to be around, particularly in safety-critical locations such as operating
theatres and ICUs (where unauthorised access is considered a security breach).
Depending on the study focus, it might be important to observe at different times of the day
and week, including night shifts and weekends [Hammersley and Atkinson, 1995]. Observing a
night shift can be a tiring and challenging experience for the researcher. RR remembers the first
experience of observing a night shift on an ICU and one of the nurses asking, “Do you feel sick
yet? Don’t worry, you will.” DF remembers being invited to sleep in the treatment room for a few
hours when studying night work on a ward, an unofficial practice that nurses took part in to help
get through the night. He accepted because he was very tired, and also to fit with community
practice. Such night and weekend working is something that earns respect amongst the healthcare
professionals you are working with and really helps you to understand the realities of work, and
device use, in such contexts. For example, when observing the night shift on the ICU, one of the
nurses explained that she finds it really difficult to sleep during the day and so was surviving on
just a few hours sleep.
Within a particular data gathering session, it can also be difficult to decide where to go and
what to observe, particularly in a large, multi-roomed ward with many staff and patients around.
Gant [2011] describes some of the challenges she faced in studying infusion device use in a
haematology ward:
It was sometimes difficult to follow through with a whole sequence of preparation and
infusion. I might find a nurse with a set tray ready prepared and follow them to a
room straight away, missing the preparation. The nurses might divert from their
intended activity and carry out a new task when the first was not possible (patient in
the toilet, for example). This meant some of my notes are of incomplete sequences,
having been interrupted between preparation and infusion.
These challenges arise due to the unpredictability of many activities that might be the focus of
study. For example, in studying clinical handover, we found that while the nurses’ handovers took
place at a regular time and in a regular place, the medical handovers were harder to track down,
with the time and the location variable due to the need to balance the requirement to handover
with the ongoing work [Randell et al., 2011a; 2011b]. This meant that handovers sometimes took
place in corridors or cafes and gaining access to such handovers required either shadowing the
people whose handovers you wanted to observe or building up such rapport that they would
inform you when and where the handover would take place.
5.2.5
Reading and “using” the situation
As should be evident by now, both the physical environment and the nature of hospital work pose
significant challenges to conducting qualitative research studies. While the realities of a particular
study setting may be difficult to anticipate in detail ahead of time, it is usually beneficial to work
with, rather than trying to ignore or change, these constraints. It is important to learn to recognise
cues to the situation to identify appropriate behaviours.
18
One factor in Gant’s [2011] study was that every patient was in an individual room, under
barrier nursing conditions (everyone entering or leaving the room had to go through rigorous
infection control procedures). Although these procedures were very time-consuming, they were
not mentally demanding, and were invariably completed alongside a nurse. These proved to be
good times to chat informally with nursing staff, to both build rapport and understand the situation
better.
O’Connor [2010] experienced a different kind of disruption: that he occasionally had to leave
the study site because it was too busy and crowded:
The resuscitation room could go from a state of quiet isolation to distributed and
organised action in a matter of two or three minutes. This may happen when two
patients enter in close succession and an emergency case follows shortly afterwards.
At these busy times the medical team consisting of doctors, nurses and observers
are added to the core team of nurses who are responsible for ensuring the
resuscitation room is always ready for patients. These additional staff would quickly
fill the room and require space to work. […] These enforced breaks enabled the
researcher to reflect on the recently observed interactions and to consider the
aspects of interest to be investigated on resumption of observations.
If the study focuses on the use of a particular technology, or on particular activities, it is often
necessary to be patient: there can often be a lot of down time while waiting to observe. But this
can be a good time to talk informally with staff, building rapport and developing a richer
understanding of the context within which the behaviours of interest take place. It is also
important to understand this variability in the work practice.
5.3 Design and Testing a System in the Wild
Studying current healthcare practices in a hospital setting can uncover important findings and
knowledge as to the needs or challenges of technology in healthcare. However, one may also want
to take that knowledge and develop a new technology to investigate its use, uptake, or impact on
existing practices. This step can be quite an undertaking as the implications of failure can be
severe. However, without such steps being taken, progress cannot be made, or progress may take
a less desirable form without the lens of human-centred design.
As good user interface design prescribes, it is important to have access to personnel who are
interested enough in the outcomes of the system development effort that they are willing to
provide you with the necessary feedback throughout the design process. This may include them
providing you with extended time helping to define the system interface or interaction design as
well as providing feedback on resulting system iterations. It is also helpful that they can secure
further testers as the system design is nearing more final and refined stages. Testing it in the
environment under simulated conditions is another important step as the system may not work as it
had in more controlled environments. In the development of our system for touchless interaction
in vascular surgery, our first test of the system in the operating theatre under simulated conditions
(i.e. no patients) included five surgeons and radiologists testing the system and giving feedback.
Through this initial test, we learned that voice control was much more robust in the noisy theatre
than we had expected and many of the testers liked the voice control in addition to gestural
control. From this feedback we made considerable redesigns to the system.
Your champion may be a clinician, but in order to deploy an information system, you may
need to work with other technical and support personnel – for instance, around issues of health and
safety or power and electronics. Access to sensitive information also puts constraints and
requirements on the system design, as you must follow hospital protocol as to what the system will
do with identifiable information (i.e. store it or expunge all information). This also means that,
when you are at the point of a field trial, if possible you should have a back-up plan in case of
failure. The demands of deploying a system in a real healthcare environment extend beyond
simply the research methods into how the system itself is designed to fit there.
As you move closer to the point of a field trial, it is important to discuss with your champion
the type of situation you need to test the system. You may be most interested in high stress
situations or difficult procedures in order to measure issues with error rates or time to completion.
Although your system may be used for other situations, you as the researcher need to ensure your
system is used in a suitable situation that allows you to gather the appropriate data. Hopefully you
19
have been capturing data or observed practices in the environment before the point of system
deployment and have a good idea of the opportunities for data capture and the problems that may
arise. However, there may be further data capture necessities or possibilities with the new system
– for instance, screen capture.
A final aspect of developing a system is to determine how long the field trial will last and what
happens when you have completed your field test. This may not be clear ahead of time, but it
helps to have this conversation with your champion in order to, at the very least, have an idea of
what his or her expectations are. It may be that, after you gather enough data, you leave the
system with your users for their continued use. You may, on the other hand, not feel that the
system is suitable to be used without your intervention, and in those cases it is important for your
champion to agree to ending the trial when you are done. It is a tricky situation when you provide
a system that, hopefully, is a value-add for your users and then take that system away from them.
We, as researchers, typically only think towards the point when we have gathered our data in order
to write our papers. But in the sensitive and critical environment of the hospital or healthcare
setting, thinking past our own needs and considering what impact one wants to make on the lives
of those in healthcare is something that should be considered early in planning.
6.
HEALTHCARE OUTSIDE THE CLINICAL SETTING: STUDIES IN THE HOME AND ON THE
MOVE
Healthcare is increasingly found outside clinical settings. Worldwide, people are aging, and much
of the drive in healthcare reform is pushing care out of hospitals. This shift is also putting
responsibility into patients’ hands, increasingly through patient centric technologies. Telehealth
and mobile medical technologies are being explored by some of the largest mobile technology
companies worldwide. This shift is seen in the marketplace and in the realities of people’s care,
making further “in the wild” studies essential to understand the effects of these changes and to
design technologies and ways of deploying them that bring about positive changes. This, in turn,
demands that we recognise and address the issues that will arise with testing in people’s homes
and with mobile health technologies. Like other “in the wild” studies, we seek to understand how
new technology is made everyday, and the new ways that people appropriate them.
6.1 Recruitment of Participants
For any type of study, recruitment of participants is an important consideration. Outside the
hospital setting, participants are typically people who have a long-term condition. In our studies,
we have worked with people with renal disease and with diabetes (many of these participants have
also suffered from other conditions in parallel).
Our studies of renal disease have involved participants using home haemodialysis machines.
There are relatively few of these, compared to those treated in satellite dialysis units or using
peritoneal dialysis technology, so it has been important to recruit as many as possible to the study.
The home haemodialysis nurse of the relevant hospital played a critical role in recruiting
participants. She informed the hospital’s home patients of the study, while visiting them, and then
arranged for us to contact interested patients.
Recruitment conducted independently by the researcher can be difficult as a group of patients
may only have their condition in common. Our recent work on looking at diabetes patients’ use of
health information [O’Kane and Mentis, 2012] presented us with the challenge of recruiting a
representative sample of patients. If we had gone to a diabetes clinic in a hospital, we could have
recruited people of different ages, genders, and socio-economic status, but we would be limiting
the study to people that had to attend these specialist clinics. Generally, these are people with the
most complications, and this would exclude people whose diabetes was under control. If we had
just used a popular diabetes Internet forum, we would have restricted recruitment to those in the
“Diabetes Online Community”, who are relatively technology savvy. In the end, we used a mixed
method approach to recruit patients. We posted advertisements on the internet board Gumtree, in
a weekly student newsletter at a local university, and at a number of grocery stores, and also
recruited people at support group meetings and used personal contacts. Even when posting
physical advertisements, we made sure we posted at a variety of different types of stores and many
locales, including a town centre, the suburbs, and places further removed from the town, to ensure
20
broad demographic coverage. This mixed method approach to recruitment ensured a variety of
patients by their type of diabetes, age, gender, socio-economic class, technical ability, and so on,
but of course it was still limiting to those who would be interested in talking about their condition.
In addition, their willingness to participate can influence the results, particularly for deep, and at
times intense, “in the wild” studies.
Observations and testing of patient-centric devices can be socially invasive as these
interactions take place in people’s private homes and wherever they use mobile medical devices.
Setting up video apparatus in homes can allow the researcher to observe the interactions with these
medical devices, but the very presence of video cameras in the home may cause discomfort for
participants. Even if designed only to take video of the interactions with the medical technologies,
there is likely to be a “big brother” feel to the apparatus. Homes are private places, so it can be
hard to find willing participants, and the presence of video capturing devices affects the situation
that is to be captured. One of the strengths of “in the wild” studies is that they aim to take into
account context and “messy” use in real life, but penetrating people’s personal lives, especially
where healthcare and illness are concerned, can make recruitment challenging.
Despite these challenges to recruitment, HCI research in the medical domain can be met with
great enthusiasm by participants, particularly for research that seems to focus on an aspect that has
caused them frustration in the past. This was the case with the diabetes information-sharing
project, as both patients and diabetes specialists had issues with current practices and current
information systems. Our very presence raised their hopes in thinking that we would not only find
solutions to the issues that they had, but we would also implement these solutions so that they
could benefit from them. This was outside the scope of a short-term research project and
subsequently, we felt that we had given them false hope. This is a theme (also referred to in
section 5.3) to which we return in the discussion.
6.2 Interview studies
We have relied extensively on interviews (often with observations) to understand people’s
perceptions and experiences of using technology. For haemodialysis, this has involved visiting
participants and carers in their homes. During a visit, the patient or carer was observed during part
of their treatment, and then they were interviewed on their experiences of using the technology in
the home. We also examined the physical setting in which the patient dialyses, and artefacts that
patients use such as dialysis charts and diaries. For studies of glucometers, it is possible to conduct
interviews in places of participants’ choice. Inevitably, discussions about technology use are
tightly bound with discussions about their conditions. Since this is a potentially intrusive and
uncomfortable situation for the interviewees, ensuring a comfortable setting for sessions is
essential. TO offered participants a choice of locations where they would like the meeting to take
place, or the option of suggesting an alternative location. As our study was restricted to a
university campus, these locations were typically coffee shops, canteens or personal offices of the
interviewees. None of our participants elected to have the sessions in a location that they were
unfamiliar with (such as a user study laboratory). Meeting in an informal and relaxed setting
allowed sessions to develop into a friendly chat about experiences.
An additional key factor in allowing this transition was using existing general knowledge
about a person’s condition and device design. TO prepared by investigating a range of devices
used by participants (glucometers in this instance); this resulted in more in-depth discussions as he
could relay his own knowledge, which could be of benefit to participants as well as appearing to
make them more comfortable in the situation. Questions such as “did you ever use feature X on
the device?” often prompted participants to described their own experiences with the device, such
as why they did or did not employ the feature or how they preferred the setting on a different
device.
6.3 Mobile Device Studies and Diary Studies
For mobile medical devices, it would be difficult to set up video capturing as the use of these
devices is not confined to a static context. Through some of our preliminary research on the use of
mobile medical devices such as glucometers used by diabetes patients, often a routine is formed
between home and work/school, but this does not encompass the variety of uses of these medical
21
devices during non-routine times. These people visit friends, go on holiday, attend conferences,
go to music festivals, and so on, influencing their use of the devices. During routine times, the
problems that exist with the invasiveness of video capturing and observation in private settings
applies, and during non-routine times where observation and video capture is next to impossible,
these devices are very hard to study “in the wild”.
Taking influence from the Experience Sampling Method [Csikszentmihalyi and Larson 1992;
Diener et al. 1984], adapted diary studies can be used in HCI to explore people’s experience with
these devices at the moment they use them [Consolvo and Walker 2003]. Diary studies can be
used to capture in-situ information [Rieman 1993], but issues of inconvenience arise in mobile or
active conditions [Palen and Salzman 2002]. There are ways to reduce this inconvenience by
adapting the diary study to only involve taking a snippet of text, audio, or video at the time of the
occurrence on a mobile device and filling out a longer entry later on a website [Brandt et al. 2007],
but this does not completely alleviate the concerns for patient participants. A recent autoethnography study using a mobile wrist blood pressure monitor and keeping a diary of its use with
a mobile phone application that could record text, audio and video brought some of these issues to
light.
In the autoethnography study, AOK was using a blood pressure monitor at a non-routine time
where she crossed an ocean to visit home, go to a wedding, take a flight to another time zone, and
attend a conference along with a routine time at home. The non-routine time caused specific
issues that the researcher had to overcome, but it also made clear the importance of testing the
device in both settings. Simply getting a patient to complete a diary study for 2 weeks will not
give an idea about the breadth of use of the device. Some of the most challenging situations can
occur during these non-routine times, but these are the hardest to capture as they are less common
and can be more invasive into a participant’s life then getting them to integrate a diary study into a
routine.
6.4 Participants as co-researchers
In some contexts, such as the diabetes research described above, the participant may take a role as
a co-researcher, bringing their expertise in their condition and of technology use to complement
the HCI expertise of the researcher. Where the technology is supporting healthcare but is not a
medical device6 (i.e. poses minimal risk of disrupting care), it has been possible to involve patients
as consultants to the research team, engaging them in participatory design to shape the research
and the resulting product.
In other contexts, this is impractical. In our study of home haemodialysis technology use by
patients and carers, we attempted to work with them as co-researchers, inviting them to capture
minor incidents with loaned handheld video equipment or pen and paper diaries. However, this did
not work in practice, as patients and carers did not have the time, energy or enthusiasm to invest
even more in their dialysis activity. The dialysis treatment setup, the treatment itself, the overall
management of the treatment, and related hospital appointments, consume so much of the time of
patients and carers, that some struggle to make time to do anything else. There are also difficulties
in trying to get patients and carers to critique home haemodialysis technology. The technology is
life-sustaining, and having it at home improves their quality of life (compared to having to travel
for dialysis several times a week) so there is naturally a very high acceptance of the technology,
regardless of any design flaws it may have. Also, for patients and carers, there is not necessarily a
distinction between a design flaw and a lack of competency on their part. Patients naturally want
to be perceived as being capable of fully handling the machine, either as a matter of pride or as a
matter of ensuring that they are perceived as possessing the required competencies for conducting
their treatment independently. All these factors discourage engagement as active co-researchers
beyond being interview participants.
6
e.g. http://www.soi.city.ac.uk/great/?page_id=824
22
7.
DISCUSSION: THE IDEAL AND THE REALITY
One of the important themes that has emerged in our studies is the gulf that can emerge between
what is intended and what is done. This arises in the initial study design and gaining access
(typically involving health service ethics procedures), through the writing of “methods” sections
of papers which typically outline what is intended, to the actuality of taking the context on its own
terms, which involves developing and applying a repertoire of approaches to respect the
constraints (physical, temporal and affective) of the environment in which the study is taking place
while also exploiting opportunities that the environment affords. In this section, we revisit and
expand on some of the themes that have emerged through our studies: the development of
expertise; managing different perspectives; dealing with emotional challenges of working in
healthcare; and the benefits of research “in the wild” in healthcare.
7.1 Expertise
As discussed in section 4.3.3, when studies start, the researcher may have limited understanding of
the clinical context, the technologies being studied or the work practices that are the focus of
study. Inevitably, expertise develops over time; in some cases (notably new MSc and PhD
students), the learning curve is initially very steep. Preparation prior to data collection is advisable.
For example, prior to interviewing people with long term conditions, it is helpful to understand as
much as possible about the condition, the technologies currently being used to manage it, and the
experiences of living with it. Useful resources include clinical descriptions of conditions, device
manuals, and internet support forums for patient experience. For example, before studying clinical
handover, RR read existing literature on clinical handover, including research studies and
guidelines. This highlighted the need to look at the work of preparing for handover as well as
informal discussions between staff before the official handover.
While such background research is beneficial and can give the researcher increased confidence
as they enter the new setting, it is important that it does not bias the researcher. Being a novice in
the healthcare domain is advantageous in that the researcher views the actors and the environment
with new eyes [Blandford and Rugg 2002; Furniss and Blandford 2006]. Initial unfamiliarity can
help with noticing details that might be taken for granted by someone with more experience. For
example, in observing use of infusion pumps, having no preconceptions of what they should be
seeing allowed Gant [2011] to avoid jumping to conclusions. In fact, future investigators might
find it useful to take advantage of this initial unfamiliarity by having two phases of data gathering:
one without knowledge of the correct device operation, followed by one with.
Regardless of any preparation undertaken, there will still be much to learn once data collection
begins. Obviously the researcher wants to learn about the topic of the research, but there is other
more social knowledge that needs to be acquired. For example, how to address the people that you
meet can be a delicate area. Medicine is highly hierarchical and, while some doctors will be happy
for you to refer to them by their first name on the basis that you are outside that hierarchy, others
will expect you to refer to them by their title and take offence if you don’t. Similarly, despite
knowledge gained from any preparation, a researcher entering into a discussion with someone
experiencing health problems will never have the same depth of knowledge and experience. For
example, referring to “diabetics” yielded a problem in one discussion as it was pointed out that the
researcher should not label the person by their condition. The use of the term clearly caused
offence, as the participant explained to the researcher that diabetes did not define them as a person,
but was something that they happened to have. Where such situations occur, while difficult at the
time, these experiences ultimately enhance the researcher’s knowledge and understanding which
will aid in subsequent data collection.
It is on entering the setting that the researcher also begins to learn about the feasibility of the
study that they have designed. When data collection begins, it is necessary to take time to reflect
on the suitability of the proposed methods as you come to understand the realities of the setting. In
paediatric cardiac surgery, Catchpole, et al. [2006] observed 38 cases (approximately 150 hours)
before beginning data collection. This was necessary due to the complex and technical demands of
the setting. However, even in less complex settings, it may be beneficial to treat your first few
days of data collection as a pilot study, an opportunity to test your methods of data collection
before a more formal phase of data collection begins.
23
Where the study involves both interviews and observations, it is often beneficial to observe at
length before doing any formal interviews, to build up a reasonable understanding of the study
setting. Brief informal interviews are important early on, to develop an understanding, but more
formal interviews are typically most useful towards the end of the study, when the researcher has a
clear idea of the important and relevant questions to ask. For example, O’Connor [2010]
conducted 6 hours of observations before conducting his first interview. Some might say that even
six hours is too short, but within the constraints of the 3 month MSc project it was essential to
gather data as quickly as possible.
7.2 Managing Different Perspectives and Priorities
Patients and clinicians are not HCI professionals and so don’t have the same motivations and
interests that we do. They do not notice the details that we might. Furthermore, they might have a
different perspective, e.g. that the device works properly; it’s just that the people don’t know how
to use it.
Creating a coherent picture can be difficult, as the observer is likely to hear different views and
conflicting facts about the same system, especially when people tell you what they think you want
to hear or what is most important to them. For example, in one study, a nurse repeated over and
over again how time consuming the new infusion pump was. Participants in a setting will have
differing perspectives on the same system. This is in part due to their differing experiences with
the technology, but also because their role affects their criteria for success [Stevenson et al. 2010;
Storni 2010]. For example, while attempting to enhance knowledge of a condition, TO sought
advice from people who were involved with research and treatment of the condition. Different
concerns were raised from different perspectives. Meetings with nurses in a Diabetes Clinic led to
discussions about a patient’s difficulties in getting correct prescriptions, while meetings with a
researcher based in the university primarily focused on the biological process involved with
controlling the condition.
In TO’s studies, the greatest variety in needs or concerns came from people with diabetes.
Each had a different attitude and approach to the management of their condition, with some taking
an extremely rigid approach and others a much more relaxed attitude. The latter was most evident
with one person who described having so many health concerns that no time could be devoted to
the management of diabetes as it was not causing problems in the short-term, unlike other
conditions.
Whilst all of the issues discussed during interview sessions were significantly important to the
interviewees, and patients of the conditions generally, they were often beyond the scope of the
research being undertaken. Some participants described how they had to face severe complications
of their condition, which were beyond the range of the researcher’s understanding. One participant
was faced with a decision about whether to undertake a full pancreas transplant or to proceed with
Islet cell transplantation. The information provided during these deviations served as excellent
background knowledge for future discussions but knowing how to proceed with interviews from
such unexpected discussion paths without offending a participant, as well as not missing
potentially useful research material, was at first trial and error. Interviewees were invited to
express the concerns they had, but ultimately collecting too wide a variety of information would
have weakened the eventual findings. Therefore, once the interviewer felt the discussions had
drifted beyond scope for too long, attempts were made to refocus the interviews back on topic by
asking more direct questions that informed the research.
More broadly, participant and researcher motivations for engaging with the research are
different, and it is important to manage expectations, but also to give participants direct value for
their participation (e.g. personally useful feedback) wherever possible.
7.3 Anticipating and managing emotional pressures
In this paper we have discussed some of the emotional challenges of doing research in healthcare
settings. However, that is not to imply that this is an aspect of the work that we should simply
accept and carry on regardless. As HCI research moves “into the wild” in healthcare, it is
imperative that we develop strategies to support researchers as they undertake such work. For the
person supervising the research, whether as PhD supervisor or Principal Investigator (PI),
24
important first steps are bringing it into the open as a topic for discussion and establishing
strategies to deal with emergency situations. For example, if the researcher is working late and is
faced with a distressing situation, is it okay to phone and discuss it? Where practical, it is good to
have two researchers involved in data collection, so that they can discuss things together. Getting
the researcher to visit the university counselling service before starting data collection can be
beneficial, to talk about the possible situations that will be faced and how to deal with them. For
the person undertaking the research, it is important to reflect on the emotional toll of the work, to
discuss this with your PI or supervisor, and to seek support as you need it, whether that means
visiting the university counselling service or drawing on your own personal support networks.
7.4 The benefits of “in the wild” research in healthcare
We have discussed many challenges to conducting HCI research in healthcare, and yet all ten
authors of this paper have chosen to work in this area. There are good reasons for this.
Firstly, the participants are a real pleasure to work with. Whether they are clinicians who care
deeply about their work or patients who generously share of their experiences and insights, people
are almost invariably eager to engage, within the limits of the time or energy they have available.
It is a real privilege to work with people who welcome you into their worlds which are often very
different from our own. Over time, you build an understanding of these rich and complex
environments and work with people who bring an interesting complementary expertise, and have a
real need for usable, useful technology that gives a positive user experience.
Secondly, although change can be difficult to effect, working in healthcare gives real
opportunity to improve lives – most obviously those of patients, but also those of everyone who
has dedicated their lives to caring for them.
Further, on a more selfish note, healthcare provides unrivalled opportunities to grow as an HCI
researcher: to rise to challenges that few other research settings offer (working with a wide variety
of people from many cultures, dealing with sensitive topics, adapting to rapid changes in research
context, etc.), and hence develop a repertoire of transferable skills that serve for future research.
While it is possible to conduct useful studies of healthcare technologies in the laboratory, or in
simulated ward settings, and it is much easier to get repeatable results through such studies, it is
impossible to fully comprehend how a technology is used or experienced without accompanying it
into the wild. Only there is it possible to really understand how it is used, appropriated, and
shaped, and in turn how it shapes the work and the lives of the clinicians and patients who interact
with it.
This work is important, of deep societal value and extremely interesting and challenging
academically. It is important because patient safety and wellbeing directly depend on it, especially
with the rising role of technology in hospital and at home. It is needed because technology is being
developed and adopted whether HCI experts are involved or not. It is of deep societal value
because we and our friends and family will be reliant on healthcare when we need it most: this
deep human value is combined with controlling the efficiencies and costs of modern healthcare. It
is extremely interesting and challenging academically because it is complex with many different
people, processes and technologies that muddle through under different pressures under
uncertainty.
8.
CONCLUSIONS
Studies in healthcare bring issues into sharp relief. Participants both have a great vested interest in
the success of research into healthcare technology, as it should lead to improved care in the future,
and also limited interest in that technology per se. As researchers, we have a duty of care towards
our research participants. We depend on their trust and engagement, and must not abuse it, and yet
we have limited power to effect substantive change in healthcare technology design or use. We
have to be careful what we wish for, and also careful what we promise, explicitly or otherwise.
In planning a study, it is essential to have a very clear idea of how you want to conduct the
study, so as to minimise the chances of being unable to gather the necessary data and to maximise
the value of the research. Ideally, there will be both scientific value and practical value to the
participating organisation(s). Engaging with clinicians and patients at the earliest opportunity is
likely to strengthen the study design.
25
Direct engagement generates qualitative and quantitative data that provides rich
representations of people’s experience with technology and of work “as performed” rather than
“as-imagined”, elucidates risks that could otherwise remain unreported, and provides a window on
the system of work that guides improvement prospectively, before serious errors occur.
In this paper, we have aimed to highlight some of the challenges and strategies for conducting
studies “in the wild” in healthcare. We have emphasised the importance of building rapport with
participants, and of valuing their expertise that complements our own. We have also emphasised
the value of good preparation together with a flexible attitude, identifying and working with
opportunities (such as times of enforced reflection) that the nature of the work affords, rather than
fighting against situational constraints. In some of our studies, for example of nurses’ use of
infusion pumps in the ICU and of patients’ use of home haemodialysis technology, it was
necessary to adapt our data gathering methods as the study progressed, based on what was found
to work in practice. Doing so allowed us to glean useful findings while minimizing disruption to
the setting being studied, and in effect, we studied activity in the setting on its own terms, as in the
study of O'Brien and Rodden [1997]. This eliminates power relationships between researcher and
situation. Ultimately, as a researcher, you create a dialogue with the situation, working with it,
responding to it and picking up on opportunities, not expecting to impose your will or your plan
on it.
Healthcare is a domain that is extreme in many respects – not so much in the physical setting,
but in the variability of settings, in the fact that it is embedded in society and in all of our lives,
and that use of technology is essential but not generally the focus of attention.
In this paper, we have used our experiences of studying the use of interactive healthcare
technologies in the wild as a vehicle for drawing out differences between planning and practice,
and highlighting strategies for exploiting opportunities in unexpected places. Our aim has been to
equip new researchers in wild places (particularly but not exclusively healthcare) with strategies to
conduct fruitful, effective, rigorous, valid studies, and also to inform the debate on engaging with
participants in ways that maintain the highest possible ethical standards and build meaningful
symbiotic relationships.
ELECTRONIC APPENDICES
Appendices will be provided electronically via the ACM Digital Library. Examples are included
below for review.
ACKNOWLEDGMENTS
This work would not have been possible without the support of many colleagues, MSc students,
participating organisations, and (particularly) the participants who have worked with us in studies.
It is a great pleasure to work with and learn from so many people who have expertise that
complements our own.
REFERENCES
BARDRAM, J. E. (1998) Designing for the Dynamics of Cooperative Work Activities. CSCW 98, Seattle,
Washington, pp.89-98
BEYER, H. AND HOLTZBLATT, K. (1998) Contextual Design. San Francisco : Morgan Kaufmann.
BLANDFORD, A, ADAMS, A, ATTFIELD, S , BUCHANAN, G, GOW, J, MAKRI, S, RIMMER, J AND WARWICK, C
(2008) The PRET A Rapporter Framework: Evaluating Digital Libraries from the perspective of
information work. Information Processing and Management , 44 (1) 4 - 21.
BLANDFORD, A. AND RUGG, G. (2002) A case study on integrating contextual information with usability
evaluation. International Journal of Human-Computer Studies. 57.1, 75-99.
BLANDFORD, A. AND WONG, B.L.W. (2004) Situation Awareness in Emergency Medical Dispatch. In Int.J.
Human-Computer Studies 61, pp. 421-452. Elsevier
BLANDFORD, A., ADAMS, A., AND FURNISS, D. (2009). Understanding the situated use of healthcare
technologies. CHI Workshop (pp. 1-4). ACM Press.
BOSSEN, C. AND L. JENSEN (2008). Implications of Shared Interactive Displays for Work at a Surgery
Ward: Coordination, Articulation Work and Context-awareness. 21st IEEE International Symposium
on Computer-Based Medical Systems, Jyväskylä, Finland.
BRANDT, J., WEISS, N. AND KLEMMER, S. R. 2007. txt 4 l8r: lowering the burden for diary studies under
mobile conditions. CHI’07 extended abstracts on Human factors in computing systems. San Jose, CA,
USA: ACM.
26
CARAYON, P., WETTERNECK, T. B., HUNDT, A. S., OZKAYNAK, M., RAM, P., DESILVEY, J., HICKS, B., ET AL.
(2005). Observing Nurse Interaction with Infusion Pump Technologies. Advances, (4), 349-364.
CARROLL, J., HOWARD, S., VETERE, F., PECK, J., AND MURPHY, J. (2002). Just what do the youth of today
want? Technology appropriation by young people. In Proc. 35th Annual Hawaii International
Conference, 1777-1785.
CATCHPOLE, K (2011). Task, Team and Technology Integration in the Paediatric Cardiac Operating Room.
Progress in Pediatric Cardiology 32 (2), 85-88
CATCHPOLE, K (2009). Observing Failures in Successful Orthopaedic Surgery. In L. Mitchell and R Flin
(eds), Safer Surgery – Analysing Behaviour in the Operating Theatre. Aldershot: Ashgate. ISBN 978-07546-7536-5
CATCHPOLE, K, DALE, T, HIRST, G, SMITH, P, GIDDINGS, A.(2010). A multi-centre trial of aviation-style
training for surgical teams. Journal of Patient Safety. 6(3), pp. 180-186.
CATCHPOLE, K, GIDDINGS, A, DE LEVAL, M, PEEK, G, GODDEN, P, UTLEY, M, GALLIVAN, S, HIRST, G, DALE, T
(2006). Identification of systems failures in successful paediatric cardiac surgery. Ergonomics 49(5-6),
pp.567-588.
CATCHPOLE, K, GIDDINGS, A, WILKINSON, M, HIRST, G, DALE, T, DE LEVAL, M. (2007) Improving patient
safety by identifying latent failures in successful operations. Surgery 142(1), pp.102-110.
CATCHPOLE, K, GODDEN, PJ, GIDDINGS, AEB, HIRST, G, DALE, T, UTLEY, M, GALLIVAN, S AND DE LEVAL, MR
(2005). Identifying and Reducing Errors in the Operating Theatre. Patient Safety Research
Programme
Final
Report
PS012.
Available
at
http://www.birmingham.ac.uk/
Documents/college-mds/haps/projects/cfhep/psrp/finalreports/PS012FinalReportDeLeval.pdf Accessed
18 June 2012.
CHARMAZ, C. (2006) Constructing Grounded Theory: A practical guide through qualitative analysis. Wiley.
CONSOLVO, S. AND WALKER, M. 2003. Using the experience sampling method to evaluate ubicomp
applications. Pervasive Computing, IEEE, 2, 24-31.
COOK, R. I., AND WOODS, D. D. (1996). Adapting to New Technology in the Operating Room. Human
Factors: The Journal of the Human Factors and Ergonomics Society, 38(4), 21. Human Factors and
Ergonomics Society.
CSIKSZENTMIHALYI, M. AND LARSON, R. 1992. Validity and reliability of the experience sampling method.
The experience of psychopathology: Investigating mental disorders in their natural settings, 43-57.
DAHL, Y., O. A. ALSOS AND D. SVANÆS (2010). Fidelity Considerations for Simulation-Based Usability
Assessments of Mobile ICT for Hospitals. International Journal of Human-Computer Interaction 26(5):
445-476.
DIENER, E., LARSEN, R. J. AND EMMONS, R. A. 1984. Person x Situation interactions: Choice of situations
and congruence response models. Journal of Personality and Social Psychology, 47, 580.
FAVELA, J., M. TENTORI AND V. M. GONZALEZ (2010). Ecological validity and pervasiveness in the
evaluation of ubiquitous computing technologies for healthcare. International Journal of HumanComputer Interaction 26(5): 414-444.
FISHER, C. AND SANDERSON, P. (1996). Exploratory Sequential data Analysis: Exploring continuous
observational data. Interaction, 3 (2): 24-34. March 1996.
FURNISS, D. (2004). Codifying distributed cognition: a case study of emergency medical dispatch.
University College London MSc thesis, available from http://eprints.ucl.ac.uk/5185
FURNISS, D. AND BLANDFORD, A. (2006), Understanding Emergency Medical Dispatch in terms of
Distributed Cognition: a case study. Ergonomics Journal. 49. 12/13. 1174-1203.
FURNISS, D. AND BLANDFORD, A. (2010). DiCoT Modeling: From Analysis to Design. In Proc. CHI 2010
FURNISS, D., BACK, J. AND BLANDFORD, A. (2011a). Unwritten Rules for Safety and Performance in an
Oncology Day Care Unit: Testing the Resilience Markers Framework. Proc. Fourth Resilience
Engineering Symposium.
FURNISS, D., BACK, J., BLANDFORD, A., HILDEBRANDT, M. AND BROBERG, H. (2011b) A Resilience Markers
Framework for Small Teams. Reliability Engineering and System Safety. 96.1. 2-10.
FURNISS, D., BLANDFORD, A. AND MAYER, A. (2011c). Unremarkable errors: Low-level disturbances in
infusion pump use. Proc. British HCI.
FURNISS, D., BLANDFORD, A., MAYER, A., RAJKOMAR, A. AND VINCENT, C. (2011d). The visible and the
invisible: Distributed Cognition for medical devices. Proc. EICS4Med.
GANT, F. (2011) Behind closed doors – a distributed cognition study of infusion pump use in round-theclock
haematology
treatment.
MSc
thesis,
available
from
http://www.ucl.ac.uk/uclic/taught_courses/distinction
GRUNDGEIGER, T., SANDERSON, P., MACDOUGALL, H. AND VENKATESH, B. (2010) Interruption management
in the Intensive Care Unit: Predicting resumption times and assessing distributed support. Journal of
Experimental Psychology: Applied, 16 4: 317-334
HAMMERSLEY, M. AND ATKINSON, P. (1995) Ethnography: Principles in Practice. London: Routledge
HARRISON, S. AND DOURISH, P. (1996). Re-place-ing space: the roles of space and place in collaborative
systems. In Proc. CSCW 1996.
HARTSWOOD, M. et al. (2002) The Benefits of a Long Engagement: From Contextual Design to The Corealisation of Work Affording Artefacts. NordiCHI, Århus, Denmark, pp.283-286
27
HOFFMAN, R.R., CRANDALL, B, AND SHADBOLT, N. (1998) Use of the Critical Decision Method to elicit
expert knowledge: A case study in the methodology of Cognitive Task Analysis. Human Factors, 40(2),
254-276.
HOLLAN, J.D., HUTCHINS, E.L. AND KIRSH, D. (2000) Distributed cognition: toward a new foundation for
human-computer interaction research. ACM Transactions on CHI, 7.2, 174-196.
HUTCHINS, E. (1995) Cognition In The Wild. MIT Press, Cambridge, MA.
JIROTKA, M. AND WALLEN, L. (2000). Analysing the workplace and user requirements: challenges for the
development of methods for requirements engineering. In Luff, P., Hindmarsh, J. and Heath, C. (eds.)
Workplace Studies: Recovering work practice and information system design. 242-251. Cambridge:
Cambridge University Press.
JORGENSON, D.L. (1989). Participant observation: a methodology for human studies. London, UK: Sage
Publications.
KAUFMAN, D. R., PATEL, V. L., HILLIMAN, C., MORIN, P. C., PEVZNER, J., WEINSTOCK, R. S., GOLAND, R., et
al. (2003). Usability in the real world: assessing medical information technologies in patients’ homes.
Journal of Biomedical Informatics, 36(1-2), 45–60. Elsevier.
LEHOUX, P. (2004). Patients’ perspectives on high-tech home care: a qualitative inquiry into the userfriendliness of four technologies. BMC health services research, 4(1), 28.
LIPSON, J.G. (1997) The politics of publishing: protecting participants’ confidentiality. In J. Morse (Ed.)
Completing a qualitative project: details and dialogue. Sage Publications.
MENTIS, H.M., O’HARA, K., SELLEN, A., AND TRIVEDI, R. (2012). Interaction proxemics and image use in
neurosurgery. Proceedings of the Conference on Human Factors in Computing, Austin, Texas, 927-936.
MENTIS, H.M., REDDY, M., AND ROSSON, M.B. (2010). Invisible Emotion: Information and interaction in an
emergency room. In Proc CSCW 2010.
NORMAN, D.A. (2010). Living with Complexity. Cambridge: MIT Press
O’BRIEN, J., AND RODDEN, T. (1997). Interactive systems in domestic environments. Proceedings of the
conference on Designing interactive systems processes, practices, methods, and techniques - DIS ’97,
247-259. New York, New York, USA: ACM Press.
O’CONNOR, L. (2010) Workarounds in accident and emergency and intensive therapy departments:
resilience,
creation
and
consequences.
MSc
thesis,
available
from
http://www.ucl.ac.uk/uclic/taught_courses/distinction
O’KANE, A. A. AND MENTIS, H. 2012. Sharing medical data vs. health knowledge in chronic illness care.
Proceedings of the 2012 ACM annual conference extended abstracts on Human Factors in Computing
Systems Extended Abstracts. Austin, Texas, USA: ACM.
OBRADOVICH, J. H., AND WOODS, D. D. (1996). Users as Designers: How People Cope with Poor HCI Design
in Computer-Based Medical Devices. Human Factors: The Journal of the Human Factors and
Ergonomics Society, 38(4), 574-592.
PALEN, L. AND SALZMAN, M. 2002. Voice-mail diary studies for naturalistic data capture under mobile
conditions. Proceedings of the 2002 ACM conference on Computer supported cooperative work. New
Orleans, Louisiana, USA: ACM.
RAJKOMAR, A. (2010). Extending Distributed Cognition Analysis for Complex Work Settings: A Case Sutdy
of Infusion Pumps in the Intensive Care Unit. University College London MSc thesis, available from
http://www.ucl.ac.uk/uclic/taught_courses/distinction
RAJKOMAR, A., AND BLANDFORD, A. (2012). Understanding infusion administration in the ICU through
Distributed Cognition. Journal of biomedical informatics, 45(3), 580-90.
RANDELL R, RUDDLE R, THOMAS R, TREANOR D. (2012). Diagnosis at the microscope: a workplace study of
histopathology. Cognition, Technology and Work. DOI: 10.1007/s10111-011-0182-7.
RANDELL R, WILSON S, WOODWARD P, GALLIERS J. (2011b). The ConStratO model of handover: A tool to
support technology design and evaluation. Behaviour and Information Technology 30(4), pp.489-498
RANDELL R, WILSON S, WOODWARD P. (2011a). The importance of the verbal shift handover report: A
multi-site case study. International Journal of Medical Informatics 80(11) pp.803-812
RANDELL R, WILSON S, WOODWARD P. (2011c) Variations and commonalities in processes of collaboration:
the need for multi-site workplace studies. Journal of Computer Supported Cooperative Work 20(1-2),
pp.37-59
RANDELL R. (2003). User Customisation of Medical Devices: The Reality and the Possibilities. Cognition,
Technology and Work 5(3), pp.163-170
RIEMAN, J. 1993. The diary study: a workplace-oriented research tool to guide laboratory efforts.
Proceedings of the INTERACT '93 and CHI '93 conference on Human factors in computing systems.
Amsterdam, The Netherlands: ACM.
RODE, J. (2011). Reflexivity in Digital Anthropology. In Proc. CHI 2011.
ROGERS, Y (2012) HCI Theory: Classical, Modern, and Contemporary. Synthesis Lectures on Humancentered Informatics, Morgan and Claypool, San Rafael, CA, USA
RUSS S, HULL L, ROUT S, VINCENT C, DARZI A, SEVDALIS N. (2012). Observational teamwork assessment for
surgery: feasibility of clinical and nonclinical assessor calibration with short-term training. Annals of
Surgery 255(4):804-9.
SEAGULL, F. J., AND SANDERSON, P. M. (2001). Anesthesia Alarms in Context: An Observational Study.
Human Factors: The Journal of the Human Factors and Ergonomics Society, 43(1), 66-78.
28
SIEK, K., AND CONNELLY, K. (2006) Lessons Learned Conducting User Studies in a Dialysis Ward. In
Reality Testing Workshop - at the 24th international conference on Human factors in computing systems
- CHI 2006, 4 pages.
STEVENSON D, HUTCHINS M, SMITH J. (2010). Human-Centred Evaluation for Broadband Tertiary
Outpatient Telehealth: A Case Study. International Journal of Human-Computer Interaction
26(5):506-536.
STORNI C. (2010) Multiple Forms of Appropriation in Self-Monitoring Technology: Reflections on the Role
of Evaluation in Future Self-Care. International Journal of Human-Computer Interaction 26(5):537561.
WERTH, J. AND FURNISS, D. (2012). Medical Equipment Library Design: Revealing Issues and Best Practice
Using DiCoT. Proc. International Health Informatics Symposium (IHI 2012), Miami, Florida, Jan 2830.
WILCOX, S. (2012) Ethnographic Field Research for Medical-Device Design. Biological Instrumentation and
Technology. March/April 2012. 117-121.
WILSON S, WOODWARD P, AND RANDELL R. (2010) PaperChain: A Collaborative Healthcare System
Grounded in Field Study Work. Proceedings of the First International Workshop on Interactive
Systems in Healthcare, CHI 2010, Atlanta, pp.177-80
WONG, W. AND BLANDFORD, A. (2003). Field Research in HCI: A Case Study. In Proc. CHINZ03. NZ
Chapter of SIGCHI. 69-74.
29
Experiencing interactive healthcare technologies:
embracing “the wild” on its own terms
ANN BLANDFORD, UCL
ERIK BERNDT, UCL
KEN CATCHPOLE, Cedars-Sinai Medical Centre
DOMINIC FURNISS, UCL
ASTRID MAYER, Royal Free Hospital
HELENA MENTIS, Microsoft Research
AISLING ANN O’KANE, UCL
TOM OWEN, Swansea University
ATISH RAJKOMAR, UCL
REBECCA RANDELL, University of Leeds
A. INFORMAL GUIDANCE ON UK ETHICS PROCEDURES
There are processes in place to check that research studies meet a high standard of ethical due
diligence, primarily to make sure patients, staff and researchers are protected. There are also
organisational considerations to be managed such as disturbance to practice, the use of resources
and potential benefits of the study. The processes for formal ethical approval vary worldwide –
some places are very stringent and some places have no formal procedures over and above any
normal study. In this appendix we cover the main elements of getting access to do research in the
UK National Health Service (NHS). This is specific to the UK and supplements the information in
the body of the paper (which is more general).
Note that there are still significant variations in procedures and expectations across study
settings. Many factors seem to underlie this, including differences between large teaching
hospitals, where they are familiar with research, and district general and community hospitals that
focus primarily on practice, and the ways the local practices have evolved over time.
8.1 Registering on the Clinical Research Network
If going for full ethics approval (which is not always essential – see below), you might wish to
register your study with an agency such as the National Institute for Health Research Clinical
Research Network (NIHR CRN) Portfolio in England. This is a database of clinical research
studies that are eligible for NHS infrastructure for research (including NHS Support Costs). For a
hospital, there is substantial benefit from a study being registered, because it enables the hospital
trust to receive payment via a research network for participation in the study. It is not possible to
retrospectively register once ethical clearance has been obtained, so it is important to consider
options such as these at the proper time.
8.2 Getting formal access
The processes in the UK have a reputation of being stringent, lengthy and uncompromising.
Researchers commonly share frustrations and stories of research delays due to NHS research
ethics clearance; the worst we have heard took nearly two years which has serious consequences
for the researchers and research projects involved. However, this is rare and there are signs that
gaining approval for access is becoming more proportionate with better administrative procedures.
The National Institute for Health Research (NIHR) will only give funding after research ethics
clearance has been approved.
Currently there are three types of studies that can be conducted in conjunction with the NHS:
•
Research projects are typically projects that aim to find out new knowledge,
establish causal relationships and can be more open-ended and exploratory. For
example, we might want to test whether medicine A performs better than medicine
B, or we might want to explore how medical devices are designed and used in
30
general to include what patients think of them. These research projects require full
ethical approval from a REC (Research Ethics Committee).
•
Audit projects tend to compare services and procedures with an established measure
or procedure. For example, we might compare how a medical device is actually used
versus how it should be used in accordance with the instruction manual or we might
measure how staff conform to established procedures for washing their hands before
and after patient contact. Audit projects do not need approval from the REC, but
need local approval from the site you’re working with.
•
Service evaluation projects assess what a service does, how well it works and how
well it integrates with other services. A service would traditionally be a team, such
as physiotherapists or a hospital function such as the oncology ward. Service
evaluations do not need approval from the REC, but need local approval from the
site you’re working with.
In practice there are shades of grey between these different categories, and the REC
coordinators who administer the process should be contacted for advice on which category any
particular research project falls within. There can be some negotiation in the specifics of the
project to avoid one category and get it in another. Due to the delays and efforts needed for full
REC approval it is sometimes advantageous to adapt a study and argue for it to be included in one
of the other two, particularly where it seems that full REC approval is disproportionate to the
study’s aims and procedures. For example, watching surgeons in a theatre with no intervention to
understand their work is very different to asking patients about their problems and potential
embarrassment with medical devices.
To complicate matters it is not uncommon to get conflicting opinions on the same study, e.g.
someone might say it needs full ethical approval whereas someone else might say it doesn’t. In the
calculation, which is unsaid, is a sense of risk too: generally if you are speaking to patients or in
patient areas then your study will err toward full ethics but this does not necessarily need to be the
case; conversely if you are studying a back office procedure away from patients this is likely to err
toward a service evaluation or an audit. There is also some risk for the person advising you and it
is less risky for them to say that you need full REC approval. Finding someone that appreciates
research pressures, is knowledgeable and confident in the research ethic procedures, and is
pragmatic is invaluable. Such a person can carve a way through the research ethics maze.
If REC approval is needed then there are electronic forms that can be filled out on the IRAS
website (https://www.myresearchproject.org.uk/). These forms are substantial, detailed and
designed for quantitative clinical studies rather than exploratory qualitative “in the wild” HCI
studies. However, these forms can be filled out and approval granted. This has happened many
times and if the work is important and valuable then this should not put researchers off. The gold
standard for designing research studies includes designing it with clinicians and patients
If you or a team you are working with already has ethical clearance for a closely related
research project, it might be possible to obtain clearance for further research as an amendment to
that existing clearance.
8.3 Getting personal access
All three categories of research will need local approval, which normally consists of following
local R&D (Research and Development) forms and procedures that can vary from site to site. This
may include getting signatures from service managers, directors and finance officers. Again their
support and action can be facilitated if you have a senior clinician onboard.
The mechanisms one might come across to be granted access include the honorary contract,
letter of access and observer contract. The first two mechanisms will be granted to researchers
who are coming into the site from outside by R&D. The third is granted to researchers by Human
Resources so they are adopted into the administrative structure of the hospital rather than
remaining more of an outsider with local approval to access the site. The requirements for these
different types of access will vary between sites and within sites, e.g. some may require a criminal
record check, some will require references from your employer, a CV, and records of
immunisations.
31
Local approval may be given without involvement R&D if the study does not require REC
approval and a local manager takes responsibility of the project and advises Human Resources to
grant the researcher an observer contract for access. You need to check locally what is needed.
B. EXAMPLES OF CONSENT FORMS AND PARTICIPANT INFORMATION SHEETS
The following are examples of a consent form and a participant information sheet. Both should be
well laid out on hospital headed note paper (omitted here for reasons of confidentiality). [Further
examples will be provided online.]
32
[Include hospital logos at top]
A digital microscope for pathology: Observation of MDT meetings
Consent form (MDT lead)
Name of Researcher:
Please read this form carefully and initial the box next to each statement.
I confirm that I have read and understand the information sheet dated 11/12/09 (version
1) for the above study.
I have had an opportunity to consider the information, ask questions and clarify anything
that I do not understand.
I understand that my participation is voluntary and that I am free to withdraw at any time
without giving any reason.
I understand that a researcher from the University of Leeds will observe MDT meetings
that I lead.
I understand that a researcher from the University of Leeds may interview me about
MDT meetings that I have participated in.
I understand that data collected during the study may be looked at by the research team
for analysis, and by responsible individuals from Leeds Teaching Hospitals Trust
Research and Development Department for the purposes of monitoring the research
project.
I agree to take part in the above study.
Signature:
Date:
Full Name:
Researcher Signature:
Date:
33
A digital microscope for pathology: Observation of MDT meetings
Information sheet for staff
You are being invited to take part in a research study. Before you decide it is important for you to
understand why the research is being done and what it will involve. Please take time to read the
following information carefully. Talk to others about the study if you wish. Please ask us if there
is anything that is not clear or if you would like more information. Take time to decide whether or
not you wish to take part.
What is the purpose of the study and why have I been asked to take part?
The current study is part of a larger project, being undertaken at the University of Leeds and
funded by the National Institute for Health Research, to develop and evaluate a digital microscope
for pathology. An important element of the project is to understand current work practice, in order
to inform the design of the digital microscope. There is growing acknowledgement that if we wish
to develop interventions that lead to meaningful improvements, we need to establish an
understanding of the process that we are trying to support, in terms of the components and context
of the process and their impact on the process and its outcomes. In order to design a successful
digital microscope for pathology – one which fits with the work practices of pathologists, which
they are happy to use within their daily work, and which leads to an improvement in the processes
of care – we need an understanding of their current work practices and the context within which
they carry out their work.
We are requesting to observe MDT meetings where pathology data is presented, so as to
understand the functionality and features that a digital microscope needs to provide if it is to
support the MDT meetings.
Due to the number of staff members potentially present at MDT meetings, it is not practical to
obtain written consent prior to the MDT meeting from all staff members. However, written
consent has been obtained from the MDT lead.
Do I have to take part?
No. You do not have to take part unless you want to. Participation is entirely voluntary. If you
decide to take part you are free to withdraw at any time without giving a reason.
What will happen if I take part?
If all staff members provide verbal consent, the researcher will observe the MDT meeting and
make notes on what happens. No patient identifiable information will be recorded in the notes.
Where appropriate, they may also ask to video record or audio record the MDT meeting.
We will do our best to ensure that the observation does not interfere with the MDT meeting.
Following the MDT meeting, we may ask to interview you about the meeting. This can be either
immediately following the meeting or, if you prefer, at another time that is more convenient for
you. This interview will take no more than 30 minutes. The interview will be audio recorded. If
you agree to be interviewed, you will be asked to sign a consent form.
All collected data will be transcribed and analysed to see if there are any common themes. All data
will be anonymised, removing all personal information, so that you will not be identifiable. All
data will be treated in confidence. The aim is not to assess your work practices, but to understand
the functionality and features that a digital microscope needs to provide. If after the observation
session you change your mind about participating, you can choose for the data to be destroyed or
returned to you immediately. Data may be looked at by responsible individuals from Leeds
Teaching Hospitals Trust Research and Development Department for the purposes of monitoring
the research project.
34
Participants in the research will not be identified by name in any publications. Quotations may be
used in publications, but all personal information will be removed so that it is not possible to
identify you.
How can I find out about the results of the study?
A summary of the results of the study will be distributed to all staff members that participate in the
research.
What if something goes wrong?
While we anticipate no harm or distress to anyone as a result of this study it is important to state
that there are no special compensation arrangements. If you are harmed due to someone’s
negligence, then you have ground for legal action but you may have to pay for it. Regardless of
this if you wish to complain, or have any concerns about any aspect of the way you have been
approached or treated during the course of this study, the normal National Health Service
complaints mechanisms are available to you.
Who can I contact for more information?
If you have any questions about the study you can contact: [give contact details]
35
9.
STATEMENT OF PREVIOUS RESEARCH:
The authors confirm that this paper is novel. It has been written specifically in response to the call
for submissions to the special issue on “Turn to the Wild”. It draws on experience from studies
that are referenced in the text, but the theme of this paper is new, and is not addressed in any other
papers either published or under review.
36