iOutcomes - Guidelines for the Production of eCOA / ePRO
Measures
Background
iOutcomes, as part of Isis Innovation Ltd is the copyright holder, distributer and provides support in
the use of a number of high quality Clinical Outcomes Assessments (COA’s) the majority being
Patient Reported Outcome (PRO) measures. These PRO measures are currently available in paper
completion-based formats and have been developed and psychometrically validated following
internationally accepted standards. iOutcomes appreciates that there is a growing demand for the
administration of PRO measures in an electronic format or mixed paper and electronic format.
iOutcomes is keen to support licensees in converting paper-based PROs to electronic delivery
formats. This document outlines the procedures that should be followed when adapting an
iOutcomes paper-based PRO measure for administration in an electronic format.
Definition of ePRO Measures
The collection of PRO data using electronic data capture modes of administration is defined as an
ePRO. ePRO modes of administration include but are not limited to:






Computers (including web-based capture and data collection)
Tablets
Palms
Mobile Phones /smart phones
Telephone - Voice recognition
Bespoke devices
Compared to paper-based PRO measures, ePROs may lead to less administrative burden, high
patient acceptance, higher levels of administrations, avoidance of secondary data entry errors,
easier implementation of skip patterns, and more accurate and complete data.
Modification of Paper-Based PRO Measures for Electronic Data Capture
There are three levels of modification:
1) Minor modification
2) Moderate modification
3) Substantial modification
This guideline focuses on minor modifications applicable to visual, screen-based reproductions of
our measures as ePROs. iOutcomes discourages users from making moderate and substantial
modifications to its paper-based PRO measures as these changes require equivalence and full
psychometric testing respectively, and may result in versions which are not comparable to the
original paper-based PRO measure, making the comparison of data across studies difficult.
A minor modification is not expected to change the content or meaning of the items and response
scales. Migrating from paper to screen text without significantly altering font size, item content,
recall period, or response options would qualify as minor modification. Users should be aware that
for some of the iOutcome PRO measures, the use of a particular format is intentionally designed into
the PRO. For example, the use of larger font in the Oxford Hip and Knee scores is deliberate so as to
facilitate the completion by the respondent demographic of the PROs. It is therefore important that
such design features are retained in any migration to ePRO. Minor modification allows for changes
to the number of items visible to the respondent at any one time e.g. one item per screen shot and
also allows for minor changes to instructions such as changing “tick the box” to “choose the
response”, in order to make the instructions relevant to the media of administration.
The introduction of scroll bars to read single items or splitting items onto more than one screen
would not be classified as minor modification and is discouraged.
Once minor modifications have been made to the ePRO, it is a requirement that iOutcomes review
the final screenshots to ensure that the modifications have not exceeded any of the limitations
outlined above.
Comparability evidence required for an Official version of an outcome
questionnaire
When an ePRO version of one of our questionnaires is to be developed where there is any
opportunity the arising ePRO data may be used in a regulatory environment, then an Official version
must be created and the following testing applied. Where the arising data is strictly not going to be
used in any regulatory environment then iOutcomes may consider allowing a Developer Authorised
version to be created but will still require approval by iOutcomes before it can be used. The decision
of whether iOutcomes will allow an Owner / Developer Authorised version is solely at iOutcomes
discretion.
Even though it is envisaged that minor modifications will have no impact on the way a patient
responds to a question, it is important that there is evidence to support this standpoint.
Two levels of evidence are required when making minor modifications to paper-based PRO
measures for administration as ePROs:
1) Cognitive Debriefing
2) Usability Testing
Cognitive Debriefing
Cognitive debriefing refers to a “think aloud” approach during which respondents in the target
population complete the ePRO measure and are asked to describe in their own words what the
instructions, questions and response options mean to them in their own words. Cognitive debriefing
should be conducted with 10 respondents in the target population
© Isis Innovation Limited 2015. All rights reserved.
The purpose of conducting cognitive debriefing following minor modification is to establish whether
the mode of administration has in some way affected the respondent’s interpretation of an item or
response option. It is easy to assume that if an item hasn’t changed then there should be no changes
in interpretation. However, if a question asks the respondent about the use of their hands in
performing delicate tasks or about their grip strength, and the patient is using a thin stylus for the
first time in their life, one could also envisage how the respondent may interpret the question
slightly differently.
The results of cognitive debriefing should be fully documented in a table which should include all the
verbatim responses of the respondents and their answers to each question. The responses should
be compared to the English items to ensure that the interpretation reflects the intended meaning of
the item.
Usability Testing
The purpose of usability testing is to examine whether respondents from the target population are
able to use the device and software and complete the ePRO correctly. It is important to remember
that some respondents may be using computer software for the first time in their life and so what
would seem like a very simple instruction to a regular user may seem completely alien to a beginner.
The process should be tailored to the type of software and media being used and should also
consider the complexity of the ePRO. It is recommended that the respondent completes the ePRO in
the presence of a researcher and follows a talk aloud process, explaining how they interpret the
items and instructions as they work their way through the ePRO. The respondent should be
encouraged to express any difficulties they are having or if any of the instructions are confusing.
They should be reassured that it is not a test of their ability to understand and use the software but
that it is a test that the software has been written clearly and is not confusing. The process should
be fully documented and all comments and problems noted. If any major problems become
apparent it will be necessary to consider alternative instructions or software in which case the
process of usability testing will need to be repeated.
It is recommended that usability testing is conducted on at least 10 respondents from the target
population. It is acceptable if these are the same respondents who take part in the cognitive
debriefing exercise.
If the results of both cognitive debriefing and usability testing are acceptable, the ePRO has
demonstrated evidence that it is comparable to the original paper-based PRO measure and can be
administered to patients.
Comparability evidence required for a Developer/ Owner Authorised version
of an outcome questionnaire
In discussion with and agreement of iOutcomes, an ePRO developer of one of our outcome
measures may be allowed to create (for approval before use) a Developer / Owner Authorised
version of one of our Questionnaires. A Developer/ Owner Authorised version should only be used
for purposes where the arising data is not going to be used in a regulatory environment. You should
discuss this with iOutcomes. If iOutcomes agrees that a Developer / Owner Authorised level of
migration testing is acceptable, then the testing process is simplified and the testing with patients
with the target condition is not required. However, iOutcomes and the developer/author of the
© Isis Innovation Limited 2015. All rights reserved.
questionnaire will still review the ePRO version to ensure the questionnaire has been migrated to
the satisfaction of iOutcomes (the Owner). A working version of the ePRO is therefore ideally
required but ePRO developers are encouraged to engage iOutcomes at the earliest possible
opportunity. If iOutcomes identifies issues with the developer ePRO version then the ePRO
developer must comply with iOutcomes change requests in order to receive a Developer / Owner
Authorised status leading to authorisation for its use in accordance with the licence of copyright.
If at a later date a developer of a Developer / Owner Authorised ePRO version of one of our
questionnaires wishes to use the same ePRO in a study where an Official version is required, then
the Developer/ Owner Authorised ePRO can be simply promoted to an Official version by
successfully undertaking usability testing and cognitive debriefing with subjects in accordance with
the process for Official version testing. This should be undertaken with iOutcomes.
Translation and Linguistic Validation of ePRO Measures
Once the ePRO measure has been tested and finalised it is then possible to translate and
linguistically validate the ePRO measure for use in other countries. Screenshots of the source UK
English ePRO measure should be used for the translation, and the methodology should conform to
iOutcome’s Translation and Linguistic Validation Guidelines.
If a paper-based PRO measure exists in the target language, the text that is identical in the ePRO
measure must be retained. Any remaining minor changes (e.g. changing “tick the box” to “choose
the response”) should be translated using the following methodology:




One Forward Translation
One Back Translation
Back Translation Review
Two sets of Proofreading
Please refer to iOutcome’s Translation and Linguistic Validation Guidelines for a description of these
steps or contact iOutcomes for further advice or assistance.
References:
This guideline is based on the Coons et al (2009) paper discussing the evidence needed to establish
equivalence between paper-based PRO measures and ePRO measures. iOutcomes recommend that
users of this guideline also refer to this paper when developing ePRO measures.
Coons SJ., Gwaltney CJ., Hays RD., et al. (2009). Recommendations on Evidence Needed to Support
Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO)
Measures: ISPOR eRPO Good Research Practices Task Force Report. Value in Health 12(4); 419-429.
Acknowledgements:
This user guide was created in association with PharmaQuest Ltd.
www.pharmaquest-ltd.com
© Isis Innovation Limited 2015. All rights reserved.