Innerview®: Pre-Clinical Research
Content Validity Procedures
Paul E.Williams, PsyD; Mary Sichi, MA
Copyright © 2014 Pearson Education, Inc. or its affiliate(s). All rights reserved. Innerview and Pearson are trademarks, in the U.S. and/or other countries, of
Pearson Education, Inc. or its affiliate(s). DSM-IV-TR and DSM-IV are registered trademarks of the American Psychiatric Association. 9068-A 04/14
page 1
Introduction
Innerview® is a computer administered mental health clinical decision support system that gathers and
organizes mental health symptoms, to support diagnosis, treatment planning and monitoring. The patient
experiences the system as a structured interview. First person prompts are presented (e.g., I’ve been feeling…)
that the patient completes by selecting from a set of response options (e.g., sad, anxious, tired) to facilitate a
health narrative. Each response option is intended to reflect a DSM–IV–TR diagnostic criterion.
This report describes the procedure that was used during development of the response options to ensure that
they were valid indicators of DSM–IV–TR diagnostic criteria. (Nondiagnostic response options that are provided
for contextual purposes as the health story is told are outside the scope of this study.) The procedure relied
on ratings of content validity by mental health professionals. Due to the inherent difficulty of representing a
given criterion with a single word or short phrase, it was not expected that 100% of raters would endorse
every response option as a valid indicator of its target diagnostic criterion. The standard that was established
held that 95% of the response options would be endorsed as valid by at least 80% of the raters, and the
remaining 5% would be endorsed by a majority of raters.
Methods
Participants
Five mental health professionals served as raters for this study. All raters held PhDs in a psychological
discipline, including neuropsychology, clinical psychology, or research psychology and had ten years or more
experience in research and development and/or clinical practice. All educational backgrounds and experience
of the raters included training in and clinical application of the DSM–IV–TR. Raters were not compensated for
their participation.
Materials
Each rater received a spreadsheet that contained rating instructions, the initial pool of 219 response options,
and the target diagnostic criterion for each response option. The corresponding prompt was also provided
when necessary to provide context.
page 2
Procedure
Raters were asked to rate a response option 1 if they felt it accurately represented the target criterion or a 0
if it did not. For those items that were rated 0, the raters were asked to explain why the response option did
not reflect the criterion and to provide suggestions for improvement. (See the Appendix for rater instructions.)
All response options that received at least one individual rating of 0 were evaluated in a clinical and editorial
review by the development team, and were either changed, replaced, or removed, using the suggestions from
the raters. At the same time, as overall content was being revised, additional response options were created to
reflect DSM–IV–TR diagnostic criteria. The revised and new response options were sent to the same raters for
a second round of ratings.
Results
An analysis of the first round of ratings showed that 149 (68%) of the 219 response options were endorsed as
valid by all 5 raters and a total of 186 (86%) were endorsed by at least 4 raters. For a second round of ratings,
104 response options were delivered to the same raters. The response options included in this deliverable
were either added during content revisions or not endorsed 100% on the first round and subsequently revised.
In this round, 78 (75%) of the revised or added response options were endorsed by 100% of the raters, and a
total of 93 (89%) were endorsed by at least 80% of the raters. Of the remaining 11 response options, 8 were
endorsed by 60% of the raters, and 3 were endorsed by 40% of the raters.
After additional internal review, the final set consisted of 290 response options that represent diagnostic
criteria. Of these, 78% were endorsed as valid by 100% of the raters, and a total of 96% were endorsed by at
least 80% of the raters. All but three of the remaining response options were endorsed by a majority of the
raters. The average rate of endorsement of response options is 94%, with a range of 40%–100%.
Discussion
The results of the study that evaluated Innerview response options as representing DSM–IV–TR diagnostic
criteria provide strong evidence for Innerview’s content validity, based on the review of qualified mental
health professionals. Response option modifications were made to accommodate the contextual background
of the health story and clinical and editorial considerations. At the same time, revisions were made with the
goal of ensuring that the wording was as nonclinical as possible while still representing the diagnostic criteria.
The original goal (95% of the items evaluated endorsed by at least 80% of the raters) was met and surpassed.
A handful of responses with endorsement rates less than the originally proposed minimum threshold were
included in the final instrument, based on their ability to meet contextual, editorial, and/or nonclinical wording
requirements of the narrative.
page 3
Appendix
Rater Instructions
What is INNERVIEW?
Innerview is an interactive computer-administered system designed to help physicians quickly and accurately understand
and monitor the common mental health and psychiatric risks seen in primary care clinics. Either in the medical setting
or at home prior to their appointment with the physician, patients will use the interface to create a story that describes
their physical and psychological symptoms. The patient will tell her or his story by responding to prompts that complete
a statement in the first person. The patient’s story will be included in a report to the physician, along with a prioritized
symptom list and diagnostic suggestions based on DSM-IV-TR criteria.
What are we asking you to do?
Your clinical background and experience, including your knowledge of DSM-IV criteria, are crucial to assessing the face
validity of the words and phrases patients will be using to tell their health stories. The language of the prompts and
response options provided to the patient is designed to be as non-clinical as possible, while at the same time capturing
the essence of the criteria that will be necessary for the diagnostic suggestions, verifications and rule-outs that will be
generated for the physician’s report.Your task is to decide how well the response options reflect the criteria to which
they are linked. Please rate all response options listed next to the DSM-IV criterion to which they are linked. Response
options can be rated 0 (Does not reflect criteria) or 1 (Adequately reflects criteria). We would also greatly appreciate
any recommendations for wording or other comments you have-please enter these in the Comments/Recommendations
column next to the rating. Please rename this excel file with your name or initials before emailing it back to us.
Things to keep in mind:
Many of the response options you are rating will very closely match the wording of the DSM-IV criteria. These are
included for the purpose of conducting a thorough study, and will be relatively easy to rate. Others will require
more judgment.
Because the system is adaptive, patients may arrive at the same prompt/response options from different routes. Rather
than include every possible statement in this survey, we have included only the portion of the statement pertinent to the
DSM-IV criteria. Therefore you will mostly be rating incomplete sentences that reflect symptom endorsement. Within
each patient’s story, these phrases will appear within the appropriate context. For example, if the statement that you
are rating reads: “sleep difficulty”, you can assume that the complete statement in the story would be something like
“I’ve been experiencing sleep difficulty”. Where necessary, additional context for the response option you are rating is
provided in brackets.
In some cases there will be two or more possible words indicated with a slash (i.e., anxiety/nervousness/worry). Please
rate all statements that are possible from these options, and if you feel a particular word works better or worse than the
others (i.e., should not be included in your overall rating), include that feedback in the comments. Other statements
may include a word in parentheses, indicating that it may or may not be included in the response option the patient
sees. Please rate the response option with and without the parenthesized word, making a note in the comments column
if necessary.
All criteria may not be listed for every disorder included here. In these cases, we will be collecting that information
through other means, either through responses listed under other disorders or through direct ratings of duration,
severity, etc.
page 4