A Reusable Template for Evaluating
Point-of-Care Information Products
Health Sciences Library System • University of Pittsburgh
1
Discussion of EBM Products on MEDLIB-L
How many questions have there been
about EBM resources in recent years?
20
15
10
5
0
1996 1997 1998 1999 2000 2001 2002 2003
How often were specific
products mentioned?
(Jan-Nov 2003)
UpToDate
DISEASEDEX
MD Consult
InfoRetriever
Clineguide
Skolar
Clinical Evidence
DynaMed
PDxMD
Zynx
7
6
6
3
3
2
1
1
1
1
Characteristics of
Point-of-Care (POC) Products
• Provide access to succinct evidence-based
diagnosis and treatment information
supporting care of individual patients
• Oriented to…
 primary care physician
 specialist seeking information outside his/her
own specialty
Health Sciences Library System • University of Pittsburgh
3
Characteristics of POC Products
The Core Document
• Core document is a topic review presented in a
product-specific, standardized structure.
• Core document is evidence-based.
• Core document content is subdivided for quicker
access.
• In addition to the core document, product may
incorporate other resources (journals, textbooks,
drug information, news or alerts, calculators).
Health Sciences Library System • University of Pittsburgh
4
Steps in This Study
2
1
IDENTIFY
products of
interest
DEVELOP
template
Test products
Score using
different criteria
Result: Second
set of weighted
scores
product reps
4
template
(structured
question format)
7
ADAPT
3
INTERVIEW
scoring
instrument
attribute
checklist
product/
attribute grid
5
LITERATURE
REVIEW
6
WEIGHTED
SCORING
Wish List
POC Products in this Study
• Product selection criteria
 Ostensibly, all have characteristics listed above.
 Product or vendor known directly or by reputation
 Willing to be evaluated or tested against other products
• Products evaluated
 Micromedex’s DISEASEDEX
 InfoRetriever
 Ovid’s Clineguide (since subsumed into Skolar)
 MD Consult’s PDxMD (since renamed FirstConsult)
Health Sciences Library System • University of Pittsburgh
6
Development of Evaluation Template
• Topic sources
 Informal brainstorming among project librarians
 Brief exposure to some of the products
 Comparison with known products
• Format
 Questionnaire
 Structured and hierarchical
 Reusable in further or additional product
evaluations
Health Sciences Library System • University of Pittsburgh
7
Template Topic Areas
1. Content
 Scope
 Methodology and core document structure
 Editorial practices
2. Architecture/Navigation
 Content storage and organization
 User experience
3. Integration with other environments
 PDAs
 Local clinical information systems
4. Niche (self-perceived product strength)
Health Sciences Library System • University of Pittsburgh
8
Interviews
• Questionnaire was used in scheduled vendor
interviews (3 interviews in person, 1 by phone).
• Group situation: 1-2 product developers or
representatives were interviewed by 2-3 librarians.
• One vendor followed up by e-mail with additional
information.
• Product/attribute comparison grid was created in
Excel and populated as interviews were completed.
Health Sciences Library System • University of Pittsburgh
9
Literature Review
• Challenges identified
 Information needs
 Information-seeking behaviors
 Barriers to implementation
• Solutions proposed
 Evidence Cart (Sackett & Strauss, 1998)
• Remaining questions
 Impact on clinical practice
 Product comparison
Health Sciences Library System • University of Pittsburgh
10
Wish List Based on Literature Review:
What would the ideal product be like?
• Broad scope
• Help with queries and
search strategies
• Rx recommendations
• Drug information
• Practice guidelines with
automatic EBM updates
• Synthesis of evidence
• Patient education modules
• On demand at point of care
• Linked directly to relevant
literature
• Customized for local use
• Flexible decision-making
models
• Fast
• PDA compatible
• Conformity to hardware and
software standards
Product Comparison Scoring Instrument
• Scores for 6 categories (4 in original questionnaire)
1. Content
2. Audience
3. Integration
4. Architecture/Navigation
5. Retrieval (originally under Content)
6. Quality Control (originally under Architecture/Navigation)
• Scores weighted to favor Wish List compliance and
non-proprietary standards
• Scoring instrument can be tailored to local preferences.
 Different weights  different scoring outcomes
Health Sciences Library System • University of Pittsburgh
12
Scoring Scheme
• Attributes were listed for each category.
• For every product, each attribute was assigned a
score.
 Default: 1 if attribute present, 0 if not present
 Scores weighted for attributes of particular interest
• 2 points for presence of each Wish List attribute
• 0 points for presence of an attribute if it involved a proprietary
standard
• Points for scores in all categories were totaled.
Health Sciences Library System • University of Pittsburgh
13
Example
Scoring Instrument
for Attributes Associated with Content
CONTENT
1. Patient Ed modules
2. Practice Guidelines
3. Clinical Decision Rules
4. Outcomes Measures
5. Differential Diagnosis
6. Broad Scope – extensive documents
a. >3000 core documents
b. >2000 core documents
c. >1000 core documents
d. >500 core documents
e. <500 core documents
7. Topics must include Drugs
8. Medical Conditions
9. Differential Diagnosis
10. Procedures
11. Subtopics
12. Graded level of evidence
13. Other: Special Tools
2
2
1
1
1
2
1
1
1
0
2
1
1
1
1
1
1
Note: Shaded areas indicate Wish List attributes.
Example
One Scoring Outcome (Total Points)
80
70
60
50
40
30
20
10
0
DISEASEDEX
InfoRetriever
Clineguide
PDxMD
Example
One Scoring Outcome (Points by Category)
15
DISEASEDEX
InfoRetriever
Clineguide
PDxMD
10
5
0
Arch/Nav
Audience
Content
Integration
Quality
Control
Retrieval
Format
Limitations of This Work
• Interview data were varied and contained gaps.
• There was either no or limited hands-on use of
products themselves.
• Outcome depends on scoring scheme; ours may not
be optimal for other researchers or libraries.
• This methodological approach disregards real
world constraints such as:
 Product cost
 IP restrictions and other implementation issues
Health Sciences Library System • University of Pittsburgh
17
Recap
1. Experience with familiar products can be used to
generate a structured, hierarchical template for
use in product evaluation.
2. Requirements for point-of-care products can be
distilled from a literature review.
3. Product characteristics can be scored using a
locally weighted instrument.
4. Template can be adapted and reused with
additional products or for further evaluation
based on product testing.
Health Sciences Library System • University of Pittsburgh
18