PERSIVAL
a System for Personalized Search and
Summarization over Multimedia
Information
1
PERSIVAL team members

Medical Informatics


Medical School – cardiac anesthesiology


Shih-Fu Chang
Center for Research on Information Access,
Health Sciences Library


Steven Feiner, Luis Gravano, Vasileios Hatzivassiloglou,
Kathleen McKeown
Electrical Engineering


Desmond Jordan
Computer Science


James Cimino, Carol Friedman, Steven Johnson
Judith Klavans, Pat Molholt, Elizabeth LaRue, David Millman
Cognitive Science

Andre Kushniruk (York), Vimla Patel (Medical Informatics)
2
Students

Computer Science









Min-Yen Kan
Simon Lok
Smaranda Muresan
Sergey Sigelman (programmer)
Medical Informatics




Eugene Agichtein
Michel Galley
Noemie Elhadad
Panos Ipeirotis
Michael Charney (programmer)
Eneida Mendonca
Lyudmila Shagina (programmer)


Yoon –Ho Seol
Di Wang
Electrical Engineering

Shahram Ebadollah
3
Goals

Personalized access to distributed, multimedia
resources




information access
information fusion
information understanding
Provision of patient-specific information




interaction within context
for clinicians, at the point of patient care
for patients, in terms that can be understood
online patient record serves as a user model
4
Rounds



Patient-centric
Current: Access
to clinical data
Missing:
Access to
literature that
fits patient
profile
5
Unique Contributions
System focus: querying, search, presentation
 Questions are asked within the context of patient
information
 A uniform, personalized view of distributed
resources on the internet through querying and
browsing
 Concise, patient specific presentation of relevant
information through summarization
 Access to textual documents linked with access to
multimedia video: library of echocardiogram
 Dynamic layout of heterogeneous information
6
Where are we now?


Prototypes of each system component
Local library of journal articles and consumer
health sites





Facilities for distributed online search
Scenarios for development and testing with three
patients
Initial system integration


20 highly ranked journals
30,000 articles
Restricted to a limited set of examples
Formative evaluation of system components
7
Overall Integrated Demo

What is the prognosis for atrial fibrillation and
myocardial infarction?





Clinician as user
On viewing patient discharge summary
Journal articles: controlled clinical trials
Re-ranking of search results using patient record
What is the treatment for endocarditis?



Patient as user
On viewing lab results
Consumer health information
8
User Interface Focus
Asking questions within context of patient
record
 Evidence based medicine to suggest
questions
 Selection of relevant information from the
patient record


Demo of Medlee
11
Distributed Search

Meta-searcher for automated interaction with
heterogeneous, distributed sources

Use of machine learning and query probes to
automatically determine topics of distributed
sources

Information extraction from web pages
13
Re-ranking search results

Re-rank articles which better match the
patient record -> more relevant articles

Use natural language techniques to analyze
article and patient records

Articles with many terms and values
matching the patient record score higher
15
Presentation Focus

Multimedia summarization






Journal articles, consumer health, video
Highlight retrieved results to help user in finding relevant
information
Personalize summary for patient
Define unknown terminology
Methods for summarizing and search echocardiograms
Dynamic layout and organization of results

Explicitly control level of detail
17
Milestones

Where we said we would be vs. where we are:


Year 2: skeletal end-to-end system prototype with minimal
personalization, interactivity, and limited coverage of structured
documents
Year 3: Extend to full prototype, with increased personalization,
interactivity, limited coordination of multimedia, full range of
structured documents, and restricted coverage of consumer documents



Use of evidence-based medicine, machine learning to categorize
sources by topic, provision of definitions, thin-client computing to
allow PERSIVAL on mobile, hand-held devices
Year 4: Scale prototype with increased robustness, personalization,
coverage to full range of documents and fully integrated multimedia.
Coordinate with end-to-end evaluation
Year 5: Refine components based on Year 4 evaluation. Transition
PERSIVAL to deployment in cooperation with Health Sciences
Library
18
Plans for next year

Increase robustness





Extend question asking to different patient contexts, different
question types
Allow summarization and re-ranking of online articles
Extend journal summarization to new genres
Extend layout to dynamically incorporate different types of
summary input
Multimedia integration



Implement scenarios for integration
Increase interaction with video summary in layout
Enhanced multimedia prototype
19