Computer Science, University College Cork, Ireland
A Reporting Framework
for Search Session
Evaluation
Cathal Hoare
Humphrey Sorensen
“Purple flower” - location: out on a walk
•
Vade Mecum
•
•
Latin, go with me
A book for ready reference
Something regularly carried
about by a person
From Ryen White’s Exploratory Search: Beyond the Query Response Paradigm
Exploratory
search
Example: Aid Exploratory Browsing
Query-by-Image
•
Provides starting point for users seeking
information about a place
•
Issues:
• Users are not familiar with their
surroundings and may not have
ontology to form productive queries.
Users’ curiosity is serendipitous. How
long can they spend searching?
Information Foraging Theory - Charnov’s Marginal Value
Theorem
Supporting
Enrichment
•
•
Find relevant information
faster
Find ‘informative
information’
Support triage in information exploration
Contextual
Memory Aide
•
Support users that want to explore on
mobile devices
•
Issues:
• Small screen size
•
Solution:
• Integrate recording of interesting
results and add ability to use them to
inform other queries
Support triage in information exploration
Contextual
Memory Aide
•
Support users that want to explore on
mobile devices
•
Issues:
• Small screen size
•
Solution:
• Integrate recording of interesting
results and add ability to use them to
inform other queries
How do we know when it
works?
Data Gathering
❖
Quantitive and Qualitative
❖
User Surveys for opinions and feedback
❖
Collect user actions to understand how interface
features are used
❖
Structured experiments in naturalistic settings
❖
Google Analytics used to collect actions before being
combined with framework resources to produce
report
Evaluating a mobile
application
When evaluating search-while-mobile
❖
Recruitment
❖
❖
Do they use one of our phones? (we must trust)
❖
Do we install on their phone? (they must trust)
❖
Logistics - will there be someone in the area to assist? What
about the weather?
❖
Recruitment - how to find users that are suitable?
❖
Challenge of creating realistic tasks and data?
Modelling Process
Recognise/
Accept
(Re)Define
Select Source
Formulate Query
Execute
Examine
Extract
Reflect/
Stop
Modelling Process
From Prof. Marchionini’s Information Seeking in Electronic Environments (1995)
Modelling Process
Modelling Process
Architecture
Reporting Interface
User Segmentation
Animated Process
Model
User Gain
Playback Controls
Conclusions
❖
There is a need to understand how mobile search is
conducted - we may be surprised
❖
Mobile search user interface evaluation presents both
familiar and new challenges
❖
We must understand how users use our solutions and
develop iteratively with less guess work
Thank You
Ádh mór