The
Re:Search Engine
Jaime Teevan
MIT, CSAIL
“Pick a card, any card.”
Case 1
Case 2
Case 3
Case 4
Case 5
Case 6
Your Card is GONE!
People Forget a Lot
Change Blindness
http://www.usd.edu/psyc301/ChangeBlindness.htm
Change Blindness
http://www.usd.edu/psyc301/ChangeBlindness.htm
Re:Search Engine
?
Merge Old and New Results
Old
Merged
New
We still need magic!
Overview
♠
♠
♠
♠
♠
♠
Memorability study
Recognition study
Assumptions
Implementation issues
Evaluation issues
Choose your own adventure
Memorability Study
♠ Participants issued
self-selected query
♠ After an hour, asked
to fill out a survey
♠ 129 people
remembered
something
Data Analysis
♠ Probability of being remembered
♥ Anything? # of words? # of fields?
♥ Features
♣Result features: clicked, not clicked, last clicked,
rank, dwell time, frequency of visit, etc.
♣Query features: query type, query length, # of
search in session, elapsed time, etc.
♠ Remembered rank v. real rank
♥ Map remembered rank to real rank
“Memorability”
P(Remem|R,C)
Clicked - C
Not clicked
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
2
3
4
5
6
Rank - R
7
8
9
10
Remembered Results Ranked High
12
Actual Rank
10
8
6
4
2
0
-2 -2 0
2
4
6
8
Remembered Rank
10
12
Recognition Study
♠ Same set-up as Memorability Study
♠ Follow-up survey: Results the same?
♥ Case 1: New results 19%
♥ Case 2: Random 4 same 38%
♥ Case 3: Clicked to top 41%
♥ Case 4: Same results 66%
♥ Case 5: Intelligent merging 81%
♠ 165 people completed both steps
Assumptions
♠
♠
♠
♠
♠
♠
Re-search v. search
Memorable v. relevant
Results change v. stay the same
Hide change v. show change
Forget v. remember as forgettable
Merge v. identify old or new
Why?
How to test?
What if I’m wrong?
Implementation Issues
♠ Page of cached result may disappear
♠ Multiple result pages
♠ Identifying repeat queries
♥ Exact query may be forgotten
♥ User identified
♥ Search sessions are not repeat queries
Evaluation Issues
♠ Various goals to test
♥ Does a merged list look like the original?
♥ Does merging make re-finding easier?
♥ Is search improved overall?
♠ Lab study
♥ How to set up re-finding task?
♥ Timing differences significant enough?
♠ Longitudinal study – What to measure?
♠ What are good baselines?
Choose Your Own Adventure
♠
♠
♠
♠
♠
♠
Re-search v. search
Memorable v. relevant
Results change v. stay the same
Hide change v. show change
Forget v. remember as forgettable
Merge v. identify old or new
♠ Implementation issues
♠ Evaluation issues
Choose Your Own Adventure
♠
♠
♠
♠
♠
♠
Re-search v. search
Memorable v. relevant
Results change v. stay the same
Hide change v. show change
Forget v. remember as forgettable
Merge v. identify old or new
♠ Implementation issues
♠ Evaluation issues
(Done)
Hide Change v. Show Change
♠ Why I think change should be hidden
♥ Example: dynamic menus
♠ How to prove
♥ New results better, called the same or worse
♥ Baseline for testing – 2 lists, change explicit
♠ What if we should show change?
♥ Memorability suggests changes to highlight
♥ Other applications where want to hide change
(Done)
Memorable v. Relevant
♠ Why I think memorability is important
♥ Relevance at a future date is what matters
♥ Necessary to hide change
♠ How to prove
♥ Baseline for lab study with target first
♠ What if relevance is what’s important?
♥ Mapping between memorable and relevant
♥ Useful related work on implicit feedback
(Done)
Re-search v. Search
♠ Why I think people repeat searches
♥ Information seeking literature
♥ Re-finding consistently reported as a problem
♠ How to prove
♥ Study shows prefer to follow known paths
♥ Search log analysis
♠ What if people just want to search?
♥ Memorable results ranked first
♥ Other domains where list consistency matters
(Done)
Merge v. Identify Old and New
♠ Why I think results should be merged
♥ Information need not necessarily one or other
♥ People don’t like to do extra work
♠ How to prove
♥ Search log analysis
♥ Look at what people do in longitudinal study
♥ Lab study – timing becomes an issue
♠ What if people want to identify query type?
♥ Other applications where merging is useful
(Done)
Results Change v. Stay the Same
♠ Why I think results change
♥ How search engines work
♥ Personalization and dynamic content
♠ How to prove
♥ Track query results
♠ What if results don’t change?
♥ Probably will in future applications
♥ Existing applications where lists change
(Done)
Forget v. Remember as Forgettable
♠ Why I think people forget
♥ Visual analogy
♠ How to prove
♥ Lab study – Do people find new information?
♥ Longitudinal study – Ever click on new result?
♠ What if remember as forgettable?
♥ Build better model of memorability
♥ Highlight important changes
(Done)
Implementation Issues
♠ Page of cached result may disappear
♠ Multiple result pages
♠ Identifying repeat queries
♥ User identified
♥ Search sessions are not repeat queries
♥ Exact query may be forgotten
(Done)
Evaluation Issues
♠ Various goals to test
♥ Does a merged list look like the original?
♥ Does merging make re-finding easier?
♥ Is search improved overall?
♠ Lab study
♥ How to set up re-finding task?
♥ Timing differences significant enough?
♠ Longitudinal study – What to measure?
♠ What are good baselines?
(Done)
Jaime Teevan
[email protected]
Strategies for Finding
Teleporting
Orienteering
Why Do People Orienteer?
♠
♠
♠
♠
The tools don’t work
Easier than saying what you want
You know where you are
You know what you find
Structural Consistency Important
All must be the same to
re-find the information!
New name
Absolute Consistency Unnecessary
New name
Focus on search result lists
Query Changes
♠ Most changes are simple
♥ Capitalization
♥ Phrasing
♥ Word ordering
♥ Word form
♥ New queries shorter
♠ What about longer time horizons?
♠ Recognition v. recall
Result List Changes
♠ Tracked 10 queries on Google for a year+
♠ 1.18 of top 10 disappear each week
♠ Rate of change likely to increase, e.g.:
♥ Raw personalization
♥ Relevance feedback
♠ People forget their queries
♥ 28% of queries forgotten within an hour
5/
25
/2
00
5
5/
18
/2
00
5
5/
11
/2
00
5
5/
4/
20
05
4/
27
/2
00
5
4/
20
/2
00
5
4/
13
/2
00
5
Example: “neon signs”
0
2
4
6
8
10
12
14