Ken Hinckley, Shengdong Zhao, Raman Sarin,
Patrick Baudisch, Edward Cutrell, Michael Shilman &
Desney Tan
oronto
In situ search for digital ink
 InkSeine
 Ink +
 Seine
:
 -n: a fishing net
 -v: to fish with a seine
 = insanely cool
ink search!

InkSeine lets the user
fish for useful information
directly from ink notes.
Fishermen catching salmon
on the Columbia River
using a seine.
(courtesy Wikipedia)
Active note taking

XLibris—pen-based system for active reading
(reading + critical thinking + learning)

InkSeine combines pen-based note taking with
searching, linking, collecting & sensemaking:
Core interaction pattern:
Ink-Search-Gather

Ink – excels at capturing
user’s thoughts & ideas
with minimum distraction
from formatting issues
Ink
Ink-Search-Gather
interaction pattern
 Search
– digging through
the file system on a tablet
is really a pain
 Active note taking benefits from
low overhead access to
supporting materials & related
docs
 Focus on personal search, but
support Web as well
Ink
Ink-Search-Gather
interaction pattern
 Gather
– Make it easy to
use what you find by
pulling content directly into
unstructured notes
 Thumbnail hyperlinks to
documents
 Clippings from documents
 Queries that can be revisited later
Ink
Ink-Search-Gather
interaction pattern
 InkSeine
unifies these
three elements in a fluid
user experience that is
integrated directly with
inking
Ink
In situ search

Leverage preexisting ink to initiate search
 No context switch to “search app” – stay in the flow
 No tedious copying of information to a “search box”

Queries as first class objects, commingled with ink notes
 Flag some ink for search for later
 Visible and salient search history as part of notes
 Queries can be copied, pasted, moved, etc.

Interleave inking, searching & gathering
 No barrier between inking / searching – note ideas while
searching, side-by-side searching, span app boundaries for
content gathering

Tightly couple queries with application content
 Queries naturally persist in originating context
 Easy re-visitation of prior queries
 Results of queries become new content
Rich tradeoffs in cost
structure of sensemaking

Continuum of time cost – can defer at any point:




1 second: Capture thought “I should search on this”
5 seconds: Trigger search & see initial results list
10s: Open a document from result list, or drag result into notes
10s-30s:





Inspect details of a few results
Apply filters to results
Add a word to the query
Grab a snapshot from a short document
Correct occasional reco errors – but Tablet’s reco is amazingly good!
 Longer: sufficient depth for common info needs; not mired by
last 10% of complex queries that require full featured search
tools
 Scroll through longer results lists
 Inspect result documents to see if they meet information needs
 Sideways searches based on Title, Date, or Author of a result
InkSeine system

VIDEO
Paper Prototype
Paper Prototype
Paper Prototype
Findings

Existing desktop search tool difficult to use
 Difficult text entry
 Enter and correct text
Findings

Existing desktop search tool difficult to use
 Difficult text entry
 Enter and correct text
 Annoying context switch and flow interruption
 Switch among apps and context
Findings

Existing desktop search tool difficult to use
 Difficult text entry
 Enter and correct text
 Annoying context switch and flow interruption
 Switch among apps and context
 Tedious information gathering
 Difficult to link to documents and gather pieces of information
Notebook Control
In Situ Search
Breadcrumb
Breadcrumb
Breadcrumb
Trigger Personal Search
Search Panel
Query Area
Query Area
Result List
Result List
Result List
Result List
Focus Result
File and Timeline Filters
Filters
Gestures for Filters
Tap - Select
Cross – Multi-select
Disjunctive cross – Multi-select
Cross & Turn Right – Exclude
Work Flow & Occlusion
Selection
Correction
Search
Work Flow & Occlusion
Work Flow & Occlusion
Crossing Widgets
Resize Search Panel
Examine Document
Document & Tracking Menu
Multiple Search Panels
First Round Observations

Gesture discovery problem
First Round Observations


Gesture discovery problem
Context preservation & occlusion problem
First Round Observations



Gesture discovery problem
Context preservation & occlusion problem
Recognition problem for out-of-vocabulary word
UIST
?
List?, Dist?, Gist?, Mist?, Hist?
Second Round Observations
Highlighter hints
 Smart positioning of search panel
 Personalized indexer

Summary







In situ search experience
Optimum workflow / maximum flexibility
Enable rich tradeoffs
Gather content
Minimize search screen real estate
Span application boundaries
Tailored to pen input
Future Directions

Pull in materials for searches from notes
Future Directions


Pull in materials for searches from notes
Support vector-based searches
Questions?