The task-centric revolution.
Weaving information into
workflows
Dagobert Soergel
College of
Information Studies
LACASIST 2009-02-05
Disclaimer
This talk pulls together many ideas
many old, some new
many other people’s, some mine
some implemented here and there,
some still awaiting implementation
The point is the total vision
2
Computer work must be organized
not around applications
but around tasks
Functional or vertical integration
3
Tasks are accomplished collaboratively
Collaboration
Cross-user or horizontal integration
4
Functional Integration
Collaborator
Search
Sense-making
Creating
History / PIM
Ontology S.
User in focus
Search
Collaborator
Search
Sense making
Task oriented processing
Sense-making
Creating work product
Creating
History-aware personal
information store. Planning
Ontology Support
Collaboration
History / PIM
Ontology S.
Outline
• The digital library of the future ─ DELOS
• CLASS. A Collaborative Lesson-planning
And Search System ─ Katy Lawley
• Sense-making ─ Pengyi Zhang
• History-aware personal information store
─ Anita Komlodi
• Relevance relationships ─ Xiaoli Huang
6
The digital library of the future:
A look at the DELOS vision
DELOS Network of Excellence in Digital Libraries
Now the DELOS association
Funded by the EU, >50 DL research groups as members
7
The digital library of the future
A broad system of interlinked information & services that
• Is person- and task-centric
• Provides rich seamlessly integrated functionality
DL = information + tools to process information
• Supports process execution and workflow
in business, government, and daily work
• Supports user-to user communication & collaboration
• Supports users as consumers and as contributors
Supports massive collaboration leveraging many small
contributions to construct very large resources
(e.g., Wikipedia, social tagging)
• Thus, supports new ways of intellectual work
8
Issues in research and teaching
• User studies to
• develop a truly user-centric view of computer-supported
functionality
• learn how users could and would collaborate if properly supported
• Methods for deriving rich user profiles with minimal active user
involvement, including discovery of users’ conceptual structures
• Interfaces that use a well-structured ontology
(a faceted classification, an entity-relationship model) to
• help users analyze a problem they face or a search topic
• help users over time to assimilate the ontology structure
• Usability, effectiveness, and impact of such interfaces
9
Issues in research and teaching
• The core issue of ontologies / classifications /
tagging schemes
• Capturing individual and shared understandings of the
concepts in a domain
• Expressing these understandings in a structured way to
communicate a common understanding within a community
• Supporting individual users in developing their own ontology
• Collaborative development of ontologies
• Automatic integration or harmonization of ontologies
• Integrated storage and search of documents and data
in many formats and degrees of structure
• Annotation, communication, and collaboration functionality
• Seamless integration of multiple systems and tools
10
CLASS
A Collaborative Lesson-planning
And Search System
Katy Lawley
11
CLASS functionality
The design
of a lesson planning system
12
1 A knowledge organization infrastructure that
fosters development of shared understandings and
supports organization of materials.
2 Mediated access to digital libraries containing many types of
materials with powerful, ontology-enhanced search
3 Intellectual property rights and access management
4 A collaborative template-based authoring system with annotation
facility
5 Communication functionality
6 Integration with the teachers total work environment, including
lesson plan evaluation
7 Interaction history. Customization and personalization, adaptive
8 An interface that provides easy access to all this functionality
9 Extensibility
13
CLASS walk-through
14
Figure 1. Opening screen for creating a lesson plan
15
Figure 2. Lesson plan outline
16
Figure 2. Lesson plan outline
17
Figure 3. Selection of applicable standards
18
Figure 4. Selection of applicable vocabulary words
19
Figure 5. Query formulation for a theme in the lesson planning module
20
Figure 6a. Browse the thesaurus
21
Figure 6b. Descriptor found in detailed information for soldiers
22
Figure 7. Display of results
23
Figure 8a. Segment Display. Detailed information about the segment
24
Figure 8a. Segment Display. Detailed information about the segment
25
Figure 8b. Segment Assessment. The full form
26
Figure 8b. Segment Assessment. The full form
27
Figure 8c. Segment Display and Segment Assessment sharing the screen
28
Figure 9. Interview segment added to lesson plan module
29
1 Knowledge organization
infrastructure
Foster development of shared understandings among users and
support organization of materials. This includes
1.1 A lesson plan template that lays out the components of a
lesson plan.
1.2 A query template to assist in formulating queries for learning
objects and other materials.
1.3 A material appraisal form using the same structure as the
query template, same structure as 1.2.
1.5 A hierarchy of educational standards from several jurisdictions.
1.4 A thesaurus / classification of topics in each accessed
database domain that is useful for searching and for giving
teachers content ideas.
30
6 Integration with the teacher’s
total work environment
6.1 Associate lesson plans with time slots for a given class.
6.2 Prepare requests for permission to use materials as needed.
6.3 Have material, such as quizzes, prepared in sufficient number
or arrange for electronic administration.
6.4 Order equipment needed as specified in the lesson plan.
6.5 Notify the school library media specialist of assignments that
require the use of the school library media center.
6.6 Have a function for recording grades and importing detailed
standardized test scores (broken down by educational
standard).
31
Experience with CLASS
• Teachers were able to search the archive.
• Teachers used the materials appraisal form for their
own purposes (not enough critical mass for
collaboration).
• Teachers were able to develop well-structured lesson
plans in a short time.
• Teachers indicated that they do not collaborate much
at present but would be inclined to collaborate more if
they had a tool like CLASS.
32
Sense-making
Pengyi Zhang
33
Beyond Search
Beyond search:
Users need support for the next step:
making sense of and applying
large quantities of information found.
34
Outline
• What is sense-making?
• Examples
• Theoretical framework and
comprehensive sense-making model
• Findings from a pilot study
• Conclusions on the design of sensemaking systems
35
Sense-making
Sense-making is
the process of creating an understanding
of a problem or task
so that further actions may be taken in an
informed manner (Stefik et al., 1999).
36
Sense-making
• Sense-making is a pre-requisite for many
other tasks such as decision making and
problem solving;
• Sense-making involves making clear the
interrelated concepts and their relationships in
a problem or task space.
37
Sense-making examples
38
Sample Sense-making Scenario 1
Task T1: al-Bashir (Abridged Version)
The US wants to take action to towards a resolution of the Darfur conflict .
Al-Bashir, the Sudanese president, is one of the key players in the area who
is believed to have significant responsibility for continuous conflicts in the
region. The administration needs to know as much as possible about alBashir in order to better negotiate with the involved parties and strategize its
efforts. Your task is to produce a report that identifies information to assess
the influence of al-Bashir and makes recommendations for policy decisions
and diplomatic actions.
Requested information includes:
• key figures, organizations, and countries who have been associated
with al-Bashir;
• his rise to power; and
• groups who have resisted him and the level of success in their
resistance.
39
40
Sample Sense-making Scenario 2
Task T2: Energy Security (Abridged Version)
At present, U.S. energy security depends on a range of
countries across the globe, many of which could be
characterized as politically unstable and afflicted with
war, piracy and terrorism. Your task is to produce a
report of the geopolitics of oil in the major suppliers of
U.S., including Mexico, Saudi Arabia, Venezuela, Nigeria,
Algeria, etc. Requested information includes:
• the political, economic, and military status of major
oil suppliers;
• threats to U.S. oil supplies;
• transit chokepoints of world oil.
41
Sample Think-aloud Protocol with Coding
Protocol (Energy Security Task)
Loops
Okay that was actually a very useful
Loop 4
search. So let’s still take this query and
look at Algeria, ‘cause obviously
Algeria and Nigeria are very close…
Processes
Conceptual
Changes
Focused
Search for
data
I understand some of the keywords in
the article but I don’t understand what
the article…
Cognitive
Mechanisms
Comparison
Sensemaking
Failure
Key item
extraction
Okay this has to do with Algeria,
southern Algeria. The minister of
energy… OPEC meeting… so I am
going to see what their connections
are with OPEC.
Building
structures
Key item
extraction
… with all the violence in Nigeria, I
was expecting to find the same types
of political outrage in Algeria…
Instantiating
structures
Comparison
and analogy
and I’m not seeing any notice of that at
all.
Updating of
knowledge
Restructuring
42
Example Concepts and Relationships
Concepts/Entity
Relationships
in new information
Relationship
in existing knowledge
Nigeria (entity)
Political violence
(concept)
Nigeria <hasSituation>
Political violence
Algeria
Algeria <is very close to>
Nigeria
Algeria <hasSituation>
Political violence ?
Political stability
Algeria <hasSituation>
Political stability
Nigeria <hasSituation>
Political violence
The minister of energy
OPEC meeting
…
Nigeria
<hasRelationshipTo>
OPEC ?
43
Sample Sense-making Scenario 3
Task
Write a newspaper article on the role of energy policy in the
presidential campaign.
44
Candidates positions on energy. Take 1
45
Candidates positions on energy. Take 2
46
Theoretical Framework
Sense-making models
– Generic sense-making models
– Sense-making models of
intelligence analysis
– Conducting research as
sense-making
– Organizational sense-making
– Individual vs. collaborative
sensemaking
Cognition
– Types of conceptual changes
– Cognitive mechanisms
– Cognitive structures of
knowledge
Learning
–
–
–
–
Schema theory
Assimilation theory
Generative Learning Theory
Structural knowledge
acquisition
47
Sense-making Elements
Activities
Processes
Sensing
Identification of gaps
Search
Making
sense
Exploratory search
Focused search
For data
For structure
Building structure
Instantiating structure
Consuming
instantiated structure
Mechanisms
Inductive, data-driven
• Key item extraction
• Comparison
• Schema induction
• Generalization
Outcomes
Accretion
Tuning
Restructuring
Deductive, structuredriven
• Definition
• Specification
• Elimination
• Explanation
• Inference
Other
• Metaphor
• Classification
• Semantic fit
• Socratic dialogues
48
An Iterative Sense-making Model
The iterations proceed from exploratory to focused search and sense-making.
Task /
Problem
Structures and
their
instantiations
with data
Decision /
Solution /
Task
completion
Identification of Gaps
Existing
Knowledge
Data loop
Search:
exploratory /
focused
Data
gap
Structure
gap
Searching
for data
Searching
for structure
Instantiating
structure
Building
structure
Outcomes
Updated knowledge
Accretion:
Instantiated structure
Tuning:
Adapted structure
Re-structuring:
New structure
Structure loop
49
Findings – Search and Sense-making Loops
• The overall search and sense-making loops followed
four stages:
–
–
–
–
Task analysis
Exploratory stage
Focused stage
Updates of knowledge representation.
• Reasons for starting a new loop of search and sensemaking:
–
–
–
–
Success of previous sense-making
Failure of previous sense-making
New lead
Failure of search
50
Conclusions on the design
of sense-making systems
51
Integration of tools
• Search tool
• Annotation / indexing tool
• Structure-building and visualization tools,
manual and computer-assisted
• Writing tool
52
Functions in the
integrated environment
53
Functions in the integrated environment
• Build structures: concept maps, templates, outlines
• Select in search results (within one document, a single
document, or multiple documents), drag and drop on
existing structure node or link (accretion), or drop in
empty space to create a new node (structure
modification)
• Have system find the node or link where a text passage
(or image) should be attached
• Assist in extracting assertions – information extraction
• Find other sources for an existing assertion (automated
accretion)
• Always preserve the source (as in MS OneNote)
54
Functions in the integrated environment
• Start a search from a structure node, using as query the
node label or a query learned automatically from the
documents already at the node
• Start a search from any selected text passage, for
example, a text passage in the draft report
• KOS-supported search
(KOS = Knowledge Organization System)
– Query expansion
– Browse KOS structure to clarify search topic, find search terms
55
Functions in the integrated environment
• Switch between structure formats, for example, from
concept map to outline or template
• Assist in creating structure, for example
– Insert relationships extracted from text into a concept map or
construct a concept map from scratch from extracted
relationships
– Clustering
– Find existing structures – for example, search for structures on
the Web
56
Functions in the integrated environment
• Create a draft report from a concept map
– Create outline and insert texts associated with each node
– Express relationships through text (text generation)
57
Functions in the integrated environment
• Screen real estate is important – a very large screen or
two monitors
• Easy input mode (so as not to distract from thinking)
– Voice input
– OCR pen input from print material
58
History-aware
personal or group
information store
Anita Komlodi
59
The total information store
•
•
•
•
•
Store everything
Connect everything
Search and find everything
Keep detailed provenance & use history
History is both past and future ─
what was done and what is to be done
60
Store everything
•
•
•
•
•
Documents of all kinds
Tasks
Actions, events
People and organizations
Concepts, issues, frameworks (ontology)
61
Search and find everything
• Flexible search − any item as starting point, any
connection type.
For example
– Find all items connected to a task
– Find all events and actions in a given time span
– Find all items connected, directly or indirectly, to a
person
• Present search results in different views, organize
into a meaningful structure
62
Tools
•
•
•
•
•
•
•
Search tool
Gather tool
Compare tool
Connection tool, including annotation function
Structure creation and display tool
Scratch pad
Ontology generation tool
– Personal ontology
– Group ontology
– Task ontology
• Writing / action tool
63
Relevance relationships
Xiaoli Huang
64
Information arranged
by role in argument
65
Relationship of info to task
Matching / allowing inference on a topic
. Matching topic / direct relevance
. Allowing inference on the topic / indirect relevance
Context
Comparison
Cause and effect
Goal
Method / Solution
Evaluation
Use these relationships when linking information to tasks
66
Relevance relationships
Matching / allowing inference on a topic
. Matching topic / direct relevance
. . Matching topic at same level of detail
. . . Direct statement
. . . Reference
. . . Definition
. . . Restatement
. . . . paraphrase,
. . . . clarification
. . Summarization
. . . Abstraction
. . Elaboration
. . Interpretation
67
Relevance relationships
Context
.
.
.
.
.
.
.
.
.
.
.
Background
Scope, broader
Framework
Assumption or expectation
Preparation
Environmental setting
. Physical environment
. Social, political, cultural background
By time sequence
Condition
. Enabling or hindering condition
68
Relevance relationships
Comparison
.
.
.
.
.
.
.
.
.
.
.
.
.
By similar vs. different
. Comparison, similar
. Comparison, different
By factor that is different
. Difference in external factor
. . Different time
. . Different place
. . Different type of situation
. Difference in participant
. Difference in act or experience
. . Difference in act
. . . Different purpose
. . . Different method
69
Some conclusions
• The system needs to think with the user
– Adapt to the user’s way of thinking
but also
– Help the user to structure a problem, organize
information, and prepare a plan for a task
• Dividing users’ work into applications is a distraction
• The gradual evolution of systems needs to make way for a
task-centric revolution
• This presents challenges
– Task-oriented organization and intelligent processing of
information
– Technical implementation
70
Questions
dsoergel @ umd.edu
www.dsoergel.com
71
Leftover slides
72
Pilot Test
• Participants
• Data collection
• Data analysis
• Findings
73
Participants
• 6 information science students
• Tasked with synthesizing data from a
variety of sources, assessing the credibility
of information, and evaluating claims
based on supporting evidence
• Trained in using Rosetta - a multilingual,
multimedia news retrieval system
74
Data Collection
• 2-hour task sessions, 6 tasks
total of 17 sessions completed
• Participants were instructed to verbalize
their thoughts as they work on the tasks
Think-aloud protocols were transcribed
• Post-session interviews as supplemental
data
75
76
Data Analysis – Coding Scheme
A Processes (from the model)
Search
Exploratory search
Sense-making
Gap identification
Exploratory search for data
Data gap
Exploratory search for structure
Structural gap
Focused search
Building structures
Focused search for data
Using automatically extracted results
Focused search for structure
Extracting relationships manually
Instantiating structures
Updating knowledge
B Conceptual Changes (from the model)
Sense-making success
Accretion
Sense-making failure
Unable to fit data into structure
Tuning
Re-structuring
Unable to build new structure
77
Data Analysis – Coding Scheme
C Cognitive Mechanisms (from the model)
Inductive mechanisms
Deductive mechanisms
Other
Key item extraction
Definition
Analogy and metaphor
Comparison
Specification
Classification
Similarity
Explanation-based
Socratic dialogues
Differentiation
Elimination
Semantic fit
Schema induction
Inference
Generalization
D Emerging Codes Added During Analysis
Reasons starting a new loop
Resolution of conflicts
New lead
Disregard conflicting evidence
Sense-making success
Compromise
Sense-making failure
Accept new evidence
Search failure
Confusion
78
Findings – Cognitive Mechanisms
• Our participants used a two-way approach:
data-driven (bottom-up)
80%
structure-driven (top-down)
20%
• Key item extraction and comparison were
used most often.
79
Findings – Dealing with Conflicts
• Disregard – “… I wanted that article to say
something else. I have to disregard it.”
• Compromise – “…it is not my understanding that
this has anything to do with oil. But okay, this is a
different take.”
• Acceptance – “…I thought these countries had
similar serious problems. But [in fact, they] seem to
be relatively stable and put a lot of effort into
establishing their economy.”
• Confusion – “…Obviously I have no idea what this
is about...”
80
Findings – Role of Instantiated Structures
• Entities (represented as names) and
key concepts (represented as keywords)
were often the basis for relevance judgments.
• The relationships embedded in new information
and between the new information and participants’
previous knowledge seemed to play an important
role in structure building and data fitting.
• Both concepts and relationships seemed to be
crucial for updating knowledge.
81
Pengyi Zhang
[email protected]
terpconnect.umd.edu/~pengyi/sensemaking/
Extending Sense-Making Models with Ideas from Cognition and
Learning Theories
Pengyi Zhang, Dagobert Soergel, Judith Klavans, and Douglas Oard
[email protected],
ASIST 2008, Sunday Oct. 26, Session on Sense Making. Online
Proceedings, Paper 29, p. 1 – 23 [34 – 56]
http://terpconnect.umd.edu/~pengyi/sensemaking/files/zhang_asist08_sen
semaking_model.pdf
82
Conclusions
The model is a useful framework
for understanding the users’ sensemaking process
• The empirical think-aloud data seem to be consistent
with the iterative sense-making model
• Users used a combination of data-driven and
structure-driven mechanisms
• Instantiated structure elements play an important
role in sense-making
83
Implications and Future Work
• Theoretical
– Better understanding of sense-making processes
by extending the existing models with ideas from
cognition and learning
– Further testing and refinement.
• Empirical
– Better system design based on findings from user
studies.
– Making design suggestions to support sensemaking, not just search.
84
A Sample Sense-making Scenario
Task
An intelligence analyst is tasked to gather, analyze, and synthesize
information related to Al-Bashir and, on that basis,
make recommendations for action in the form of a report.
Background
The Darfur conflict is a crisis in the Darfur region of western Sudan.
Al-Bashir, the Sudanese president, is one of the key players in the
area who is believed to have significant responsibility for
continuous conflicts in the region. As part of an effort to resolve
these armed conflicts, the administration needs to know as much
as possible about al-Bashir in order to better negotiate with the
involved parties and strategize its efforts.
85
A Sample Task Scenario
Task T1: al-Bashir (Abridged Version)
Omar Hasan Ahmad al-Bashir is a Sudanese military
leader, dictator, and current president of Sudan. Your
task is to produce a report identifying information to
assess the influence of al-Bashir as a basis for policy
decisions and diplomatic actions. Requested information
includes:
• key figures, organizations, and countries who have
been associated with al-Bashir;
• his rise to power; and
• groups who have resisted him and the level of
success in their resistance.
86
87
88
89