International Technology Alliance
in
Network & Information Sciences
Human-Information
Interaction
TA6 – Project 4
David Mott
IBM UK
Peer Review
September 2014
Project Overview
Project 4 aims to advance our
understanding of the factors that affect
shared understanding and information
exploitation in military coalition
environments
COALITION NEEDS
 to be assisted in collaborative problemsolving by systems that share common
understanding of the knowledge and
reasoning with human users
 to engineer the coalition socio-technical
environment in ways that support
collective/team cognition
 to exploit large bodies of diverse,
unstructured information content and
other information assets in the execution
of their tasks
PROJECT FOCUS
 understanding the interaction between
cognitive, social and technological factors
in collaborative problem-solving contexts,
by cognitively-rich agent simulation
 supporting fact extraction from plain text
by using deep linguistic processing and
facilitating human-machine cognitive
capabilities by externalising reasoning in a
Controlled Natural Language
 understanding how to support human
decision-makers by matching information
assets to requirements and by use of
meaning-rich conversational interactions
ITA Peer Review, Sept. 2014
2
Multidisciplinary Research Team
 IBM UK
 David Braines
 David Mott
 Paul Stone
 Airbus Group
 Gavin Powell
 Matthew Roberts
 Boeing
 Anne Kao
 Stephen Poteet
 Ping Xue
 IBM US
 Geeth de Mel
 Dstl
 Kris Challa
 ARL
 Cheryl Giammanco
 Jon Bakdash
 Carnegie Mellon University
 Katia Sycara
 Yuqing Tang
 PSU
 Tom La Porta
 Nan Hu
 University of Cambridge
 Ann Copestake
 Diarmuid O Seaghdha
 University of Cardiff
 Alun Preece
 Will Webberley
 University of Southampton
 Paul Smart
 Darren Richardson
 UCLA
 Mani Srivastava
 Robin Wentau Ouyang
 Matthew Johnson
Key Technical Accomplishments
Task 1: Collective Cognition in Coalition Environments
 a cognitive social simulation capability to support experimental studies
of team cognition in complex task environments.
 an initial cognitive computational model of collaborative problem
solving in a specific task context (i.e., the ELICIT task).
 a human experiment using the ELICIT task, designed to gather data
relating to the effect of different information sharing strategies on task
performance.
 integration of Controlled English (CE) components into the simulation
capability to support linguistic forms of agent communication.
ITA Peer Review, Sept. 2014
4
Key Technical Accomplishments
Task 2: Fact Extraction and Reasoning using Controlled
Natural language
 extraction of CE facts from natural language (NL) sentences, applying
knowledge expressed in CE to transform linguistic semantics into domain
semantics.
 extension of CE for more complex problem solving strategies with metareasoning, assumptions, rationale, as applied to realistic analytic tasks
 handling of NL ambiguities and uncertainties, extending linguistic
mechanisms to allow the CE domain model to guide NL processing
 exploration of collaborative man-machine problem-solving systems as
applied to “logic puzzles” used in training intel analysts.
ITA Peer Review, Sept. 2014
5
Key Technical Accomplishments
Task 3: Coalition Context-Aware Assistance for Decision
Makers
 the definition and validation, including initial trials with human
subjects, of a conversational protocol for free-flowing human-machine,
machine-machine, and machine-human exchanges
 a model of resource allocation in network systems as a "stochastic
knapsack problem" to handle uncertain factors like unreliable wireless
medium or variable quality of sensor outputs
 a technology integration experiment to show how key research
elements can be combined to support rapid but informed decisionmaking capabilities at lower echelons in coalition operations
ITA Peer Review, Sept. 2014
6
ACT-R Cognitive Social Simulation Capability
(CSSC)
State-of-the-art
 Multi-agent simulation techniques have been
used to model collective behavior.
 ….however, such models do not take account
of the cognitive capabilities and limitations of
human agents.
 Cognitive architectures provide computational
frameworks for the development of
cognitively-plausible agents.
 …however, few applications to team contexts.
Coalition needs/benefits
 We need to know how to exploit the collective
cognitive potential of coalition organizations.
 We need to assess the effect of human factors
and socio-technical interventions without
disrupting the operational environment.
Paul Smart (Southampton)
CMU, IBM UK, Airbus Group
Cognitive architectures can be used
to model socially- distributed cognitive
processes and improve our understanding of the
dynamics of team cognition.
 ACT-R:
 a mature cognitive architecture that can be used
to model a broad range of cognitive processes.
 extend existing single-agent ACT-R research by
focusing on interactions of multiple agents,
allowing study of socially-distributed cognition
 extend ACT-R architecture to support multipleagent research
 Cognitive social simulation:
 integration of cognitive architectures into social
simulation results in cognitive social simulation.
 individual machine agents share the same kind of
cognitive capabilities and limitations as their
human counterparts.
 Experimentation:
 based on collaborative simulation of the DoD
CCRP ELICIT identification task
 simulation capability instrumented to collect and
measure performance factors based upon existing
ELICIT metrics
7
ACT-R Cognitive Social Simulation Capability
(CSSC)
ACT-R
Goal Module
Declarative
Module
Vocal Module
Procedural Module
Matching
ACT-R/CSSC
Imaginal
Module
Selection
Self Module
Messaging
Module
Agent
Characteristics
Messaging
The ACT-R/CSSC is an extension to the core ACT-R
architecture to support cognitive social simulation experiments.
Execution
Language
Module
Web Module
Website
Language
Processor
8
ACT-R Cognitive Social Simulation Capability
(CSSC)
9
ACT-R Cognitive Social Simulation Capability
 Simulation capability supports studies into team-based
problem solving.
 Current experimental efforts focused on the DoD CCRP
ELICIT task.
 Modelling the performance of human subjects in
different organizational environments.
Coordinator
Team Leader
Team Member
Shared Repository
(limited access)
Extended ACT-R cognitive
architecture to support studies into
team cognition. Custom modules
support linguistic communication
(language module), interaction with
shared repositories (web module),
and configuration of agent behaviour
(self module). The simulation
capability also supports the real-time
participation of human subjects.
Long-Term Goals
Understand effect of task environment
features on collective cognitive
performance:
 perform human experiments, and gather
data, with a variant of the ELICIT task.
 extend cognitive computational models with
machine agents embodied in the
environment
 evaluate cognitive models with experimental
simulation studies.
The ACT-R Cognitive Social Simulation capability provides a generic platform for exploring
ITA Peer Review, Sept. 2012
10
the interaction between cognitive, social and technological factors in collective cognition.
CE to support fact extraction and problem solving
State-of-the-art
 Traditional NL mechanisms have difficulty
extracting detailed information about complex
situations
 Deep parsing systems, can extract more subtle
and detailed information from text, but it is a
complex task to convert the linguistic output into
domain-specific facts
 Performance of complex cognitive tasks requires
integration of different modeling and reasoning
capabilities in a common language
Coalition needs/benefits
 Military tasks require text from NL sources to
analyse, fuse and infer high value information
 A deep parsing system, with linguistic to domain
semantic transformations, could extract more
domain specific facts to support these tasks
 Collaborating human agents need support in
problem-solving tasks in a way that is
understandable and externalises their domain
models and problem-solving strategies.
David Mott (IBM UK)
Boeing, Cambridge, ARL
CE can integrate modelling and
reasoning capabilities to assist performance of
complex cognitive tasks, including transformation
of linguistic information from a deep parsing
system into high value domain facts
 Use of DELPH-IN linguistic resources
 The ERG, a deep and detailed grammar of English
 Minimal Recursion Semantics (MRS) to represent the
output of the parse in a logical form
 Transforming linguistic semantics to domain semantics
 Rules expressed in CE for application of CE domain
models
 Exploration of linguistic theories, in a way that is more
natural than typical formal languages
 CE reasoning and modelling system
 User-defined domain model for expressing concepts,
facts, logical rules and problem-solving strategies
 Assumptions to capture uncertainty, ambiguity and
sentence interpretation
 Rationale and proof tables to visualise reasoning and
sources of uncertainty
 Meta-model for reasoning about the language itself, e.g.
mapping words to concepts and to create new rules
from sentences
11
ITA Peer Review, Sept. 2014
Linguistic reasoning to extract CE facts
Reasoning to support problem solving
CE to support fact extraction and problem solving
CE domain models,
reasoning and problem
solving strategies
CE linguistic reasoning can apply domain
knowledge to fact extraction, bridging the gap between
deep linguistic analysis and domain semantics; CE
domain reasoning can externalise human problem
solving to solve an analytic task in an explainable way.
Long-Term Goals
Improve linguistic processing
and fact extraction
 linguistic reasoning to handle a wider range
of sentences, e.g. following the MRS test
suite
 construction of domain models from
distributional semantics
 more domain knowledge to resolve
ambiguities of sentence interpretation
A machine agent that can participate in collective
cognition and externalises human reasoning
Improve problem solving capability




Extend CE expressivity
More sophisticated handling of assumptions
Improve visualisation of rationale
Handle more complex logical problem solving
CE problem solving techniques can be applied to both fact extraction from NL sentences and
ITA Peer Review, Sept. 2012
14
reasoning from basic extracted facts to intel relevant high value information
Human-Machine Conversations
State-of-the-art
 Reducing cycle time in rapid
establishment of data-to-decision
pipelines (D2D) still requires significant
technical skills and training
 Handling of uncertainties in exploiting
soft and hard information is a key
problem, even for technical users
Coalition needs/benefits
 Low training overhead and flexible
human-machine collaboration (HMC) to
create D2D pipelines
 Exchange of information precisely and
unambiguously whilst retaining human
readability and flexibility, pinpointing
sources of uncertainty and providing
explanation
Alun Preece (Cardiff)
IBM UK, IBM US, PSU, UCLA, ARL
Human-machine conversations using
natural language (NL) and Controlled English
(CE) can facilitate information exploitation and
asset tasking at the tactical edge.
 A formal conversational protocol – captured as a CE
model – enables HMC in the scope of a range of
tactical-level use cases:
 “spot” reports (including uncertain QoI)
 information fusion (including rationale)
 asset tasking (including uncertain availability)
 Combining the expressivity of NL and the precision
and reasoning capabilities of CE
ITA Peer Review, Sept. 2014
15
Human-Machine Conversations
NL/CE Conversational Protocol
 Aim: to enable conversational interactions that flow freely between natural language and CE,
specifically to support dynamic establishment of D2D pipelines
 Conversational protocol draws on research in agent communication languages and
philosophical linguistics (speech acts)
Types of interaction include:
 Confirmatory dialogues to reduce
ambiguity (“did you mean…?”)
 Query-response, with optional rationale
(“Why?”)
 Machine-generated (text/graphical)
“gist” forms to summarise complex CE
ITA Peer Review, Sept. 2014
Human-Machine Conversations
Experimentation: scenario-based
Experimentation: human subjects
Main aims:
• to determine the degree to which a software
agent can extract CE from unrestricted NL
conversations
• to test the agent’s robustness with untrained
users
 D2D vignette from earlier ITA work
 Technology integration experiment
including: conversational agent (steps
1+4), CE Store (2), SAM (3)
 Demo with CERDEC AI TECD Jan 2014
 Takes account of uncertainty of asset
availability (e.g. bandwidth) and QoI (e.g.
due to weather)
ITA Peer Review, Sept. 2014
 20 subjects viewed a series of scenes and
described them in natural language via a
text-based interface
 The system provided feedback in CE and a
score (terms recognised by system)
Human-Machine Conversations
 Human subject experiment shows rapidlydeveloped conversational agent able to
extract exploitable soft information
“There is two
policemen are
riding on a horse.
The horses color
are white and
brown! They are
riding in the same
direction.”
there is a group named '#35' that
has 'policemen' as description and
has the entity concept 'person' as member ….
there is a group named '#38' that
has 'The horses' as description and
has the entity concept 'horse' as member and
has the colour 'brown' as18colour and
has the colour 'white' as colour.
a conversational approach
founded on use of natural and controlled
natural language (CE) allows users at the
tactical edge to exploit soft and hard
information – including coping with
uncertainties and QoI – using mobile
computing approaches with low training
overhead.
Longer-Term Goals
 support a wider range of conversational
actions (queries, commands, narratives, model
and lexical updates, interjection, etc)
 integrate information from more sources, e.g.
sensor input from the mobile device, social
media
 improve handling of uncertainties, and show
utility of this via Experimentation Framework
The D2D pipeline can be viewed as a collection of conversational interactions between
Peer Review,analytic
Sept. 2012 services and decision-makers
human & machine agents: dataITA
sources,
18
Academic & Military Impact
Technical Leadership
 to CERDEC in the use of CE for ontology development
 to linguistic researchers, including DELPH-IN and ARL, in the use of CNL for domain modelling and
reasoning, and in the consideration of rationale as a new topic in academic research
 to SMEs (Prof Don Shemanski and David Alberts) in the use of CE as a means of building a
collaborative man machine reasoning system to solve analytic puzzles
 Alun Preece et al received 2013 ACM MiSeNet MobiCom Workshop Best Paper Award
 Dave Braines gave an invited plenary speech at NS-CTA on CE for hybrid reasoning
 Alun Preece, Tien Pham and Dave Braines are on the TPC for NATO IST/SET 126 Symposium
Transitions
 Dstl projects:
 use of a CNL for intelligence analysis (MIPS)
 CE for intelligence, cyber assets, (+P5) policy based service composition, (+P5) missions and assets,
 staff rotation at IBM Hursley to explore a CE agent for analysing Chinese
 ARL projects:
 application of CE and experimental framework for NS-CTA, TerraHarvest and ARL anomaly teams
 demonstration of the conversational capability to CERDEC
 use of CE to orchestrate high profile NS-CTA experiment, briefed to the NS-CTA board
 staff rotation of Westpoint cadets at IBM Hursley to build interactive CE database of intelligence profiles
 Potential:
 CE as a business language in IBM Information Visualisation product
 training in CE for modelling, for future transition with ARL analysts in the Agri-Development Teams
 assistance for training and simulation of emergency situations by linking ACT-R and Unity3D modelling19
 conversational interface in advisory capacity to a UK police force at 2014 NATO Summit
Closing Remarks
KEY COLLABORATIONS ENABLED
Reaching In
 Southampton and CMU have extended the ACT-R
architecture to support cognitive social simulation
experiments.
 IBM UK and Southampton have integrated CE
components into the ACT-R-based simulation
capability.
 Cambridge and IBM UK are transforming ERG
linguistic semantics into CE domain facts
 Boeing and IBM UK have developed representations
of NL uncertainties as CE assumptions
 Cardiff and IBM UK have developed conversational
interfaces, with sensor mission-matching algorithms
Reaching Out
 Cardiff and PSU have integrated measures of
uncertainty into allocation algorithms
 UCLA, IBM UK and Cardiff have extended
conversational interfaces to crowdsourcing (+P6)
 Cardiff, IBM UK and NS-CTA have integrated CE
models and reasoning into demonstrations (+NS-CTA)
 Aberdeen and IBM UK have linked argumentation
and CE assumptions for hypothetical reasoning (+P6)
 IBM UK, Cardiff and RPI have applied CE to model
policy based service composition (+P5)
 IBM UK and ARL have explored with Fraunhofer
(FKIE) on CE and the Battle Management Language
KEY PUBLICATIONS
 Smart, P. R., Sycara, K., & Tang, Y. (2014) Using
Cognitive Architectures to Study Issues in Team
Cognition in a Complex Task Environment.
 Smart, P. R., Richardson, D. P., Sycara, K., & Tang,
Y. (2014) Towards a Cognitively Realistic
Computational Model of Team Problem Solving
Using ACT-R Agents and the ELICIT
Experimentation Framework.
 Mott, D., Poteet, S., Xue, P, Kao, A, Copestake, A.
(2014), Natural Language Fact Extraction and
Domain Reasoning using Controlled English
 Xue, P., Poteet, S., Kao, A., Mott, D., & Giammanco,
C. (2014) Representing Uncertainty in CE
 Preece, A., Braines, D., Pizzocaro, D., & Parizas, C.
(2014) Human-Machine Conversations to Support
Multi-Agency Missions
 Preece, A., Gwilliams, C., Parizas, C., Pizzocaro, D.,
Bakdash, J., & Braines, D. (2014) Conversational
Sensing
 3 long papers, 9 short papers, 6 demonstrations, 1
workshop for the 2014 Fall Meeting
Metrics
Publications
PhD Students
Journal: 3
Conference: 34
ITA Peer Review, Sept. 2014
On board:4
Graduated: 0
+ 1 post doc + 4 RAs
5/12 to 5/14
8/12 to20
8/14