Crowdsourcing Evidence, Argumentation,
Thinking and Evaluation
(CREATE)
Ruthanna Gordon, Ph.D.
Technical SETA
Booz Allen Hamilton
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
1
CREATE Overview
• CREATE will develop and test methods to improve analytic
arguments, by enabling dispersed groups of individuals to identify
and evaluate
• Reasons
• Evidence
• Assumptions
• Alternative hypotheses.
• CREATE will develop:
• Structured methods to elicit and aggregate the elements of an
argument.
• Approaches to crowdsource these methods, so that many
individuals can collectively develop and refine an argument.
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
2
Intelligence analysis involves making and evaluating
arguments.
Example
Opposing “Despite real improvements, the Iraqi Security Forces (ISF)—
Reason
particularly the Iraqi police—will be hard pressed in the next 12-18
Main Claim months to execute significantly increased security responsibilities,
and particularly to operate independently against Shia militias with
success. Sectarian divisions erode the dependability of many units,
Supporting many are hampered by personnel and equipment shortfalls, and a
number of Iraqi units have refused to serve outside of the areas
Reasons
where they were recruited.”
- “Prospects for Iraq’s Stability: A Challenging Road Ahead,”
National Intelligence Estimate, January 2007
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
3
• Analytic arguments have been produced in much
the same way for over 60 years.
• Written as narratives; generally no formal
representation
• Emphasis on consensus, disagreements usually
resolved privately
• Prose can mask argument complexity
WMD Commission: “Perhaps most troubling, we found an
Intelligence Community in which analysts have a difficult
time stating their assumptions up front, explicitly explaining
their logic, and, in the end, identifying unambiguously for
policymakers what they do not know.”
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
4
Improving Analysis
• Following the WMD Commission, many IC initiatives have worked to
improve analytic reasoning:
– At IARPA:
• Sirius training games increase skills for mitigating cognitive biases
• ACE and ICPM leverage crowdsourcing to improve anticipation of world events and
consideration of alternative futures
– Across the IC:
• Critical thinking modules added to analytic training courses
• Clearer standards for reporting confidence in evidence
• Structured techniques and other tools for organizing thorough analysis
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
5
Structured Analytic Techniques
•
SATs clearly represent argument
structure, making it easier to:
–
–
–
•
identify unstated assumptions
introduce objections and rebuttals
see how much support each claim has.
However, such representations are
seldom used because they require:
–
Extra time that may conflict with analytic
deadlines
–
Extra skills or knowledge that may not be
available to an individual analyst
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
6
CREATE Approach
•
New and improved SATs can address usability
issues directly, and better address key flaws in
reasoning.
•
Crowdsourcing can make SATs much easier to
use. Each analyst can contribute just where s/he is
most capable.
•
Structured crowdsourcing can help analysts help
each other, providing a framework for critique and
comparison of alternative hypotheses.
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
7
Research Teams
University of Melbourne
- Monash U
- NTVI
- Yanna Rider Consulting
Monash University
- Strathclyde U
- Birbeck U
- University College
London
Syracuse University
- Colorado State U
- SRC, Inc.
- University of Arizona
George Mason University
- California State U –
San Bernardino
- U of Mary Washington
- U of Nebraska – Lincoln
- U of South Carolina
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
8
Research Teams
• Interdisciplinary research teams include expertise in:
–
–
–
–
–
Behavioral and social sciences
Informal logic and critical thinking
Mathematics and statistics
Computer science
Software rapid prototype development
• Range of Approaches
– Argument mapping with probabilistic estimates for value of evidence
– Bayesian reasoning network with online Delphi-style crowdsourced
ideation
– Narrative reframing and structured argumentation
– Wiki-inspired organization of arguments, directed by profiles of
reasoning strengths
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
9
Evaluating Complex Analytic Reasoning
• Appropriate test problems
–
–
–
–
Address aspects of reasoning relevant to real-world analysis
Clear rubrics for evaluation, including objective and human-rated
May vary in complexity, contextual detail, and level of abstraction
Avoid being trivial for ST-using crowds or impossible for unaided
individuals
– For Phase 1, all information required for response must be
provided in the problem
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
10
Evaluating Complex Analytic Reasoning
• Appropriate evaluation techniques
–
–
–
–
Valid, reliable, and replicable
Go beyond accuracy of conclusions
Can handle freeform responses (not solely multiple choice)
Address multiple aspects of complex reasoning (bias, evidence
weights, gap identification, etc.)
– Provide fast, meaningful feedback on at least some aspects of
responses
– Can fairly handle output from diverse systems, potentially in
diverse formats
– Can respond to unanticipated good (and bad) arguments
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
11
Key Aspects of Reasoning
• Assess evidence for plausible hypotheses, including those that are
ultimately rejected.
• Estimate probability of hypotheses.
• Justify key judgments by explicitly referencing the reasons,
assumptions and evidence on which they are based.
• Indicate quality, credibility and diagnosticity of key information
sources.
• Describe potential objections to reasons, hypotheses and
assumptions.
• Express and explain potential implications of key unknowns.
• Identify new information to seek out, that might increase certainty
of key judgments.
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
12
Contact Information
 Dr. Steven Rieber – Program Manager
301-851-7521
[email protected]
 Dr. Ruthanna Gordon – Technical SETA
(contractor)
301-851-7762
[email protected]
INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
14