Probabilistic Programming for Advancing
Machine Learning (PPAML)
Program Kick-Off
Providence, RI
November 13-15, 2013
Dr. Kathleen Fisher
I2O Program Manager
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Agenda, Wednesday Nov. 13
0800-0900
Check-In Desk
North Rosemoor Ballroom
0800-0900
Breakfast
South Rosemoor Ballroom
Session 1: Introductions
North Rosemoor Ballroom
0900-0930
Welcome and Introduction to PPAML
Kathleen Fisher
0930-1030
Overview of each Technical Area performer
Kathleen Fisher
1030-1100
Break
1100-1200
Speed Dating
1200-1300
Lunch
1300-1330
Break (check e-mail)
South Rosemoor Ballroom
Session 2: Domain Experts (TA1)
North Rosemoor Ballroom
1330-1430
TA1: Challenge Problems, Summer Schools, & Evaluation
Galois
1430-1500
Break
Session 3: Probabilistic Programming (TA2) & Inference Engine (TA4) North Rosemoor Ballroom
1500-1530
MIT TA2/TA4 Presentation
MIT
1530-1600
BAE TA2/TA4 Presentation
BAE Systems
1600-1630
CRA TA2/TA4 Presentation
Charles River Analytics
1630-1700
Break
1630-1700
Galois One-on-One Session
LaSalle Room
Rosemoor Ballroom Foyer
Session 4: Posters
1700-1800
Poster Session & Reception
1800
Adjourn
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Agenda, Thursday Nov. 14
0800-0900
Check-In Desk
0800-0900
Breakfast
Session 5: Probabilistic Programming (TA2) & Inference Engine (TA4) (cont.)
0900-0930
Indiana TA2/TA4 Presentation
0930-1000
Gamelan TA2/TA4 Presentation
1000-1030
Stanford TA4 Presentation
1030-1100
Break
Session 6: Machine Learning (TA3)
1100-1130
Princeton TA3 Presentation
1130-1200
UCR TA3 Presentation
1200-1300
Lunch
1300-1330
Break (check e-mail)
Session 6 (cont.): Machine Learning (TA3)
1330-1400
ACS TA3 Presentation
1400-1430
SRI TA3 Presentation
1430-1500
Break
1500-1530
1530-1600
Breakout Session Topics:
- Challenge Problems (North Rosemoor Ballroom) - Group A
- Summer Schools (Promenade Room) - Group C
- Measuring Performance (Hartwell Room) - Group B
ACS One-on-One Session (concurrent)
SRI One-on-One Session (concurrent)
Session 8: Breakouts 2
1600-1700
1600-1630
1630-1700
1700
Indiana University
Gamelan
Stanford University
North Rosemoor Ballroom
Princeton University
University of California, Riverside
South Rosemoor Ballroom
North Rosemoor Ballroom
Applied Communication Sciences
SRI International
North Rosemoor Ballroom, Promenade Room,
Hartwell Room
Session 7: Breakouts 1
1500-1600
North Rosemoor Ballroom
South Rosemoor Ballroom
North Rosemoor Ballroom
Breakout Session Topics:
- Challenge Problems (North Rosemoor Ballroom) - Group C
- Summer Schools (Promenade Room) - Group B
- Measuring Performance (Hartwell Room) - Group A
UCR One-on-One Session (concurrent)
Princeton One-on-One Session (concurrent)
Adjourn
LaSalle Room
LaSalle Room
North Rosemoor Ballroom, Promenade Room,
Hartwell Room
LaSalle Room
LaSalle Room
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Agenda, Friday Nov. 15
0800-0900
Check-In Desk
North Rosemoor Ballroom
0800-0900
Breakfast
South Rosemoor Ballroom
North Rosemoor Ballroom, Promenade
Room, Hartwell Room
Session 9: Breakouts 3
0900-1030
Breakout Session Topics:
- Challenge Problems (North Rosemoor Ballroom) - Group B
- Summer Schools (Promenade Room) - Group A
- Measuring Performance (Hartwell Room) - Group C
0900-0930
MIT One-on-One Session (concurrent)
LaSalle Room
0930-1000
BAE One-on-One Session (concurrent)
LaSalle Room
1000-1030
CRA One-on-One Session (concurrent)
LaSalle Room
1030-1100
Break
North Rosemoor Ballroom, Promenade
Room
Session 10: Breakouts 4
1100-1230
Breakout Session Topics:
- Front-end & integrating TA3 (North Rosemoor Ballroom)
- Back-end & integrating TA3 (Promenade Room)
1100-1130
Indiana One-on-One Session (concurrent)
LaSalle Room
1130-1200
Gamelan One-on-One Session (concurrent)
LaSalle Room
1200-1230
Stanford One-on-One Session (concurrent)
LaSalle Room
Lunch
South Rosemoor Ballroom
Session 11: Breakout Reports
North Rosemoor Ballroom
1330-1500
Breakout Reports, Discussion & Wrap-up
Breakout leaders
1500
Adjourn
1500-1530
Galois One-on-One Session
1230-1330
LaSalle Room
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Logistics
•
WiFi information:
•
•
•
Chess clock: 20 minutes (talk) + 10 minutes (questions)
•
•
Exception: TA1 talk, 45 minutes (talk) + 15 minutes (questions)
Wiki for sharing information
•
•
Network: hhonors
Passcode: DARPA
We’ll upload kick-off presentations; please give Rinku or Jonathan your slides
Meetings
•
•
•
We’ll be scheduling monthly status calls
We’ll be scheduling site visits: First round in Feb. and March of 2014
Two PI meetings per year (summer and winter)
• Summer PI meetings co-located with Summer Schools
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Programmatics
•
Schedule
•
•
•
Collaboration
•
•
•
Led by a TA2 performer
Produce a working end-to-end PPS
Produce working version of Team CP
Overall Program Metrics
•
•
•
•
•
Strong interaction among all performers is critical to program success
Associate Contractor Agreement (ACA)
PPS Design Team
•
•
•
•
Three phases: Phase I (22 months); Phases II & III (12 months each)
Teams will not be competitively evaluated / No anticipated down-selection
Shorter: Reduce LOC by 100x for ML applications
Faster: Reduce development time by 100x
More Informative: Develop models that are 10x more sophisticated
With Less Expertise: Enable 100x more programmers
Milestones
•
•
•
1.5 months before PI meeting: TA1 performer will evaluate the effectiveness of each team’s
PPS on each of the CPs defined at that point in the program
At PI meeting: each team will demonstrate the effectiveness of their PPS on the Team CP
Summer School: TA1 performer will evaluate the effectiveness of each team’s PPS in enabling
participants to build useful ML applications with minimal interventions from PPAML performers
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Probabilistic Programming:
Advancing Machine Learning
Kathleen Fisher, I2O Program Manager
February 25, 2013
Approved for Public Release; Distribution Unlimited
7
Machine Learning is Ubiquitous and Very Useful
ISR
Natural Language Processing
Image Search/Activity Detection: Find and
identify objects and actions in video
Object Tracking: Follow vehicles as they
move through a city and are recorded in
multiple video streams (DARPA CZTS)
Patterns of Life: Process wide area aerial
surveillance data and associated tracks to
infer location types and object dynamics
Bird Migration Patterns: Model spatio-
temporal distribution of birds (by species);
involves large-scale sensor integration
DARPA LAGR: Vision-based robot navigation
Google Glasses: Perform searches based on
images taken by user cell phone cameras
© Netflix
Microsoft Matchbox: Match players based on
© Apple
set of seismic events given detections and
misdetections on a network of stations
.....
Netflix Challenge
Predict user ratings
for films based on
previous ratings
Siri
Voice recognition and
Natural Language
Processing (NLP)
DARPA Grand
Challenge
Fully autonomous
ground vehicles
competition
Nuclear Test Ban Treaty Compliance: Deduce
Predictive Analytics
Watson: Computer system capable of
answering questions posed in natural
language
Topic Models: Statistical model for
discovering the abstract "topics" that occur in
a collection of documents
Distributed Topic Models: Asynchronous
distributed topic discovery
Citation Analysis: Given citations, extract
author, title, and venue strings and identify
co-reference citations
Entity Resolution: Discovering entities that
are mentioned in a large corpus of news
articles
NLP Sequence Tagging: Tagging parts of
speech and recognizing named entities in
newspaper articles
their gaming skill set
Predictive Database: Understand information
based on causal relationships in data
Bing Image Search: Search for images on the
web by selecting text in word document
Amazon Recommendation Engine:
Recommend items based on consumer data
Cyber and Other
ORNL’s Attack Variant
Detector: Discover
compromised systems
Yahoo’s Bayesian Spam Filtering: Self-
adapting system based on word probabilities
Cyber Genome Lineage: Reverse engineer
malware samples to find shared “genetic"
features between different malware samples
Gene Sequencing: Determine order of
nucleotides in a DNA molecule
Disclaim er: I m ages of specific products are used for illustration only. Use of these
im ages does not im ply endorsem ent of inherent technical vulnerabilities.
Approved for Public Release; Distribution Unlimited
8
Explaining the Magic by Example:
Channel State Estimation
Phenomenon:
Channel properties between cell
tower and cell phones, each
with >1 antenna (MIMO)
Channel
Properties of
Communication
Link
Question: Given received
signal, what was sent?
What bit
sequence was
transmitted?
y = Hx + n
Training Data:
Received signals given pilot
(known) transmissions
ProtoModels:
Channel matrix H plus vector n
of Gaussian noise.
Selected Model approximates
noise and channel matrix
Pilot Sequence
Results
KullbackLeibler
Divergence
Approved for Public Release; Distribution Unlimited
Belief Propagation
(BP)
9
Why Is It Hard?
• Brittleness of implementations &
lack of reusable tools.
PHY decoder: $100M/standard;
Infer.NET: max 20% on model
Phenomenon of
Interest
Questions
• High level of required expertise
10K solvers,
100s of grad student hours per model
ProtoModels
• Painfully slow & unpredictable
solvers
Massive data sets, complex
algorithms, tricky coding for graph
traversal and numeric stability
Training Data
• Challenges constructing models
Limited modeling vocabulary;
models entwined with solvers
Loss Function
Approved for Public Release; Distribution Unlimited
Machine Learning
Algorithm
(A Solver)
10
We’re Missing a Tool to Write These Applications
ISR
Too Complex to Imagine
1. Seismic Monitoring for
Nuclear Test Ban Treaty
2. Image Search and
Activity Detection
PerSEAS, VIRAT,
VMR
3. Object tracking (video)
PerSEAS, VIRAT,
CZTS
4. Patterns of Life
5. Bird Migration Patterns
ONR
Natural Language Processing
6. Watson
7. Topic Models
8. Distributed Topic Models
9. Citation Analysis
CALO
CALO
Requires
Heroic Effort
BOLT, DEFT,
MADCAT, RATS
10. Entity Resolution
BOLT, DEFT,
MADCAT
11. NLP Sequence Tagging
BOLT, DEFT,
MADCAT
Predictive Analytics
12. Microsoft Matchbox
13. Netflix Challenge
14. Predictive Database
XDATA
Cyber and Other
15. Cyber Genome Lineage
16. Gene Sequencing
Inexpressible with Current Tools
Cyber Defense
Approved for Public Release; Distribution Unlimited
11
The Missing Tool (Explained by Example)
Model:
mem strength person = gaussian 100 10
lazy person = flip 0.1
Source: Noah Goodman, POPL (2013)
pulling person = if lazy person then (strength person) / 2
else strength person
total-pulling team = sum (map pulling team)
winner team1 team2 = greater (total-pulling team1)
(total-pulling team2)
Query:
System will calculate probability distribution
strength Bob
for Bob’s strength given known facts
Facts:
[Bob, Mark] = winner [Bob, Mark] [Tom, Sam]
[Bob, Fred] = winner [Bob, Fred] [Jon, Jim]
The user describes the model at a high level. An inference engine analyzes the
program, query, data, and available hardware resources to produce best solution
Approved for Public Release; Distribution Unlimited
12
Richer Example:
Seismic Monitoring for Nuclear Test Ban Compliance
Goal: Deduce a bulletin listing seismic events with time, location, depth, and magnitude
Given: All the seismic detections and misdetections observed by a network of stations
Comparison:
•
Existing UN System: > 28K LOC in C; took several years to build; cost ~$100M
•
NET-VISA: ~25 LOC in BLOG; written in less than 1 hour*; cost $400K
Existing
UN System
NET-VISA
Signals are
mixed up
~10,000
detections per
day, 90% false
254 monitoring
stations worldwide
Source: Stuart Russell, NIPS Probabilistic Programming Workshop (2012)
Probabilistic Programming could make ML applications better and easier to build
Approved for Public Release; Distribution Unlimited
13
The Probabilistic Programming Revolution
Traditional Programming
Probabilistic Programming
• Model
Code models capture how the data was
generated using random variables to
represent uncertainty
• Code Libraries
• Model Libraries
Libraries contain common model
components: Markov chains, deep
belief networks, etc.
• Programming
Language
• Probabilistic
Programming
Language
PPL provides probabilistic primitives &
traditional PL constructs so users can
express model, queries, and data
• Inference Engine
Inference engine analyzes probabilistic
program and chooses appropriate
solver(s) for available hardware
• Hardware
Hardware can include multi-core, GPU,
cloud-based resources, GraphLab,
UPSIDE/Analog Logic results, etc.
• Application
• Compiler
• Hardware
High-level programming languages facilitate building complex systems
Probabilistic programming languages facilitate building rich ML applications
Approved for Public Release; Distribution Unlimited
14
The Promise of Probabilistic Programming Languages
•
Shorter: Reduce LOC by 100x for machine learning applications
•
•
•
Faster: Reduce development time by 100x
•
•
•
•
Seismic Monitoring: 28K LOC in C vs. 25 LOC in BLOG
Microsoft MatchBox: 15K LOC in C# vs. 300 LOC in Fun
Seismic Monitoring: Several years vs. 1 hour
Microsoft TrueSkill: Six months for competent developer vs. 2 hours with Infer.Net
Enable quick exploration of many models
More Informative: Develop models that are 10x more sophisticated
•
•
Enable surprising, new applications
Incorporate rich domain-knowledge
•
•
•
•
•
Produce more accurate answers
Require less data
Increase robustness with respect to noise
Increase ability to cope with contradiction
Sources:
• Bayesian Data Analysis, Gelman, 2003
• Pattern Recognition and Machine Learning,
Bishop, 2007
• Science, Tanenbaum et al, 2011
With less expertise: Enable 100x more programmers
•
Separate the model (the program) from the solvers (the compiler),
enabling domain experts without machine learning PhDs to write applications
Probabilistic Programming could empower domain experts and ML experts
DISTRIBUTION STATEMENT F. Further dissemination only as directed by DARPA, (February 20, 2013) or higher DoD authority.
15
Research Challenges
•
Design probabilistic programming languages and end-user tools
•
•
•
•
•
Create model/query/prior-data analyses to determine best solver or combination
of solvers for given problem
Build an inferencing infrastructure to efficiently “solve” high-level models
•
•
•
•
•
Expressiveness vs. performance
Usability by end-users who are not machine learning experts
Profiling and debugging tools
Improve solver performance by leveraging research from PL community
Define an API so new solvers can be slotted in
Develop new solvers
Leverage strengths of different kinds of hardware: CPU, multi-core, GPU, cloud, G5, …
Develop a broader community
•
•
Enable all modelers to connect to all solvers
leverage everyone’s efforts
Develop model libraries for popular model
building blocks.
to
PL/ML collaborations are already bearing fruit:
Example shows 100x performance improvement
Source: Noah Goodman, PLDI (2013)
Approved for Public Release; Distribution Unlimited
16
If This Technology Is Wildly Successful…
1
Seismic Monitoring for
Nuclear Test Ban Treaty
7 Topic Models
Smart Autonomous
Vehicles
8 Distributed Topic Models
9 Citation Analysis
10 Entity Resolution
11 NLP Sequence Tagging
Global Scale ISR
from Satellites
12 Microsoft Matchbox
14 Predictive Database
Rethink
Robotics
BigDog
Control
Big Data Climate
Forecasting
Auto-updating
Biological
Repository
Auto-fill
Databases
Predictive UX &
Customized ISR
Clustering, Sequence
Data in Biology, etc.
Nonlinear Switching State Space Models
Bayesian Matching &
Changepoint Detection
Approved for Public Release; Distribution Unlimited
17
Notional Proposed Program Schedule, Products &
Transition Plan
Phase I (22 months)
Domain
Experts
Challenge
problems (3)
Probabilistic
Programming
Machine
Learning
Inference
Engine
Summer
School
Sessions
Phase II (12 months)
Challenge
problem (1)
Challenge
problem (1)
Challenge
problem (1)
Challenge
problem (1)
Challenge
problem (1)
Phase III (12 months)
Challenge
problem (1)
Challenge
problem (1)
Intermediate
Phase I Code
Final Phase I
Code
Final Phase II
Code
Final Phase III
Code
Intermediate
Phase I Code
Final Phase I
Code
Final Phase II
Code
Final Phase III
Code
Intermediate
Phase I Code
Final Phase I
Code
Final Phase II
Code
Final Phase III
Code
Phase I
Session
Initial Session
Phase II
Session
Phase III
Session
PI Meetings
Kick-off
PI Meeting
PI Meeting
PI Meeting
PI Meeting
PI Meeting
PI Meeting
PI Meeting
Month S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J
2017
Year
2013
2014
2015
2016
Products
•
•
•
•
Transition Plan
Probabilistic languages and end-user tools
Extensible, high-performance compilers
Library of reusable models
Supporting community
• Would include government customers from ISR, NLP,
Predictive Analytics, Cyber and other domains in
annual “Summer Schools” and in challenge problem
selection
Approved for Public Release; Distribution Unlimited
18
J
Proposed Program Structure
Technical Area 1:
Domain Experts
ISR
Technical Area 2:
Probabilistic Programming
Cyber &
Other
.....
PA
NLP
PL & ML
Representation
Experts
Probabilistic Programming Languages and
End-User Tools
Technical Area 3:
Machine Learning
New
Solvers
API to
Integrate
Solvers
Domain Experts
Current
Solvers
ML Solver &
Compiler
Experts
Technical Area 4:
Inference Engine
CPU
MultiCore
GPU
Cloud
Approved for Public Release; Distribution Unlimited
GP5
Compiler
Experts
19
Performers
TA 1: Domain
Experts
TAs 2 & 4:
Probabilistic
Programming &
Inference Engine
Galois*/OSU/Kitware/Raytheon Company & others (TBD)
Evaluator; Challenge Programs; Summer Schools
BAE*/Tufts/
Stanford/
Northeastern
Create Open
PP Platform
(OP3);
intensional
contracts
Gamelan*/
Lyric Labs/
Stanford/
Freer, LLC
Feedback
about model
likelihood and
inference
convergence
Stanford*
(TA 4 only)
Efficient back-end
solvers by
applying opt. at
multiple levels of
abstraction
CRA*/UCBerkeley/
UCI
Develop HLLs
(i.e. BLOG)
and compile
to Figaro;
algebra of
solutions
MIT*/
U. Oxford/
Cambridge/
Continuum
Analytics
Venture PPS;
SIVM;
advance
theory of PP
Indiana*/
McMaster
Symbolic
probabilistic
intermediate
language
Provide support to TAs 2 & 4
teammates to integrate the novel
algorithms, representations or analyses
discovered into the developed PPS
TA 3: Machine
Learning &
Programming
Languages
11 Total Primes (*)
Princeton*
Generic
variational
inference;
streaming
computation;
model diagnosis
and fitness
SRI*/
UT-Dallas/
NYU
Research
advances in
lifted inference
and relational
representation
UC-Riverside*
Inference
engines for
continuous-time
(asynchronous
event) systems
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
ACS*/
Columbia
Discriminative
solvers to solve
discriminative
tasks specified
in probabilistic
programs
Galois - Technical Area 1
1. Develop Challenge Problems (CPs)
•
•
Apply formal procedure for selecting CPs:
a) solicitation via Wiki and b) evaluation
based on “required” and “diversity” criteria
Initial set of CPs: Quad-Rotor Sensor Fusion;
Tracking Bird Migration; Wide Area Motion
Imagery Track Linking
2. Evaluate PPSs w.r.t. each CP
•
•
•
•
•
•
© Gaui
© Audobon of FL
Source: DARPA.mil
4. Foster Collaboration
3. Run Summer Schools
•
•
•
Provide teams with problem specific APIs
Evaluate each team’s solutions on a “fresh”
evaluation data set
Apply standard and non-standard evaluation
metrics
Conduct benchmarking studies
Use PL specific evaluation metrics to evaluate
expressiveness and flexibility of PPLs
Generate evaluation report before each PI
meeting
Recruit participants from within and outside
the PPAML community
Develop curriculum and define tutorial
problems with help from TA2-4 teams
Generate evaluation reports, detailing the
performance of each PPS
•
•
•
Create Internal Wiki: program documents;
materials for CP selection process; data and
code for CPs; evaluation API materials, etc.
Create External Wiki: publicly released CPs
and educational materials and pointers to
publications
Organize workshops co-located with
academic conferences (e.g.,
NIPS, ICML, POPL, etc.)
Source: smithsystem.com
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Source: technorati.com
Source: carolmunro.com
Subs: OSU, Kitware, Raytheon Company, and others
MIT – Technical Areas 2 and 4
Subs: Cambridge University, U. of Oxford, Continuum Analytics
Goal: Make the Venture PPS useable by a wide array of people for a wide array of ML tasks
Technical Approach:
•
Develop debugging and profiling tools
•
Develop a web-based interactive user environment
•
Develop a library of reusable models
•
Develop a suite of on-line training materials
•
•
•
Extend current front-end languages (StarChurch for
machine learning experts and VentureScript for
domain experts)
Advance the state of the theory of PP
Improve the runtime performance of the backend
(Stochastic Inference Virtual Machine)
• Target parallel hardware
• Adopt hybrid solver techniques
• Enable a "foreign function interface" to enable
calls to custom solvers for specific model types
• Develop a benchmark suite to drive
performance improvements
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
StarChurch
Gamelan Labs, Inc. – Technical Areas 2 and 4
Team Members: Stanford University, ADI Lyric Labs
Goal: Agile inference - Probabilistic Programming System that provides rapid feedback to
program transformer about likelihood and convergence of probabilistic programs
Technical Approach:
• Create Probabilistic Java by unifying Dimple (graphical models) and Chimple (Church
in Java) into a single, fast, comprehensive, and scalable PPS
• Develop probabilistic unit testing, debugging, and profiling capabilities
• Provide high performance back-ends using new forms of map-reduce based parallel
inference algorithms executing in the cloud (e.g., Amazon EC2)
• Use enhanced Stochastic Lambda Calculus (eSLC) as intermediate representation
• Leverage diversity of existing solvers and some newly assimilated solvers
• Proposed model execution speed up is achieved by new forms of map-reduce based
parallel inference algorithms executing in the “cloud.”
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Stanford University – Technical Area 4
Goal: Develop efficient inference techniques by applying optimizations at multiple levels
Technique / Speed-Up
Technical Approach
Statistical / 103
Learn more efficient inference from past experience
Compilation / 104
Remove overhead and target efficient hardware
(e.g., FPGAs, GPUs, and CPU vector units)
Meta-Compilation / 102
Generate highly-tuned assembly code
Compilation: Specializing on random
Statistical: Local joint distributions that can be learned
from program execution and used for inverse sampling.
choices that affect control flow
(structural variables) permits the
generation of very efficient code.
Meta-Compilation: Montgomery multiplication
kernel (top left), compiled by gcc -O3 (right) and
stochastic superoptimization (bottom left).
Superoptimized code is 16 lines shorter, 1.6x
faster than gcc -O3, and slightly faster than
expert handwritten assembly.
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
BAE Systems – Technical Areas 2 and 4
Subs: Northeastern University, Tufts University, Noah Goodman
(consultant)
Goal: Create an Open Probabilistic Programming Platform (OP3) using new language
construct: intensional contracts.
Technical Approach:
•
(TA2) Extend Racket Language Platform to
develop DSLs for novices and experts
•
(TA2) Develop IDE building on Racket
•
(TA2 and TA4) Extend the contract feature of
Racket to include intensional contracts
• Support analysis of generative models and
selection of inference algorithms
•
(TA2 and TA4) Design a stochastic IR
• Compilation target from the DSLs, inference
algorithms, and intensional contracts
• Static and dynamic analysis to reason over
the IR to optimize probabilistic programs
w.r.t. desired intensional properties
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Source: racket-lang.org/
Charles River Analytics – Technical Areas 2 and 4
Subs: UC Berkeley, UC Irvine
Goal: Develop a fully integrated PPS that
allows users to create rich models and
automatically derive efficient
implementations using:
•
high-level languages (HLLs);
•
compositional inference; and a
•
development environment
Technical Approach:
Front-End (TA2):
• Develop HLLs (BLOG & visual language)
• Improve Figaro as a compilation target
IR for HLLs
• Develop compilers for HLLs into Figaro
• Build model libraries and design
patterns for HLLs and Figaro
Back-End (TA4):
• Develop inference and data API based
on solution algebra
• Develop solution optimizer
• Develop deployment methods to
efficiently deploy solvers on HW
• Investigate new solvers
Approach and Architecture
Development Environment:
• Develop debugging & verification tools
• Develop profiling tools
• Develop query and results interface
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Indiana University – Technical Areas 2 and 4
Sub: McMaster University
Goal: Develop an efficient PPS centered around a symbolic probabilistic
intermediate language that represents a variety of models and algorithms
Technical Approach:
• Build automatic debugging and profiling tools
•
Automate parallel inference and online learning on multiple machines and
cores (w/o need for input preprocessing and output postprocessing)
•
Combine multiple inference algorithms on the same model via a tactic
language a la Coq and a front-end GUI for interactive experimentation
•
Support nonparametric models
Syntax of Symbolic Probabilistic Intermediate Language
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Princeton – Technical Area 3
Goals
Technical Approach
Scalable computation
Develop new generic variational inference
algorithms based on stochastic variational
inference
Streaming data sources
Combine the idea of the population
distribution from frequentist statistics with
Bayesian probabilistic models
Model diagnostics and fitness
Adapt ideas from Bayesian statistics:
posterior predictive checks and predictive
sample reuse
Model
(Program)
Inference
Algorithm
Posterior
Distribution
Data Stream
Applications
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Evaluation &
Revision
SRI – Technical Area 3 – Lifted Inference for PP
Subs: UT-Dallas, NYU
• Goal: Provide higher-level, lifted inference at abstract level (e.g., happy(Person)
for a generic, unbound variable Person) instead of individual random variables
(happy(bob), happy(mary), …)
• This leads to symbolic, logic reasoning for symbolically manipulating terms
representing probabilistic distributions
Power of Lifted Inference
• Both logic and probabilistic reasoning methods
•
•
•
•
•
•
•
First-order Variable Unification
Search
Sampling
Message passing of structured information
Use of determinism and quasi-determinism
Compilation
Variational inference (as symbolic simplification)
• Lazy symbolic evaluation leads to “proof trees” or
“explanation trees” that get increasingly refined
from coarse to detailed. This is faster and helps
presenting understandable debugging information
KL Error
apply, providing a unification reducing to each of
them at the extremes. Some methods are:
Time (minutes)
Alchemy 2.0, which uses lifted inference,
is much better in terms of accuracy than
propositional approaches such as
Alchemy 1.0 and Tuffy. [UAI, Gogate and
Domingos, 2011]
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Applied Communication Sciences – Technical Area 3
Subs: Columbia University
Goal: Enable the use of discriminative
solvers to solve discriminative tasks
specified in probabilistic programs
Technical Approach:
• Leverage the semantic content of
probabilistic programs to construct
new kernels and new features in
discriminative solvers
•
Examples of Discriminative Tasks
Ranking (search engine)
Regression
Source: Wikipedia
Source: Wikipedia
Classification
Expand the scope of discriminative
models to solve complex tasks by
developing new loss functions to
compare structured objects
© University of Washington
Sequence Labeling (part of speech tagging)
•
Apply meta-learning to automatically
select appropriate solver parameters
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Source: www.clips.ua.ac.be
UC Riverside – Technical Area 3
Goal: Develop inference methods for
continuous-time systems
Technical Approach:
•
Markov Models:
(using time-ordered product
expansion, TOPE)
•
•
Many Machine Learning
Problems Involve
Continuous-Time Data
Social Network Dynamics
Deterministic, anytime inference
Auxiliary Gibbs sampling
Source: www.becreative-inc.com
•
History-based Models:
(piece-wise constant intensity
models, PCIMS)
•
Economic Trade
Biological Systems
Source: stocksonwallstreet.net
Computer Server Logs
Auxiliary Gibbs sampling
(using Poisson process thinning)
Source: alumni.kcl.ac.uk
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
Source: wiki.r1soft.com
32
www.darpa.mil
Distribution Statement A (Approved for Public Release, Distribution Unlimited)
33