Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Technical Cognitive Systems
Michael Beetz
Intelligent Autonomous Systems
Technische Universität München
Summer Term 2012
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Autonomous Robot Control as a Structured Dynamic System
Controlled process
World process
Robot
Environment
Sensor process
Controlling Process
Plan-based control
Perception
manipulation /
body movement
belief
state /
reasoning
navigation
sensor data
processing
probabilistic
estimation
Cognitive capabilities
Learning
Knowledge
processing /
decision
making
Planning
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
What’s Missing in CR: The Interface Layer
adapted from Pedro Domingos: “What’s Missing in AI: the Interface Layer”
Field
Interface Layer
Operating
Systems
Programming
systems
Databases
virtual
machines
high-level
languages
relational
model
Below
Layer
hardware
the
compilers, optimizers, ...
query
optimization,
db
design,
transaction
mgmt
Above
Layer
software
the
programming
enterprise applications
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
What’s Missing in CR: The Interface Layer
adapted from Pedro Domingos: “What’s Missing in AI: the Interface Layer”
Field
Interface Layer
Operating
Systems
Programming
systems
Databases
virtual
machines
high-level
languages
relational
model
Personal
robotics
CRAM
Below
Layer
hardware
the
compilers, optimizers, ...
query
optimization,
db
design,
transaction
mgmt
grounding
in robot, AI
tools,
the
nuts and bolts
of intelligent
robotics, ...
Above
Layer
software
the
programming
enterprise applications
robot application programming
raise the conceptual level at which service and personal robot applications are programmed!
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
A Software Architecture of the Robot Controller
Structured Reactive Controller (SRC)
Task-oriented
Layer
Process
Modules
Robot Abstract Machine
Registers
Structured
Reactive Plan
(SRP)
Process
Modules
Fluents
Control
Structures
Implementation of the Robot Abstract Machine
Functional
Layer
Control
Signals
Perceptual
Subsystem
Action Subsystem
Navigation
System
Image
Processing
System
Sensor
Data
Probabilistic
State
Estimators
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. a software architecture for plan-based robot control systems
2. the functional layer of autonomous robot control systems
3. the robot abstract machine
4. control patterns for specifying flexible and reliable behavior
5. integration of different mechanisms
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
The Functional Layer of the PR2 robot controller
e
pos
Table/ObjectsOnTable
sp
T il
t s
pee
d
Gra
End effector position
and orientation
Sampling-based
motion (re)planning
Task Layer
Task executive
Ma
p
Grid planner
Base goal position
Manipulation
ion
Grasping
Reaching
Scene Interpretor
for Manipulation
llis
Co
Arm executive
Base executive
Dynamic Obstacle
Map
Functional Layer
Partial-view grasp
planner
3D Perception
Navigation
Base controller
PR2 Hardware Controllers
Laser tilt controller
Basic Control
Layer
Arm joint position controller
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Environment Maps as Resources
• many plan-based robot controllers use environment maps as data structures
in order to accomplish its tasks more reliably and more efficiently.
• Maps support self localization, generation of good navigation plans,
localization of objects, ...
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Semantic Maps
• semantic maps enable robots to perform
tasks such as “get a book from the shelf in
room A-120 and bring it to Wolfram’s
desk.”
• task-relevant objects are represented as
hierarchically structured 3D models
• Object models can be retrieved through
relational symbolic queries such as “the
desk in room A-120.”
• aspects of the environment structure are
represented explicitly to allow for behavior
adaptation
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Probabilistic State Estimation: Self Localization
computes P(loc|percepts, actions, init)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Door State Estimation
computes P(doorangles|percepts, actions, init)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Point Cloud Perception Unit
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Navigation System
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
The Robot Abstract Machine
concurrent reactive
control
as
k
Principle 1
Sensor and process events
trigger and synchronize
concurrent processes.
Principle 3
Large parts of the concurrent
control program are plans.
Reasoning and planning
mechanisms operate on
representations of the
concurrent control program
and can reason about it and
modify it during execution.
te
ll
knowledge & reasoning
Principle 2
Logical queries
infer decisions
and parameters
using specialized
inference
methods.
• Components of robot
abstract machine
– data abstractions
– primitive statements
– control structures
planning reactive
behaviour
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
RPL: Data Abstractions
• Fluents + fluent networks
• designators: symbolic descriptions of locations, actions and objects
• low-bandwidth data:
do computation and resource critical computations as control processes
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
RPL: Probabilistic Belief States and Fluents
fluents = registers + update mechanisms
Examples:
ROBOT-X*
ROBOT-Y*
POWER-VOLTAGE*
POWER-VOLTAGE-WARNING?*
GO-PATH-DONE?*
GO-PATH-TERMINATED?*
NR-OF-NAV-GOALS*
...
ROBOT-X*
ROBOT-Y*
⇒ ROBOT-O*
NR-OF-LOCAL-MAXIMA*
LOC-ACCURACY*
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Abstract Percepts and Concurrent Percept-driven Plans
14??
+
16??
1135
POS-Y
780
THRACC
THRAMB
px
>
ARRIVED?
x2
>
<
^
IN-HALLWAY?
_
ACTIVE-LOCALIZATION-NECESSARY?
<
POS-ACCURACY
POS-AMBIGUITY
50
x2
POS-X
>
with policy whenever( active-loc-nec? )
relocalize;
execute concurrently
execute-in-order
go-to (1); wait for ( arrived-at(1)? );
go-to (2); wait for ( arrived-at(2)? );
execute-in-order
set-navigation-mode(office);
wait for( entering-doorway? );
set-navigation-mode(doorway);
wait for( entering-hallway? );
set-navigation-mode(hallway)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Plan Language Primitives
Process Modules, Assignments, NO-OP & FAIL
CONTROL INPUTS
ACTIVATE
DEACTIVATE
PARAM1
PROCESS
PARAMETERS
PROCESSMODULE
...
FLUENT1
OUTPUT
FLUENTS
FLUENTn
PARAMn
SUCCESS
with local fluents (IMG-ID-FL DONE-FL)
GRAB-IMAGE(:CAMERA
:LEFT
:SIZE
(240 320)
:COLOR
:TRUE
:GRABBED-IMAGEIMG-ID-FL
:DONE
DONE-FL)
wait for(DONE-FL)
FLUENT-VALUE(IMG-ID-FL)
FAILURE
FEEDBACK SIGNALS
ACTIVATE
DEACTIVATE
RESOLUTION
GRAB-IMAGE
IMAGE
COLOR
SUCCESS
FAILURE
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Plan Primitives: Implementing Process Modules
Examples:
ROBOT-GOPATH
ROBOT-START-GOPATH
ROBOT-STOP-GOPATH
ROBOT-FACE-COORDS
ROBOT-START-FACE-COORDS
ROBOT-STOP-FACE-COORDS
ROBOT-SET-NAVIGATION-MODE
ROBOT-SAY
...
PROCEDURE ROBOT-START-GOPATH ([< X1 , Y1 >, ..., < Xn , Yn >])
FUNCALL ROBOT-START-GOPATH-DEMON*([< X1 , Y1 >, ...,< Xn , Yn >])
FUNCALL ROBOT-START-GOPATH-IMPL*([< X1 , Y1 >, ..., < Xn , Yn >])
DEFVAR ROBOT-START-GOPATH-IMPL*
← PROXY-SERVER:ROBOT-START-GOPATH
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Design criteria for concurrent reactive robot control
languages
• process concept
– naming processes
– synchronization
• signals
– Success
– Failure
• communication
• exception handling
• failure handling and recovery
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Control Structures
Control Structure
in parallel do p1 ...pn
try in parallel p1 ...pn
with policy p b
plan with name N1 p1
...
with name Nn pn
order ni < nj
Example of its usage
in parallel do navigate(h235,468i)
build-grid-map()
try in parallel
detect-door-with-laser()
detect-door-with-camera()
with policy relocalize-if-nec()
deliver-mail()
plan with name S1 put-ontable(C)
with name S2 put-on(A,B)
with name S3 put-on(B,C)
order S1 ≺ S3 ,
S3 ≺ S2
+ Semaphores, priorities, interrupt detection
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
RPL: Semantics of Control Structures (1)
• start(T) — Function. Situation in which T starts
• completed(T,S) — Predicate. Program step T is successfully completed in
or has been completed before situation S
• end(T) — Function. Situation in which T terminates
• active(T,S) — Predicate. Program step T is active in situation S
• failed(T,S) — Predicate. Program step T failed in or before situation S
• steps(T,S) — Function. Primitive subprogram steps of the program step T
in situation S.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
RPL: Semantics of Control Structures (2)
T = in parallel do(T1 , T2 ) →
• (active(T1 , S) ↔ active(T , S) ∧ ¬completed(T1 , S) ∧ ¬failed(T1 , S))
• ∧(active(T2 , S) ↔ active(T , S) ∧ ¬completed(T2 , S) ∧ ¬failed(T2 , S))
• ∧(completed(T , S) ↔ completed(T1 , S) ∧ completed(T2 , S))
• ∧(failed(T1 , S) → failed(T , S))
• ∧(failed(T2 , S) → failed(T , S))
• ∧(active(T1 , S) → steps(T1 , S) ⊂ steps(T , S))
• ∧(active(T2 , S) → steps(T2 , S) ⊂ steps(T , S))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Knowledge-enabled action selection and parametrization
Cleaning Up
Setting the Table
Making Pancakes
Tasks
Legend:
Prolog Queries
Mobile Manipulation
Locating Objects
searchLocation
Reasoning about
Object Similarity
Plan Parametrization
neededObjectsForMeal
Logical
Inference
Probabilistic
Inference
Pick and Place
stable
reachable
Physics
Simulation
blocking
Inverse
Kinematics
Subtasks
Perspective Taking
arplace
visible
Action-Related
Concepts
occluding
Off-Screen
Rendering
Prolog
Predicates
Reasoning
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans
From STRIPS Plans to plans for autonomous robots
a STRIPS plan:
1. PICK-UP(B)
2. STACK(B,C)
3. PICK-UP(A)
4. STACK(A,B)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans
Using predefined plan schemata
We don’t want the robot to get hit by a truck while thinking about how to cross
the street.
(PUT-ON ?X ?Y)
(SEQ (IF (ONTABLE ?X)
(PICK-UP ?X)
(UNSTACK ?X ?Z))
(STACK ?X ?Y))
Preconditions: (HAND-EMPTY) ∧ (CLEAR ?X) ∧ (CLEAR ?Y)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans
Universal Plans
dealing with incomplete information
(DEF-PLAN-SCHEMA (PUT-ON* ?X ?Y)
(IF (NOT (ON ?X ?Y))
(SEQ (IF (NOT (CLEAR ?Y))
(CLEAR-THE-TOP-OF ?Y))
(IF (IN-HAND ?X)
(STACK ?X ?Y)
(SEQ (IF (NOT (CLEAR ?X))
(CLEAR-THE-TOP-OF ?X))
(PICK-UP ?X)
(STACK ?X ?Y))))))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans
Improving reliability
• (N-TIMES 3
(PICK-UP ?X)
UNTIL (HIGH? HAND-FORCE-SENSOR))
• (WITH-POLICY (WHEN-EVER
(DROPPED? HAND-FORCE-SENSOR)
(PICK-UP ?X))
(STACK ?X ?Y))
• (N-TIMES 3
(PUT-ON ?X ?Y)
UNTIL (ON ?X ?Y))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans:
Integration von Perceptions
• We can’t say : “(PICK-UP BLOCK-A)”
BLOCK-A is an object outside the computer; combinatorics
• Indexical-Functional (Deictic) Representation:
make reference to the robot’s situation or goals
the-block-I’m-pushing, the-bee-chasing-me, ...
• Instead, we could say:
(LET ((X FALSE))
(!= X (LOOK-FOR-ONE ((CAT BALL) (COLOR RED))))
(WITH-POLICY (TRACK X)
(PICK-UP X)))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans: Designators
• (LET ((X (LOOK-FOR-ONE ((CAT BALL) (COLOR RED))))
(PICK-UP X)))
• Can X be bound to the red ball? (Tarskian Semantics)
• No. Variables can at best be bound to (partial) descriptions of objects,
that is, to information sufficient to manipulate and reacquire the objects.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans: Designators
Codesignation of designators
(EQUATE D1 D2): operation that takes two designators and links them under
the assumption that they refer to the same object.
(DEF-PLAN FIND-AND-PICKUP (?DES)
(LET ((NEW (LOOK-FOR DESIGNATOR)))
;; NEW is a list of things that look like DESIGNATOR
(IF (= (LENGTH NEW) 1)
(SEQ (EQUATE DESIGNATOR (CAR NEW))
(PICK-UP DESIGNATOR))
(FAIL)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Designing robot action Plans: sketchy Plans
• “Getting a glass of water from the kitchen”
– classical planning: you have to know exactly where the water glass in the
cupboard is, whether the glass is clean, where the bags in the corridor lie, what
will happen in the kitchen
– but: the cat might jump at you, sth’s burning in the kitchen, no clean glass, ...
• The choice of detailed actions depends on the context in which they are executed
• “Reactive Planning:” dealing with the real world
– execution monitoring and low-level replanning
– monitoring the world
– adapting to the situations encountered
• RAPs’ principles:
– what to do next is entirely based on the current situation
– when a RAP finishes successfully it is guaranteed to have satisfied its goal and
performed all necessary sensor actions
– when it fails it has exhausted any possibility to accomplish the goal
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Concurrent Reactive Navigation Plans
A-110
A-121
111111
000000
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
5
000000
111111
000000
111111
000000
111111
000000
111111
A-111
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
4
000000
111111
000000
111111
000000
111111
000000
111111
execute concurrently
execute-in-order
A-113
1111111111111111111111
0000000000000000000000
0000000000000000000000
1111111111111111111111
3
0000000000000000000000
1111111111111111111111
0000000000000000000000
1111111111111111111111
2
0000000000000000000000
1111111111111111111111
0000000
1111111
0000000
1111111
0000000000000000000000
1111111111111111111111
0000000
1111111
0000000
1111111
A-120
Dieter’s
Desk
0000000
1111111
1
1111111
0000000
0000000
1111111
A-117
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
go-to (1); wait for ( arrived-at(1)? );
go-to (2); wait for ( arrived-at(2)? );
execute-in-order
set-navigation-mode(office);
wait for( entering-doorway? );
set-navigation-mode(doorway);
wait for( entering-hallway? );
set-navigation-mode(hallway)
Design Rationale 1: use expressive behavior specification
languages!!!
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Structured Reactive Navigation Plans
(desk-1,desk-2)
navigation plan
with subplans
traverse-narrow-passage(h635, 1274i,h635, 1076i)
parameterizations
sonar ← off
colli-mode ← slow
path constraints
h635, 1274i,h635, 1076i
justification
narrow-passage-bug-3
traverse-narrow-passage(...)
traverse-free-space(...)
default-go-to ( desk-2 )
Design Rationale 2: means for structuring continuous behavior!!!
Design Rationale 3: subplan macros for compact and transparent
representation!!!
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
High-level Navigation Plans
highlevel-plan achieve (loc(rhino, hx, y i)
with cleanup routine abort-navigation-process do
with valve wheels do
loop
try in parallel
wait for(navigation-interrupted?)
generate-and-execute-nav-plan(hx, y i)
until is-close?(hx, y i)
Design Rationale 4: use general, embeddable, transparent, and interruptable
plans!!!
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
at location hx,yi p Plans (Drew McDermott)
named subplanNi
doat location hx, y i p by
with valve Wheels
dowith local vars done? ← false
doloop
try in parallel
wait for Task-Interrupted?(Ni )
sequentially do
Navigate-Tohx, y i
p
done? ← true
until done? = true
Design Rationale 5: use modular and declarative plan macros!!!
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Deliver(o,d) Plans
partial ordernamed subplans1
doif execution-state(p, to-be-acquired)
then At-Location L pick-up(o)
named subplans2
doif execution-state(p, loaded)
then At-Location D put-down(o)
:orders1 ≺ s2
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Robot Courier Plan
top-level-plan OfficeCourier( )
1 with stabilizer relocalize when necessary
2
reload battery when necessary
3 dowith plan adaptor manage the plan with respect to the user’s
4
requests and the robot’s utility model
5
donamed subplanmain − plan
6
dowith stabilizer situation specific stabilizers
7
dowith plan adaptor situation specific plan adaptors
8
dopartial order set of delivery plans
9
containing at location subplans
10
:order ordering constraints on at location subplans
Design Rationale 6: represent plan management routines explicitly!!!
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Navigation (1)
A-110
A-121
111111
000000
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
5
000000
111111
000000
111111
000000
111111
000000
111111
A-111
000000
111111
000000
111111
000000
111111
000000
111111
000000
111111
4
000000
111111
000000
111111
000000
111111
000000
111111
A-113
1111111111111111111111
0000000000000000000000
0000000000000000000000
1111111111111111111111
3
0000000000000000000000
1111111111111111111111
0000000000000000000000
1111111111111111111111
2
0000000000000000000000
1111111111111111111111
0000000
1111111
0000000
1111111
0000000000000000000000
1111111111111111111111
0000000
1111111
0000000
1111111
A-120
Dieter’s
Desk
1111111
0000000
1
0000000
1111111
0000000
1111111
A-117
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Navigation (2)
in parallel do
sequentially do
start-nav-plan (1 2)
wait-for(navigation-completed?)
with local fluents
entering-dw?-fl ← fluent-net (| hx, y i − hxdw , ydw i |< 1m)
leaving-dw?-fl ← fluent-net (| hx, y i − hxdw , ydw i |> 1m)
set-navigation-strategy(office)
wait-for(entering-dw?-fl)
set-navigation-strategy(doorway)
wait-for(entering-hw?-fl)
set-navigation-strategy(hallway)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Image Processing (1)
ROI-SPEC
ROI-SPEC
IMAGE
IMAGE
CREATE-ROI
ROI
CANNY
IMG-OP
ROI
IMAGE
PROCESS-IMAGE
ROI
IMAGE
IMAGE
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Image Processing Macro
IP
1
2
3
4
5
6
7
8
9
pipeline ip-example(:inputimg-in, :outputimg-out)
:stepcreate-roi
rpl-create-roi(:borders hh100, 10i, h240, 240ii)
:stepprocess-image
rpl-process-image(:operator :canny
:threshold 80)
:connections(:input :image) → (process-image :input :image)
(:input :image) → (create-roi :input :image)
(create-roi :output :roi)
→ (process-image :input :image)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Image Processing Macro Expansion
Macro Expansion ip-example(img-in, img-out)
1 with local fluentscreate-roi-done-fl
2
← create-fluent(f1, false)
3
create-roi-roi-fl
4
← create-fluent(f2, false)
5
process-image-done-fl
6
← create-fluent(f3, false)
7
process-image-image-fl
8
← create-fluent(f4, false)
9
process-image-roi-fl
10
← create-fluent(f5, false)
11 do
12
drive(img-out, process-image-image-fl)
13
partial ordernamed subplancreate-roi
14
dorpl-create-roi(fluent-value(img-in)),
15
100, 10, 240, 240,
16
create-roi-roi-fl,
17
create-roi-done-fl)
18
wait forcreate-roi-done-fl
19
named subplanprocess-image
20
dorpl-transform-image
21
(fluent-value(img-in)),
22
fluent-value(create-roi-roi-fl)),
23
2, 80,
24
process-image-image-fl,
25
process-image-roi-fl,
26
process-image-done-fl)
27
wait forprocess-image-done-fl
28
:ordercreate-roi ≺ process-image
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Integration of Image Processing
with valvewheels
dowith valvecamera
dotry in parallel
sequentially
dofind-a-predicted-line(pred-line-fl,
...,
done?)
wait fordone?
wait forconfidence-fl > 0.7
wait time5
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Conversational Plans
with policywhenevernew-email-arrived?
doprocess incoming email
dowith policywheneverambiguous-cmd?
doclarify command
do...
ask for help
...
inform people
...
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Task-specific Plans (1)
RPL
1
2
3
4
5
6
7
8
9
10
11
12
procedure try-load-letter(letter-desig)
n-times2
do
execute-speech-act
(request :sender robot
:content (achieve (on-board letter-desig)),
:channel :sound)
robot-ask-button(params)
try in parallel
wait time15wait forbutton-pressed?
untilbutton-pressed?
if¬button-pressed?
thenfail“:nobody-loaded-a-letter”
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Lowlevel Plans: Task-specific Plans (2)
RPL procedure load-letter(letter-desig)
1 with valvewheels
2 dowith valvebuttons
3
dotry-load-letter(letter-desig)
4
update-desig(letter-desig
5
:color pressed-button
6
:loc container)
7
update-robot-container(:add letter-desig)
8
iflength(get-desigs(:color pressed-button
9
:loc container))
10
>1
11
then
12
fail“carrying two identical letters”
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Representation in Plan-based Control
• plan representation is a specific form of knowledge representation
• plan representation =
syntax
+ semantics
+ inference mechanisms
+ plan interpretation
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Requirements for Plan Representations
adapted from: Rich and Knight
1. Representational Adequacy
– control patterns and intentions
2. Inferential Adequacy
– causal, teleological, and structural inferences
3. Inferential Efficiency
– fast inference
4. Acquisitional Efficiency
– learning plan schemata
but Trade-off: expressiveness ⇔ efficiency
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Representational adequacy
• specify flexible and reliable behavior
• provide control abstractions
• discretizing continuous behavior
• reactivity and concurrency
• ...
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Inferential adequacy
• causal: the subplan remes the letter from the desk
• teleological: the subplan is executed to deliver the letter specified in user
request 3
• structural: the letter is delivered by first picking it up and then driving to
its destination while holding on to it, and releasing it there
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Inferential efficiency
• the available computational resources are largely determined by the working
speed of the robot.
.
.
WOLFRAM's SHELF
WOLFRAM's SHELF
Room A-111
Room A-111
(18 1 12)
(CALL PROJECTION OPP)
(17 5 17)
(CALL PROJECTION OPP)
JAN's SHELF
JAN's SHELF
A-118's TABLE
MICHAEL's SHELF
Room A-120
A-118's TABLE
MICHAEL's SHELF
(18 1 14)
(DO CANCEL OPPORTUNITY)
Room A-118
Room A-117
Room A-120
.
(17 5 31)
(DO CANCEL OPPORTUNITY)
Room A-118
Room A-117
.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Acquisitionally efficiency
• support the automatic acquisition of plan schemata
• how easily and concisely can plan schemata be specified?
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Typical plan language design strategy
1. define a restricted domain modeling, action, and plan language
2. implement an effective planning system for this language
3. add to the expressiveness of the language
4. go to 2
Example: the POP algorithms
• POP. UCPOP: handles universal quantification and conditional effects.
ZENO: temporal planner handles deadline goals and continuous change;
BURIDAN: probabilistic planner reasons about uncertainty. CBURIDAN:
extends BURIDAN with sensing actions and contingent execution. XII:
executes sensing actions to handle incomplete information with the Internet
softbot. Occam: planner optimized for information gathering.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. what is plan representation
2. requirements for plan languages
3. designing expressive plan languages for efficient reasoning
4. data structures for plan representation
5. from STRIPS plans to modern robot plans
6. plans of an robot office courier
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
An alternative approach
Plan representation language =
= Reactive Plan Language
+ Macro mechanisms
+ special purpose mechanisms for
– code tree management
– subplan naming and indexing
– ...
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Why should such expressive plan languages be a good idea?
reactive plan languages enable programmers to write control routines that
• work,
• are concise,
• are modular,
• are transparent,
• ...
but: feasibility of reasoning about such plans
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Why should planning for expressive plan languages be
feasible?
• a planner does not need to reason about all languages written in a plan
language but only about those that it can generate
• for example: a transformational planning system can generate
– the “empty” plan
– plans that result from inserting an instantiated plan schema into a valid plan
– plans that result from revising a valid plan
– that is all
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Why does this help?
Suppose P is a set of properties such that all plans that satisfy P are easy to
reason about
Concurrent,
Percept-driven
Plans
If P is a set of properties such that
• the empty plan satisfies P
Revision of
Plans using R
(including
insertion)
empty plan
• the library plans satisfy P
• the insertion of library plans
preserves P
• the revision rules R preserve P.
then all plans that an SRC constructs
and revises satisfy P.
Plan Library
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Desirable Properties of Robot Action Plans
Robot Action Plans should be
• general: work for typical situations and events
• embeddable: work without additional synchronizations
• interruptable: continue successfully after interruptions
• modular and transparent so that automatic planning algorithms can
reason about them
• ...
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. what is plan representation
2. requirements for plan languages
3. designing expressive plan languages for efficient reasoning
4. data structures for plan representation
5. from STRIPS plans to modern robot plans
6. plans of an robot office courier
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Representing Plans through Code Tree
(PAR α
(IF c β γ )
δ
)
(BRANCH 1)
(IF c β γ )
(IF-ARM TRUE)
(IF-ARM FALSE)
β
γ
transparently?
• Solution: a query language for code
(BRANCH 3)
trees
(BRANCH 2)
α
• Problem: how to access subplans
δ
(AND (TASK-GOAL ?P-TSK
(PERCEIVE-1
(λ (?X) (AND (CATEGORY ?X BALL)
(LOC ?X h0, 8i)))))
(RETURNED-VALUE ?P-TSK {})
(HOLDS (AND (CATEGORY ?X BALL)
(LOC ?X h0, 8i))
(TT ?P-TSK))
(CODE-PATH ?P-TSK ?CP))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Replacing Subplans
(PAR α
(IF c β γ )
δ
)
(BRANCH 1)
(BRANCH 3)
(BRANCH 2)
α
(IF c β γ )
(IF-ARM TRUE)
(IF-ARM FALSE)
β
γ
δ
(PAR α
(SEQ (IF c
δ)
(CODE-REPLACE
(PAR α
(IF c β γ )
δ
)
((BRANCH 2))
(SEQ (IF c β γ )ε ) )
)
(BRANCH 1)
(BRANCH 2)
α
(SEQ (IF c
)
(IF c β γ )
(IF-ARM TRUE)
(IF-ARM FALSE)
β
γ
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
An Abstract Plan Revision Rules

Plan
1. Revision is promising
AT CodePath 
 2. Know CodePath of Plan
== Pattern 
to be Revised


3.
Have
all Information
↓

to
Instantiate
NewPlan
NewPlan
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
An Implementation of a Plan Revision Rules

(?TO-BE-REVISED AT ?CP)









(SEQ ?TO-BE-REVISED


(STABILIZE

((λ (?OB) ?UNACH-GOAL) 

?VAR)))
↓
(AND (TASK-GOAL ?SUBTSK
(ACHIEVE (?PRED ?DES ?VAL)))
(SUBTASK+ ?SUBTSK ?TSK)
(HOLDS (PHYSICAL-TRACK ?DES ?OB)
(DURING ?SUBTSK))
(HOLDS ?UNACH-GOAL
(END-TASK ?SUBTSK)))
(== ?UNACH-GOAL (?PRED ?OB ?VAL))
(VALUE ?VAR (BEGIN-TASK ?SUBTSK) ?DES)
(TASK-CODE-PATH ?SUBTSK ?CP)))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Plan Adaptors
With-plan-adaptor Whenever the robot receives new requests
if its set of Jobs J might interfere
then it changes the course of action
to avoid these interferences
Concurrent reactive plan
With-plan-adaptor Whenever the robot detects an open door
that was assumed to be closed
if this situation is an opportunity
then it changes its course of action
to make use of the opportunity
Concurrent reactive plan
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Discussion: Plan Representation
Reasoning
+ representational adequacy
+ inferential adequacy
Representation
+ inferential efficiency
+ acquisitional efficiency
Learning
Execution
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Conclusion
• What is the right plan representation?
– use specialized plan representations
– engineer the plan representations
• How do we design the right plan representation?
• Can we learn it?
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Where are we?
Reasoning
Representation
Learning
Execution/
Embedding
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. reasoning tasks and transformational planning
2. temporal projection
3. failure diagnosis
4. a reasoning example
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. reasoning tasks and transformational planning
2. temporal projection
3. failure diagnosis
4. a reasoning example
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Reasoning Tasks
•
•
•
•
•
•
•
•
•
•
•
prediction
anticipation of failures
diagnosis of anticipated failures
generating alternative plans
plan generation
committment management
environment monitoring
alternative assessment
plan elaboration
meta-level control
...
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Monitoring routines that revise plans
Whenever the robot receives new commands
if its set of Jobs J might interfere
then it changes the course of action
to avoid these interferences
Whenever the robot detects an open door
that was assumed to be closed
if this situation is an opportunity
then it changes its course of action
to make use of the opportunity
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Transformational Planning of Reactive Plans
CRITICIZE
ANALYZED
PLAN
EXECUTION
SCENARIO
EXECUTION
SCENARIO
EXECUTION
SCENARIO
0,0
0,0
2,5
2,5
5,5
5,5
2,7
2,7
SCORE BUGS
BETTER
PLAN?
UPDATE
RESULTS
UPDATE
PLAN QUEUE
BEST
PLAN
REVISE
• criticize step:
– project execution scenarios
– detect failures in execution
scenarios
– diagnose failures and assess their
PLAN QUEUE
severity
• revise step:
– apply revision rules to worst
failure of best plan to generate
alternative plans
PROPOSED
PLAN
PROPOSED
PLAN
PROPOSED
PLAN
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. reasoning tasks and transformational planning
2. temporal projection
3. failure diagnosis
4. a reasoning example
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Temporal Projection
User
Commands
Plan
Project
Causal
Models
Revise
Projected
Execution
Scenario
Compare
Behavior
Specifications
• temporal projection
Discrepancies
= Behavior
Flaws
Analyze
Failure
Models
is the computational process of
predicting what will happen when a
robot executes its plan
Applicable
Revision
Rules
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Causal models for robotic agents in complex environments
Controlled Process
Controlled Process
Exogenous Events
Exogenous Events
Environment
Effects
Environment
Perceptions
Controlling Process = Robot Control System
Behaviors
Plan
Fluents
Models of
exogenous
events
Sensor
Data
Interpretation
Effects
Perceptions
Models
of Behaviors
Controlling Process = Robot Control System
Behaviors
Plan
Fluents
Sensor
Models
Data
of Sensing
Interpretation
Processes
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
The big problem: real data → real action
real state 6= observed state
issued command 6= real effects
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Analyzing a Plan Revision
IF
a goal is clobbered
by an exogenous event
THEN stabilize the goal state
immediately after achieving it.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
A Closer Look
IF 1. goal(ob) is clobbered by an exogenous event
2. desig is a data structure returned
by the sensing routine when the routine saw ob
3. the robot tried to achieve goal(desig)
with the plan PACHIEVE (goal)
4. there exists a RPL variable var with value desig
THEN REPLACE PACHIEVE (goal)
WITH
SEQ PACHIEVE (goal)
STABILIZE(goal(var))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
“Ontology” of Execution Scenarios
Plan projector must project
• the evolution of the controlled process
• the belief state of the robot
plan
plan step
RPL variable
fluent
↔
↔
↔
plan interpretation
task
designator
RPL value
state of
plan interpretation
↔
↔
↔
↔
world
event
object
state/event
time instant
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Projecting Navigation Plans (1)
execute concurrently
execute-in-order
go-to (1); wait for (go-to-completed?);
go-to (2); wait for (go-to-completed?);
with local fluents distance-to-doorway
← fluent-network (| hx, y i − hxdw , ydw i |)
entering-dw?-fl ← distance-to-doorway < 1m
entering-hw?-fl ← distance-to-doorway > 1m
execute-in-order
set-navigation-mode(office); wait for(entering-dw?-fl);
set-navigation-mode(doorway); wait for(entering-hw?-fl);
set-navigation-mode(hallway)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Projecting Navigation Plans (2)
with policy as long as in-hallway?
whenever passing-a-door?
estimate-door-angle()
with policy seq wait for open?(A-113)
deliver-mail-to(Dieter)
1. go-to(A-111)
2. go-to(A-120) before 10:30
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Projecting Navigation Plans (3)
end(low-level-nav-plan(...))
leaving
doorway
entering
doorway
leaving
c
b
hallway
c
b
c b
b
c
leaving
doorway
c b
b
c
c
b
entering
hallway
entering
doorway
begin(low-level-nav-plan(...))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Plan Execution as a Hybrid Automaton
• linear change within control modes
• jump conditions ≈ cubes in the state space
=
v
=
v
dist(hx; y i; hxdw ; ydw i)
execute concurrently
execute-in-order
go-to (2); wait for ( arrived-at(2)? );
execute-in-order
set-navigation-mode(office);
wait for( entering-doorway? );
set-navigation-mode(doorway);
wait for( entering-hallway? );
set-navigation-mode(hallway)
100
Æ
doorway os(o0 )
Æ
doorway sin(o0 )
x
_
=
v
y
_
=
v
x
dist(hx; y i; hxdw ; ydw i)
go-to (1); wait for ( arrived-at(1)? );
>
Æ
hallway os(o0 )
Æ
hallway sin(o0 )
x
_
y
_
<
= 2300 ^ y = 700
100
Æ
doorway os(o1 )
Æ
doorway sin(o1 )
x
_
=
v
y
_
=
v
x0 =2400, y0 =600
x
_ = vo os(o0 Æ )
= vo sin(o0 Æ )
y
_
x
= 2300 ^ y = 700
x
Æ
o os(o2 )
Æ
o sin(o2 )
x
_
=
v
y
_
=
v
= 2300 ^ y = 900
Æ
o os(o1 )
Æ
o os(o1 )
x
_
=
v
y
_
=
v
dist(hx; y i; hxdw ; ydw i)
<
100
Æ
doorway os(o1 )
Æ
doorway sin(o1 )
x
_
=
v
y
_
=
v
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Representing a Hybrid System: PHAMs
=
v
=
v
dist(hx; y i; hxdw ; ydw i)
jumpRelation(cm0 ,h2400,800i,h30,80i)
jumpCondition(cm0 ,e1 ,y≥900)
jumpSuccessor(e1 ,cm1 ,75%)
x =2400, y =600
o jumpSuccessor(e1 ,cm2 ,25%)
o jumpRelation(cm1 ,hi,h30,80i)
jumpRelation(cm2 ,hi,h40,76i)
...
0
=
x
_
=
0
v
os(o0
v
sin(o0
100
Æ
doorway os(o0 )
Æ
doorway sin(o0 )
x
_
=
v
y
_
=
v
x
dist(hx; y i; hxdw ; ydw i)
y
_
>
Æ
hallway os(o0 )
Æ
hallway sin(o0 )
x
_
y
_
<
= 2300 ^ y = 700
100
Æ
doorway os(o1 )
Æ
doorway sin(o1 )
x
_
=
v
y
_
=
v
Æ)
Æ)
x
= 2300 ^ y = 700
x
Æ
o os(o2 )
Æ
o sin(o2 )
x
_
=
v
y
_
=
v
= 2300 ^ y = 900
Æ
o os(o1 )
y
_ = vo os(o1 Æ )
x
_
=
v
dist(hx; y i; hxdw ; ydw i)
<
100
Æ
doorway os(o1 )
Æ
doorway sin(o1 )
x
_
=
v
y
_
=
v
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
PHAMs: Time and Continuous Change
Representing time:
¶ Exog = sequence of exogenous events:
init & sequence of clock ticks with spacing dtc
· e→p rule Clock-Rule
if now(to )
then with probability 1.0
event clock-tick(t)
causes now(t) ∧ clip now(to )
The value of variables in control modes:
~ ≡ valuesAt(t0 ,vals
~ 0 ) ∧ now(t)
¸ stateVarVals(vals)
~ ) ∧ vals
~ = vals
~ 0 + (t − t0 )flow
~
∧flow (flow
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
PHAMs: Control Mode Jumps
¹ randomlySampledSuccessorMode(e,cm) ≡ ...
º p→e rule Mode-Jump
while mode(cm) ∧ jumpCondition(cm,cond,edge)
~ ∧ satisfies(vals,cond)
∧ stateVarsVal(vals)
with an average spacing of τ time units
occurs jump(edge)
» e→p rule Jump-Effects
if randomlySampledSuccessorMode(edge,cm)
~ ∧ flowCond(cm,flow
~ ) ∧ now(t)
∧ initialValues(cm,val)
∧ mode(cmold ) ∧ flow(flowold ) ∧ valuesAt(told ,valold )
then with probability 1.0
event jump(edge)
~ ) ∧ valuesAt(transTime,val)
~
causes mode(cm) ∧ flow(flow
∧ clip mode(cmold ) ∧ ...
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Properties of PHAMs
ý “Probably approximately accurate predictions”
¶ Control mode jumps are predicted to occur
with probability p at most δ delayed
· Between two subsequent events the state evolves according to the
continuous flow of the mode
¸ Successor modes are chosen according to the probability distribution over
the transitions
¹ Every sequence of control modes is a branch of the hybrid automaton
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Probabilistic Sampling of Projections
• Problem: size of the hybrid automaton¢
• Solution: make bounded risk inferences for predictions
– Example: Is there an θ chance that the execution of the plan will result in
an execution scenario that satisfies the property p?
ü Probabilistic sampling-based projection
risk
= 5%
τ = .1%
τ = 1%
τ = 5%
θ
1%
1331
⊥
⊥
10%
100
121
392
20%
44
49
78
40%
17
17
22
60%
8
8
9
80%
3
3
3
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. reasoning tasks and transformational planning
2. temporal projection
3. failure diagnosis and revision rules
4. a reasoning example
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
XFRM-ML System
“Did the robot overlook a ball while performing task TSK?”
“Did the robot try to get a ball in its hand while performing task TSK?”
“Does the robot know that the ball is in its hand after performing task TSK?”
Projected Execution Scenario
XFRML System
Task Network
(ACHIEVE (LOC #D23 toilet))
(ACHIEVE-FOR-ALL
( λ (X) (AND (LOC X <0,6>)
(CAT X BALL)))
( λ (X) (LOC X <0,8>)))
Horn Clause
Rulebase
Plan Revision
Rulebase
(GO <0,8>)
(PUT #D25 toilet)
(LOOK-FOR
’(CAT BALL))
CAUSES
CAUSES
(TRACK #D25)
Inference System
Timeline
(LOC B1 <0,6>)
(LOC B1 <0,8>)
(LOC B1 toilet)
(LOC bomb <0,8>)
Execution Scenario
(BEGIN (GO <0,8>)
(END (PUT ...))
(BEGIN (RAIN))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Examples of revision rules
1. IF
a goal is clobbered by a robot action
THEN make sure that the clobbering
subplan is executed necessarily before
the clobbered goal is achieved
2. IF
a goal is not achieved
because the robot overlooked an object
THEN use another sensing routine
for perceiving these objects
3. IF
if a goal is not achieved
because the robot had an ambiguous
object description
THEN achieve the goal for all objects
satisfying the object description
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
XFRM-ML Predicates
For Tasks
(TASK-GOAL tsk goal),
(VALUE exp sopi val),
(RETURNED-VALUE tsk val),
(STATUS tsk sopi stat), ...
For Timelines
(HOLDS s time-spec),
(CAUSES ev s), ...
For Plans
(TASK-CODE-PATH tsk cp),
(TASK-PLAN tsk plan), ...
Tasks × Timelines
(CAUSING-TASK ev tsk)
(BELIEF-AT b sopi)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
XFRML Queries: Examples
• The robot tried to get a ball from location h0, 8i into its hand:
(AND (TASK-GOAL ?TSK (ACHIEVE (LOC ?DES HAND)))
(HOLDS (VISUAL-TRACK ?DES ?OB)
(DURING ?TSK))
(HOLDS (AND (CAT ?OB BALL) (LOC ?OB h0, 8i))
(BEGIN-TASK ?TSK))).
• The robot overlooked a ball at h0, 8i:
(AND (TASK-GOAL ?P-TSK
(PERCEIVE-1 (λ (?X) (AND (CAT ?X BALL)
(LOC ?X h0, 8i)))))
(RETURNED-VALUE ?P-TSK {})
(HOLDS (AND (CAT ?X BALL) (LOC ?X h0, 8i))
(TT ?P-TSK)))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
XFRML Queries: Examples (Cont.)
• After the execution of plan P the robot “knows” it has a ball from location
h0, 8i in its hand.
(AND (TASK-GOAL ?A-TSK (ACHIEVE (LOC ?DES HAND)))
(HOLDS (TRACK ?DES ?OB)
(DURING ?A-TSK))
(HOLDS (AND (CAT ?OB BALL) (LOC ?OB h0, 8i))
(BEGIN-TASK TOP))
(BELIEF-AT (LOC ?DES HAND)
(END-TASK TOP)))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
1. reasoning tasks and transformational planning
2. temporal projection
3. failure diagnosis
4. a reasoning example
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Execution Scenario
Task Network
(WITH-POLICY ...
...
(:TAG RC ...))
TSK-12 =
(ACHIEVE-FOR-ALL
( λ (X) (AND (LOC X <2,5>)
(CAT X BALL)))
( λ (X) (LOC X <2,7>)))
(ACHIEVE-FOR-ALL
( λ (X) (AND (LOC X <5,5>)
(CAT X BALL)))
( λ (X) (LOC X <2,5>)))
(ITER 1)
(PERCEIVE-1
(λ (X)
(AND (CAT X BALL)
(LOC X <5,5>))))
(ITER 2)
(LOOP
(PROCESS SINGLE-DELIVERY
... )
UNTIL (NULL GOAL-VAR-1)
..)
(ITER 1)
(ITER 2)
(ITER 3)
(PERCEIVE-1
(λ (X)
(AND (CAT X BALL)
(LOC X <2,5>))))
(ACHIEVE (( λ(D)
(LOC D <2,5>))
D-1)
(LOOP
(PROCESS SINGLE-DELIVERY
... )
UNTIL (NULL GOAL-VAR-2)
..)
(ACHIEVE (( λ(D)
(LOC D <2,7>))
D-8)
(LOOP
(PROCESS SINGLE-DELIVERY
... )
UNTIL (NULL GOAL-VAR-2)
..)
TSK-153 =
(PICK-UP HAND-1 D-8)
(UNHAND HAND-1)
Timeline
(BEGIN-TASK
TOP*)
EV-88 =
(BEGIN-TASK
(GO <5,5>))
(END-TASK
(UNHAND
HAND-1))
TIMEINSTANT-1
TIMEINSTANT-28
(LOC B-1 <5,5>)
(LOC B-1 <2,5>)
(BEGIN-TASK
(PICK-UP
HAND-1 D-8))
TIMEINSTANT-37
(LOC B-1 HAND-1)
(BEGIN-TASK
TOP*)
(BEGIN-TASK
(GO <5,5>))
TIMEINSTANT-103
(LOC B-1 <2,7>)
(LOC B-2 <5,5>)
(LOC B-2 <2,7>)
(LOC B-3 <2,5>)
(LOC B-3 <2,7>)
(CAT B-1 BALL)
(CAT B-2 BALL)
(CAT B-3 BALL)
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Failure Detection
•
Definition of Behavior Flaws:
(← (BEHAVIOR-FLAWS ?TSK ?BUGS)
(ABSTRACT-TASK-PLAN ?TSK (ACHIEVE-FOR-ALL ?DESC ?GOAL))
(SETOF ?BUG
(AND (HOLDS (?DESC ?OB) (BEGIN-TASK ?TSK))
(THNOT (HOLDS (?GOAL-EXP ?OB) (END-TASK ?TSK))))
(REPRESENTS-FLAW
?BUG (THNOT (HOLDS (?GOAL ?OB) (END-TASK ?TSK))) ?TSK))
?BUGS))
•
Query:
(?- (BEHAVIOR-FLAWS TSK-12 ?BUGS) ())
•
Result of the Query:
?BUGS = { BUG-1
(THNOT (HOLDS ((λ (X) (LOC X h2, 5i)) BALL-1)
(END-TASK TSK-12)))
TSK-12
BUG-2
(THNOT (HOLDS ((λ (X) (LOC X h2, 5i)) BALL-2)
(END-TASK TSK-12)))
TSK-12 }
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Failure Diagnosis
Task Interference
Characteristic Condition
(AND (REPRESENTS-FLAW ?BUG ?FLAW ?TSK)
(ABSTRACT-RPL-EXP ?TSK
(ACHIEVE-FOR-ALL ?DESCR-EXP ?GOAL-EXP))
(== ?FLAW
(THNOT
(HOLDS ?UNACHIEVED-GOAL
(END-TASK ?TSK-2))))
(HOLDS ?UNACHIEVED-GOAL (DURING ?TSK))
(CLOBBERS-BEFORE
?CE ?UNACHIEVED-GOAL (END-TASK ?TSK-2))
(CAUSING-TASK ?CE ?VIOLATOR)
Subplan to be revised
(?-VAR ?CP (TASK-CODE-PATH ?TSK ?CP))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Failure Diagnosis
(?- Characteristic Condition (Task Interference)
((?BUG BUG-1)))
=⇒ ?CE
?TSK
?VIOLATOR
...
= EV-88
= TSK-12
= TSK-153
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Plan Revision
WITH ?CE
?TSK
?VIOLATOR
...
(:TAG RC
(PLAN
((:TAG MAIN
(TOP-LEVEL
(:TAG COMMAND-1
(ACHIEVE-FOR-ALL
( λ (X) (AND (LOC X <5,5>)
(CAT X BALL)))
( λ (X) (LOC X <2,5>)))
(:TAG COMMAND-2
(ACHIEVE-FOR-ALL
( λ (X) (AND (LOC X <2,5>)
(CAT X BALL)))
( λ (X) (LOC X <2,7>)))
= EV-88
= TSK-12
= TSK-153
(:TAG RC
(PLAN
((:TAG MAIN
(TOP-LEVEL
(:TAG COMMAND-1
(:TAG PROTECTEE-0
Apply Transformation Rule
Order-Conflicting-Subplans
(:TAG COMMAND-2
(:ORDER VIOLATOR-1 PROTECTEE
(ACHIEVE-FOR-ALL
( λ (X) (AND (LOC X <5,5>)
(CAT X BALL)))
( λ (X) (LOC X <2,5>)))
(ACHIEVE-FOR-ALL
( λ (X) (AND (LOC X <2,5>)
(CAT X BALL)))
( λ (X) (LOC X <2,7>)))
(:TAG PROTECTEE-0
(ACHIEVE-FOR-ALL
( λ(X) (AND (LOC X <5,5>)
(CAT X BALL)))
( λ(X) (LOC X <2,5>)))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
(LOOP
Summer Term
2012
(LOOP
Technical Cognitive Systems
(LOOP
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
The Example Revisited: Revision 1
(:TAG RC
(PERFORM (PARTIAL-ORDER ((:TAG MAIN α)))
BY (PARTIAL-ORDER
((:TAG MAIN
(TOP-LEVEL
(:TAG COMMAND-1
(:TAG PROTECTEE-0
(REDUCE (ACHIEVE-FOR-ALL
’(λ (X) (AND (LOC X h8, 8i)
(CATEGORY X BALL)))
’(λ (X) (LOC X h0, 10i)))
β)))
(:TAG COMMAND-2
(REDUCE (ACHIEVE-FOR-ALL
’(λ (X) (AND (LOC X h0, 10i)
(CATEGORY X BALL)))
’(λ (X) (LOC X h0, 12i)))
(:TAG VIOLATOR-1
(LOOP γ
(PICK-UP ***)
δ))))))
(:ORDER VIOLATOR-1 PROTECTEE-0)))
BECAUSE TASK-INTERFERENCE.REV-1))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Discussion
• Criteria:
– necessity
– expressiveness
• classical planning operations
(means-end, SNLP)
• revisions of transformational planning systems (Hacker, Chef, but not:
Gordius)
• applicable to other reactive plan languages like RAP (Firby) or ESL (Gat)
– conciseness
– computational overhead
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Conclusion
• Gaining expressive power through modeling
– the physical behavior
AND the process of plan interpretation
AND temporal, causal, and teleological
relationships between
• the world,
• plan interpretation, and
• the robot’s behavior
• Additional expressive power allows for
– plan revisions for failures caused by
•
•
•
•
sensing routines that overlook or misperceive objects,
unfaithful tracking of objects,
unreliable physical actions, or
interference between concurrent actions.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Outline
Introduction and Motivation
Enabling Technologies
Cognition-enabled Robots
Integrated Intelligent Systems at IAS
Technical Cognitive Systems
Outline of the Course
Plan-based control
Interface
Layer for Cognitive Robots
Introduction Enabling Technologies Cognition-enabled Robots IAS
Michael Beetz
Summer Term 2012
Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
System Architecture
Controlled process
World process
Robot
Environment
Sensor process
Controlling Process
Plan-based control
Perception
manipulation /
body movement
belief
state /
reasoning
navigation
sensor data
processing
probabilistic
estimation
Cognitive capabilities
Learning
Knowledge
processing /
decision
making
Planning
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Process modules
Control Inputs
Activate
Process
Parameters
Deactivate
Navigation
Process
Module
Designator
Process Status
CurrentPoseFluent
ParticleCloudFluent
Feedback
Fluents
Return Value
Status Signals
• Interface to actual hardware (e.g. through ROS).
• Symbolic parametrization.
• Translate symbolic parameters to actual commands.
• Tight integration of reasoning.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Navigation process module
• Example action designator:
(an action (type navigation) (goal (a location (to reach) (object cup))))
• Translation of action designator to ROS actionlib goal.
• Execution of ROS action.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Navigate process module
Translation of designator
(an action (type navigation) (goal (a location (to reach) (object cup))))
1. Perform validity checks (action designator for navigation).
2. Get the goal location, type location designator.
3. Get a solution for that designator.
4. Generate a ROS action goal from that solution.
(<− ( a c t i o n − d e s i g n a t o r − s o l u t i o n ? d e s i g n a t o r ? r o s − a c t i o n − g o a l )
( action−designator ? designator )
( designator−property ? designator ( type navigation ))
( designator−property ? designator ( goal ? goal−designator ))
( location−designator−solution ? goal−designator ? goal−location )
( ros−navigation−action−goal ? goal−location ? ros−action−goal ))
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Navigation process module
Execution
1. Resolve designator.
2. Send goal to ROS action.
3. Wait until action terminates.
d e f −p r o c e s s −module N a v i g a t i o n ( a c t i o n D e s i g n a t o r ) :
actionGoal = resolve ( actionDesignator )
i f a c t i o n G o a l not v a l i d :
n a v i g a t i o n −e r r o r ( ” U n a b l e t o r e s o l v e a c t i o n d e s i g n a t o r
m o v e b a s e a c t i o n c l i e n t . sendGoalAndWait ( a c t i o n G o a l )
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Manipulation process module
• Example designator:
(an action (type manipulation) (to grasp) (object cup))
• Translation of action designator to sequence of cartesian goal poses for the
arm:
– Calculate grasp pose (grasp planning).
– Calculate approach pose.
– Verify that poses are reachable.
• Creation of arm trajectory.
• Execution of arm trajectory.
• Control of gripper.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Manipulation process module
Designator resolution
1. Perform validity checks (action designator for manipulation).
2. Get the object designator of the object to grasp.
3. Call grasp planner.
4. Until a valid grasp has been found, iterate over grasps.
5. Calculate a pre-grasp position for the grasp.
6. Verify that it is reachable.
7. Calculate the grasp position.
8. Verify that it is reachable.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Manipulation process module
Designator resolution
(<− ( a c t i o n − d e s i g n a t o r − s o l u t i o n ? d e s i g n a t o r
( g r a s p ? pre−grasp−pose ? grasp−pose ) )
( action−designator ? designator )
( designator−property ? designator ( type manipulation ))
( designator−property ? d e s i g n a t o r ( to grasp ))
( designator−property ? designator ( object ? obj ))
( g r a s p s ? o b j ? o b j − g r a s p s ) ; ; −> c a l l s g r a s p p l a n n e r
( member ? o b j − g r a s p − p o s e ? o b j − g r a s p s ) ; ; i t e r a t e o v e r g r a s p s
( pre−grasp−pose ? o b j ? obj−grasp−pose ? pre−grasp−pose )
( r e a c h a b l e ? pre−grasp−pose )
( grasp−pose ? obj−grasp−pose ? grasp−pose )
( r e a c h a b l e ? grasp−pose ) )
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Manipulation process module
Execution
1. Resolve designator.
2. Execute code for action (e.g. grasp):
2.1
2.2
2.3
2.4
Open gripper.
Move to pre-grasp position.
Move to grasp position.
Close gripper.
d e f −p r o c e s s −module M a n i p u l a t i o n ( a c t i o n D e s i g n a t o r ) :
action , parameters = r e s o l v e ( actionDesignator )
...
i f a c t i o n == ’ g r a s p ’ :
openGripper ()
preGraspPose , graspPose = parameters
moveToPose ( p r e G r a s p P o s e )
moveToPose ( g r a s p P o s e )
closeGripper ()
i f gripperOpeningDistance () < 0.01:
m a n i p u l a t i o n −e r r o r ( ” g r a s p f a i l e d ” )
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Implementation of pick-up plan
Naive implementation.
( d e f −g o a l ( a c h i e v e ( o b j e c t −i n −hand ? o b j e c t ) )
( w i t h −d e s i g n a t o r s
( ( p i c k −up−a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o g r a s p )
( object ? object )))
( l i f t −a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o l i f t )
( object ? object ))))
( p e r f o r m ? p i c k −up−a c t i o n )
( p e r f o r m ? l i f t −a c t i o n ) ) )
Problems
• Maybe object moved / inaccurate perception.
• Not at the right position.
• No error handling.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Implementation of pick-up plan
Adding perception
( d e f −g o a l ( a c h i e v e ( o b j e c t −i n −hand ? o b j e c t ) )
( w i t h −d e s i g n a t o r s
( ( p i c k −up−a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o g r a s p )
( object ? object )))
( l i f t −a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o l i f t )
( object ? object ))))
(perceive-object ?object)
( p e r f o r m ? p i c k −up−a c t i o n )
( p e r f o r m ? l i f t −a c t i o n ) ) )
Problems
• Maybe object moved / inaccurate perception.
• Not at the right position.
• No error handling.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Implementation of pick-up plan
Executing the action at the right location
( d e f −g o a l ( a c h i e v e ( o b j e c t −i n −hand ? o b j e c t ) )
( w i t h −d e s i g n a t o r s
( ( p i c k −up−a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o g r a s p )
( object ? object )))
( l i f t −a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o l i f t )
( object ? object )))
(location (a location (to execute)(action pick-up-action)(action lift-action))) )
( p e r c e i v e −o b j e c t ? o b j e c t )
(at-location (location)
( p e r f o r m ? p i c k −up−a c t i o n )
( p e r f o r m ? l i f t −a c t i o n ) ) ) )
Problems
• Maybe object moved / inaccurate perception.
• Robot is not at the right location.
• No error handling.
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems
Department of Informatics
Intelligent Autonomous Systems
Technische Universität München
Implementation of pick-up plan
Error handling
( d e f −g o a l ( a c h i e v e ( o b j e c t −i n −hand ? o b j e c t ) )
( w i t h −d e s i g n a t o r s
( ( p i c k −up−a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o g r a s p )
( object ? object )))
( l i f t −a c t i o n ( an a c t i o n ( t y p e m a n i p u l a t i o n ) ( t o l i f t )
( object ? object )))
( l o c a t i o n ( a l o c a t i o n ( to execute )
( a c t i o n p i c k −up−a c t i o n )
( a c t i o n l i f t −a c t i o n ) ) ) )
( p e r c e i v e −o b j e c t ? o b j e c t )
( with-failure-handling
( n a v i g a t i o n −f a i l e d ( e r r o r )
( r e t r y −w i t h −new−l o c a t i o n ) )
( o b j e c t −not−r e a c h a b l e ( e r r o r )
( r e t r y −w i t h −new−l o c a t i o n ) ) )
( at−l o c a t i o n ( l o c a t i o n )
( p e r f o r m ? p i c k −up−a c t i o n )
( p e r f o r m ? l i f t −a c t i o n ) ) ) )
Introduction Enabling Technologies Cognition-enabled Robots IAS Technical Cognitive Systems Course Outline Plan-based control Interface Layer for
Michael Beetz
Summer Term 2012
Technical Cognitive Systems