Centre for Autonomous Systems (CAS)
Computational Vision and Active Perception Lab (CVAP)
School of Computer Science and Communication (CSC)
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
KTH
EUl 7th Framework Program

Integrated fully operational robotic system for realistic

Useful to ordinary users
environments
•
Fetch and carry operations
•
Simple human-robot interface
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
ISR Objectives
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
People in GRASP
 Danica
Kragic
 Patric
Jensfelt
 Niklas
Bergström
 Jeannette
1
Bohg
postdoc
 Administrator:
EUl 7th Framework Program
Jeanna Ayoubi

PACO+: Perception, action and cognition through learning of
object-action complexes (02-2010)

CoSy - Cognitive Systems for Cognitive Assistants (08-2008)

CogX - Systems that Self-Understand and Self-Extend (082012)

Several national projects:
•
Swedish research council
•
Foundation for strategic resaerch
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Ongoing projects
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Hardware
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Vision system structure
A wide field for attention, recognition in foveated view
Technically by a combination of wide field and foveal cameras
Problem: transfer from wide field to foveal view
Steps:
•Divide scene into 3D objects
•Select objects through attention (hue and 3D size)
•Fixate (and track) object of interest
•Recognize objects in foveal view
EUl 7th Framework Program

Many objects of interest static. Harder!

Multiple cues are essential
•
Stereo data exist on along contours
•
Color data exist between contours
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
3D Cues: stereo and motion
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Processes
Where
Segmentation
Knowledge
What
EUl 7th Framework Program
Hypotheses
Adaptation
Attention
Shape and size
Knowledge
Recognition
Region of Interest
Wide field
Left
Foveal
Right
Left
Calibration
Registration
Segmentation
Global hue
SIFT features
Fixation
Local hue
Recognition
Gaze direction
EUl 7th Framework Program
Right
Registration
Attention
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Flow of information
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Grasp learning
EUl 7th Framework Program
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Relations on Primitives
Co-planarity
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
PACO+: Elementary Grasping Actions
EUl 7th Framework Program
Co-linearity
Co-colority
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Examples
EUl 7th Framework Program
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Mapping Human to Robot Grasps

Direct mapping from human to
robot hands

Maping fingertip positions to
joint angles (Barret, Robonaut,
DLR hand)

Grasping of cylinders and
polyhedral objects

Object shape parameters
included in the mapping
(radius, width, depth)
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Corrective movements
EUl 7th Framework Program
CoSy:
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Cognitive Systems for Cognitive Assistants

7 partners

2 demonstrators
•
PlayMate (structured
environment)
Interaction with objects on a
table top, continuous crossmodal learning, situated
dialogue
•
Explorer (unstructured
environment)
Interactive modeling space,
talking and reasoning about
space
EUl 7th Framework Program
•
•
Imagine a cognitive system that models not only the environment, but its own
understanding of the environment and how this understanding changes under action. It
identifies gaps in its own understanding and then plans how to fill those gaps so as to deal
with novelty and uncertainty in task execution and to extend its abilities and knowledge so
as to perform future tasks more efficiently.
We aim to develop: a unified theory of self-understanding and self-extension with a
convincing instantiation and implementation of this theory in a robot
Reflection
Planning
Extension
Explanation
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
CogX: Systems that Self-Understand and Self-Extend
Exploration
Experience
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
Role in GRASP
 WP
leader for WP2: Representations and ontology
for learning and abstraction of objects
 Also
•
•
•
•
•
•
in:
WP1:
WP3:
WP4:
WP5:
WP6:
WP7:
EUl 7th Framework Program
What to represent (from humans)?
What to represent (control systems oriented)?
What to represent (visual cues)?
Vision system development
Input to the simulator
Integration and applications
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
THE END
EUl 7th Framework Program
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise
EUl 7th Framework Program
GRASP
Emergence of Cognitive Grasping
through Emulation, Introspection and Surprise