Acquiring Semantic Object Maps for
Household Tasks
Dejan Pangercic
Intelligent Autonomous Systems Group
Technische Universität München
Munich, November 2011
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Outline
1. Semantic Object Maps
2. Texture Blending
3. Perception of Handles
4. Cabinet Opening
5. Conclusions
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
+
Semantic Object Maps (SOM s)?
Structures that enable the robot to answer the following
categories of queries:
◮ “What do parts of the kitchen look like?”;
◮ “How can a container be opened and closed?”;
◮ “What is inside of cupboards/drawers?”;
◮ “Where do objects of daily use belong?”.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
System Overview
F56595D7B7678399E48D2BB731
◮
End-to-end system for
generation of SOM+ s
◮
Low-cost low-quality
sensor
◮
Logic-based formal
language and
background knowledge
for the representation of
SOM+ maps
◮
Detection of the
handles with surfaces
with specular reflectivity
1
23451
6789A1BC7DEF
177A89
6789A1BC7DE1
28FAA879
B4A5D2953F962642942D83B64D6783
678FF79 FE
D
39A879
D1
9AA879
9E
97CE89
AD1
279FADA879
8A14C9E891
E87369D8F9F565973624E42656783953F9254373198A95467D567839C8F2B
2798A879171
6C9F1
9E1$8ADF
B 89A1!"989
9AA8791
71B 89AF
A7D1
#9AA8791
123245678398A9B2C5367D9C5E953F982D69836881
2819A8
&A8DCA8791
#9A1719DC1
%D8F
7ECF
! 'A1&997AA879
5EE7D56783922
19 9!"#$9%! &9$ ! 9#$%
'%( )%99&(!9A&*+
1A8(C1717 'AF171E8C)1DF
198"DCA879171 89AF
17EF7D89 FE17 'A1997AA879
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
*
[ICRA 2012, RAS 2011]
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Outline
1. Semantic Object Maps
2. Texture Blending
3. Perception of Handles
4. Cabinet Opening
5. Conclusions
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Goal 1
⇒
Photo-realistic
visualization
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Goal 2
⇒
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Goal 3
⇒
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Goal 4
⇒
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Reconstruction of Diffuse Texture Maps
◮
◮
◮
◮
Texture Map defined as τ ⊂ R2 and mapped onto the
2-dimensional space.
Texture mapping means the mapping of a texture τ onto a
surface S ⊆ M, where M - triangle mesh.
Mapping function: f : S 7→ τ .
Topology-preserving bijective mappings preferred.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Multi-view Texture Reconstruction
Surface
Segmentation
◮
Surface
Unfolding
Color
Reconstruction
Color Blending
Assuming Lambertian Bidirectional Reflectance
Distribution Function.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Surface Partitioning
◮
Region-growing based segmentation (with normal feature).
◮
Region Adjacency Graphs to correct over-segmentation.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Surface Unfolding
◮
◮
◮
Using conformal mapping which preserves angles.
Transformation of an elementary circle in the texture
domain into an elementary circle on the surface.
Minimization of discrete conformal energy that corresponds
to the “non-conformality” in a least-square sense.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Surface Unfolding 2
◮
Minimization of discrete conformal energy that corresponds
to the “non-conformality” in a least-square sense:
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Color Reconstruction
◮
Estimation of the correct camera image for each point
(excluding occlusions) based on the neighborhood of
points.
◮
Poor resolution.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Mesh Re-parametrization
◮
Re-parametrization of the mesh to obtain densely sampled
surface.
◮
Barycentric coordinates to interpolate the mapping
between the vertices.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Color Interpolation
◮
Minimization of the visibility of discontinuity artifacts.
◮
Guidance vector field V composed of color gradients and
boundary constraints.
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Color Interpolation 2
◮
Membrane interpolant problem:
◮
First boundary contraint - smooth transition.
Second boundary contraint - invariance of gradient
operator for multiplicative factors.
◮
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Results - Texture Reprojection1
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Results - Texture Reprojection2
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Outline
1. Semantic Object Maps
2. Texture Blending
3. Perception of Handles
4. Cabinet Opening
5. Conclusions
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Non-reflective Handles
◮
min distance hd
◮
According to “ADA
Standards for Accessible
Design”
◮
Rusu et al., ICAR2009
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Specular Handles
◮
Kinect sensor
◮
Distorted projected pattern
leads to erroneous
matching and
consequentially to invalid
measurement
◮
IDEA: Lack of missing
data as indicator of the
presence of handles
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
ROI
◮
Detect ROI in 3D using
RANSAC
◮
Create binary masks from
detected ROI and invalid
measurements
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Handle Candidates
◮
Series of dilation and
erosion operations
◮
Perform bit-wise
conjunction between both
masks
◮
Euclidean clustering using
a region growing approach
◮
Keep clusters C that
correspond to the
expected size and height
(according to ADA)
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Pose Estimation
◮
Grasp point Pxyzrpy :
compute convex hull H
around every cluster ci ,
hull’s centroid as YZ in
robot’s base
◮
X component as hd (ADA)
or contact point between
the handle and fingertip
◮
Hough line fitting for RPY
orientation
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Results - Detection of Handle
2 handles, 9 different poses:
56789AB
69A
1234
12C34
D214
12E34 1214
positive
negative
positive
45
5
F12E34
negative
4
0
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Outline
1. Semantic Object Maps
2. Texture Blending
3. Perception of Handles
4. Cabinet Opening
5. Conclusions
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Learning of Articulation Models - Arm Control
observe
gripper
pose yi
◮
Using impedance
controller from Willow
Garage (pr2_cockpit stack)
◮
Initialization: gently pull the
handle backwards by
moving the Cartesian
equilibrium point towards
the robot
◮
Record the trajectory of
robot’s gripper y1:n with
yi ∈ SE(3)
estimate
articulation
model M̂, θ̂
cabinet object
interaction
generate
next control
point xCEP
i
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Learning of Articulation Models - Model Fitting
Iteratively:
◮ (Re-)estimate the
kinematic model M ∈
{rigid, prismatic, rotational}
observe
gripper
pose yi
◮
Estimate model-specific
parameter vector θ ∈ Rd
(encoding radius, rotation
axis) of the articulated
object: M̂, θ̂ =
arg maxM,θ p(M, θ | y1:n )
◮
Fit the parameter vector of
all model candidates using
an MLESAC estimator
estimate
articulation
model M̂, θ̂
cabinet object
interaction
generate
next control
point xCEP
i
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Learning of Articulation Models - Model Selection
observe
gripper
pose yi
Select the best model
according the Bayesian
information criterion (BIC)
◮
Use the model to predict
the continuation of the
trajectory and to generate
the next Cartesian
equilibrium point xCEP
n+1 .
◮
Finally, determine opening
angle / opening distance
◮
Output: Kinematic model
estimate
articulation
model M̂, θ̂
cabinet object
interaction
generate
next control
point xCEP
i
VIDEO:
◮
*
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Articulation Model Representation and Storage
Sample a noise-free trajectory x1:n from the model over the
configuration range.
Store and retrieve the trajectory from KnowRob DB (Tenorth,
IROS 2009):
*
?− r d f _ t r i p l e ( knowrob : ’ i n−ContGeneric ’ , knowrob : ’ Cup67 ’ , B) ,
r d f _ h a s ( B , knowrob : o p e n i n g T r a j e c t o r y , T r a j ) ,
f i n d a l l ( P , r d f _ h a s ( T r a j , knowrob : p o i n t O n T r a j e c t o r y , P) ,
Points ) .
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Test Setup
29 cabinets, 5 kitchens of Bosch and TUM, PR2 Robot, 104
trials
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Results - Accuracy of Estimated Articulation Models
Evaluation of the opening distances and estimated opening
radii:
cabinet type
drawers
doors
average
trials
18
35
53
translational err.
0.664cm
0.516cm
0.567cm
rotational err.
4.96◦
36.2◦
25.6◦
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Results - Whole System
activity
handle detection
operate cabinet and learn model
re-execute learned model
overall performance
trials
104
97
54
104
successes
97
54
54
54
success rate
93.3 %
56.2 %
100 %
51.9 %
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Results - Whole System - Failure Causes
type of failure
(a) handle detection
(b) slippage of gripper
(c) robot too weak at start
(d) robot too weak during motion
(f) overall failures in learning phase
(g) sum (a)+(f)
trials
104
97
97
97
97
104
failed trials
7
6
6
23
43
50
rate
6.7 %
6.2 %
6.2 %
23.7 %
44.3 %
48.1 %
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Outline
1. Semantic Object Maps
2. Texture Blending
3. Perception of Handles
4. Cabinet Opening
5. Conclusions
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Conclusions
Conclusions
◮
End-to-end system that covers all steps required to
automatically reconstruct textured SOM+ models of
kitchens using low-cost Kinect sensor.
◮
Texture reconstruction approach that consists of the
following steps: surface partitioning (segmentation and
slicing), surface unfolding, color reconstruction, and color
blending.
◮
Open source
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Semantic Object Maps
Texture Blending
Perception of Handles
Cabinet Opening
Thanks!
Collaborators: Ben Pitzer, Jürgen Sturm, Thomas Rühr, Martin
Schuster, Moritz Tenorth, Zoltan-Csaba Marton, Michael Beetz
Contact:
{pangercic}@cs.tum.edu
Munich, November 2011
Acquiring Semantic Object Maps for Household Tasks
Dejan Pangercic
Conclusions