Hybrid Visual Servoing with Hierarchical
Task Composition for Aerial Manipulation
Vincenzo Lippiello and Bruno Siciliano
Prisma Lab
Department of Electrical Engineering and Information Technology
University of Naples Federico II
Italy
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Outline
 Introduction
 Dynamic task priority control
 Sketch of the stability proof
 Hybrid vision-based control




End-effector position and orientation
Camera field-of-view
Centre of gravity
Joint-limits avoidance
 Simulation and experimental results
 Conclusion
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Problem description
INTRODUCTION
 Scenario and goals
 Assembling structures with a VToL aerial
vehicle fitted with a robot arm and a
camera mounted on the vehicle base
 Assumptions
 Vehicle and robot arm controlled in velocity
 Constraints




Task in the Cartesian space
Task in the
Accuracy
image space
Camera field of view
Task in the
Stability
configuration
Mechanical limits Task in the
space
joint space
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Main task definition
DYNAMIC TASK PRIORITY CONTROL
 Consider the variables of a configuration-dependant
main task (#0):
 The following differential relationship holds:
Jacobian of the main task for the controlled variables:
FULL
RANK
Jacobian of the main task for the uncontrolled variables:
Controlled velocity:
Pose-configuration state:
Underactuation
of the vehicle base
Uncontrolled angular velocities:
IEEE International Conference on Robotics and Automation
IMU
May 16-21, 2016 – Stockholm, Sweden
Velocity command
DYNAMIC TASK PRIORITY CONTROL
 By inverting the previous equation and by
considering a regulation problem yields
Main task error:
Matrix gain (positive definite):
 It corresponds to an exponentially-stable error
dynamics
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Secondary task definition
DYNAMIC TASK PRIORITY CONTROL
 Redundancy in the system (vehicle + arm), i.e.
,
useful to characterize the system behaviour (internal
motion) through a number of secondary tasks
Variables of a configuration-dependent task #1:
Jacobian of the secondary task for the controlled variables:
Jacobian of the secondary task for the uncontrolled variables:
IEEE International Conference on Robotics and Automation
FULL
RANK
May 16-21, 2016 – Stockholm, Sweden
Task-priority control velocity command
DYNAMIC TASK PRIORITY CONTROL
 New combined velocity command:
Main task
Second task not
affecting the main task
Compensation of the
velocity generated by
the uncontrolled
variables
 New recursive formulation for the Jacobian matrix on
the uncontrolled variables
Projector into the null-space of the main task:
IEEE International Conference on Robotics and Automation
and
May 16-21, 2016 – Stockholm, Sweden
Stability with two tasks (sketch)
DYNAMIC TASK PRIORITY CONTROL

is full-rank only if the two tasks are orthogonal,
i.e.
, or independent, i.e. not
orthogonal and
 By computing the dynamics of the error-system for
the two considered tasks:
HURWITZ
MATRIX
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Generalized task-priority control
DYNAMIC TASK PRIORITY CONTROL
 Generalization to the case of η secondary tasks
Main task
Secondary tasks not affecting the
main task
where
IEEE International Conference on Robotics and Automation
Compensation of the
velocity generated by
the uncontrolled
variables
May 16-21, 2016 – Stockholm, Sweden
Dynamic task-priority control velocity
DYNAMIC TASK PRIORITY CONTROL
 The task composition, i.e. the tasks effectively
selected from the formulated task list, and the
corresponding priority can be changed at runtime
 Dynamic activation or deactivation of tasks:
 Time-dependent exponential smoothing-factor
 The subscript number indicates the index of the
corresponding task stack (a desired composition of tasks at
a given time), which has been activated sequentially
 The error is bounded during the transition
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Hybrid vision-based control
HYBRID VISION-BASED CONTROL
 The dynamic task priority control allows
combining in a unique hybrid-control
framework a number of tasks formulated in
different domains
 Cartesian space: end-effector pose w.r.t. target
pose
 Camera image space: camera FoV
 Vehicle-base configuration space: arm CoG w.r.t.
base CoG
 Arm joint space: mechanical joint limits
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Reference frames and camera model
MODELING
 Main goal is the control of the
reference frame of the robotarm end-effector w.r.t. a target
 Camera mounted on the vehicle
base
 Body frame ≡ Camera frame
 Pinhole camera mode
(Perspective transformation)
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Object pose estimation
PERCEPTION
 A number of known visual markers fixed with the object
 For example: ARTag
 Pose of the target object
 Mean of more measurements
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Object pose estimation
PERCEPTION
Norm of the position error
during grasping
Norm of the orientation
error during grasping
Norm of the position error
during plugging
Norm of the orientation
error during plugging
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Hybrid vision-based control
HYBRID VISION-BASED CONTROL
 Tasks:
 Gripper position and orientation
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Hybrid vision-based control
HYBRID VISION-BASED CONTROL
 Tasks:
 Gripper position and orientation
 Camera FoV
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Hybrid vision-based control
HYBRID VISION-BASED CONTROL
 Tasks:
 Gripper position and orientation
 Camera FoV
 Arm-Base CoG
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Hybrid vision-based control
HYBRID VISION-BASED CONTROL
 Tasks:




Gripper position and orientation
Camera FoV
Arm-Base CoG
Joint limits avoidance
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Simulation result
RESULTS
 GAZEBO simulator




5 kg quadrotor
6 DoFs robot arm
25Hz downward camera
Grasping of a bar
 Case studies:
1.
2.
3.
4.
Position & orientation
1) + FoV
2) + CoG
3) + joint limits
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Simulation result
RESULT
End-effector orientation
Centre of gravity
CoG error in norm
Joint limits distance
rad
End-effector position
IEEE International Conference on Robotics and Automation
Field of view
Case 1) magenta
Case 2) red
Case 3) green
Case 4) blue
(*) Vertical lines indicate
the transition between the
approaching and the
grasping phases
May 16-21, 2016 – Stockholm, Sweden
Experiments
RESULT
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Experiments
RESULT
End-effector position
End-effector orientation
Centre of gravity
Joint limits distance
Field of view
(*) Plots correspond to a
single grasping task
execution
(**) Vertical lines indicate
the transition between the
approaching and the
grasping phases
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Conclusion
MAIN CONTRIBUTIONS
 Position-based visual servoing with a hybrid
hierarchical task composition
 Redundancy in the system (vehicle + arm) useful to
characterize the system behaviour (internal motion)
through a number of secondary tasks
 Vehicle-base under-actuation taken into account with a
new iterative formulation
 New formulation that guarantees decoupling of
independent tasks
 Dynamic smooth task activation
IEEE International Conference on Robotics and Automation
May 16-21, 2016 – Stockholm, Sweden
Hybrid Visual Servoing with Hierarchical
Task Composition for Aerial Manipulation
Thank You for your kind attention!
Vincenzo Lippiello
[email protected]
www.prisma.unina.it
IEEE International Conference on Robotics and Automation
PRISMA Lab
Department of Electrical Engineering
and Information Technology
University of Naples Federico II, Italy
May 16-21, 2016 – Stockholm, Sweden