Lecture 11
Vision-based Landing of an
Unmanned Air Vehicle
MASKS © 2004
Invitation to 3D vision
Applications of Vision-based Control
Predator
Global Hawk
SR/71
MASKS © 2004
Invitation to 3D vision
UCAV X-45
Fire Scout
Goal: Autonomous landing on a ship deck
Challenges
• Hostile environments
 Ground effect
 Pitching deck
 High winds, etc
Why vision?
• Passive sensor
• Observes relative motion
MASKS © 2004
Invitation to 3D vision
Simulation: Vision in the loop
MASKS © 2004
Invitation to 3D vision
Vision-Based Landing of a UAV
•
Motion estimation algorithms


•
Real-time vision system


•
•
•
Linear, nonlinear, multiple-view
Error: 5cm translation, 4° rotation
Customized software
Off-the-shelf hardware
Vision in Control Loop
Landing on stationary deck
Tracking of pitching deck
MASKS © 2004
Invitation to 3D vision
Vision-based Motion Estimation
Current pose
Image plane
Feature Points
Pinhole Camera
Landing target
MASKS © 2004
Invitation to 3D vision
Pose Estimation: Linear Optimization
•
•
•
•
•
•
Pinhole Camera:
Epipolar Constraint:
Planar constraint:
More than 4 feature points
Solve linearly for
Project
onto
MASKS © 2004
Invitation to 3D vision
to recover
Pose Estimation: Nonlinear Refinement
•
Objective: minimize error
•
Parameterize rotation by Euler angles
•
Minimize by Newton-Raphson iteration
•
Initialize with linear algorithm
MASKS © 2004
Invitation to 3D vision
Multiple-View Motion Estimation
Pinhole Camera
Multiple View Matrix
Rank deficiency constraint
MASKS © 2004
Invitation to 3D vision
Multiple-View Motion Estimation
•
n points in m views
•
Equivalent to finding
•
Initialize
•
Least squared solution:
•
Use
to linearly solve for
Iterate until
converge
•
MASKS © 2004
with two-view linear solution
Invitation to 3D vision
s.t.
Real-time Vision System
•
•
Ampro embedded Little Board PC
 Pentium 233MHz running LINUX
 440 MB flashdisk HD robust to vibration
 Runs motion estimation algorithm
 Controls Pan/Tilt/Zoom camera
Motion estimation algorithms
 Written and optimized in C++ using LAPACK
 Estimate relative position and orientation at 30 Hz
UAV
MASKS © 2004
Pan/Tilt Camera
Invitation to 3D vision
Onboard Computer
Hardware Configuration
On-board UAV
Vision System
Vision Computer
Camera
RS232
Vision
Algorithm
WaveLAN
to Ground
Frame
Grabber
RS232
Navigation System
Navigation Computer
INS/GPS
WaveLAN
to Ground
MASKS © 2004
Invitation to 3D vision
RS232
RS232
Control &
Navigation
Feature Extraction
•
•
•
•
•
•
Acquire Image
Threshold Histogram
Segmentation
Target Detection
Corner Detection
Correspondence
MASKS © 2004
Invitation to 3D vision
Camera Control
•
Pan/Tilt to keep features in image center
 Prevent features from leaving field of view
 Increased Field of View
 Increased range of motion of UAV
MASKS © 2004
Invitation to 3D vision
Ground Station
Comparing Vision with INS/GPS
MASKS © 2004
Invitation to 3D vision
Motion Estimation in Real Flight Tests
MASKS © 2004
Invitation to 3D vision
Landing on Stationary Target
MASKS © 2004
Invitation to 3D vision
Tracking Pitching Target
MASKS © 2004
Invitation to 3D vision
Conclusions
•
•
Contributions
 Vision-based motion estimation (5cm accuracy)
 Real-time vision system in control loop
 Demonstrated proof of concept prototype:
first vision-based UAV landing
Extensions
 Dynamic vision: Filtering motion estimates
 Symmetry-based motion estimation
 Fixed-wing UAVs: Vision-based landing on runways
 Modeling and prediction of ship deck motion
 Landing gear that grabs ship deck
 Unstructured environments: Recognizing good landing
spots (grassy field, roof top etc)
MASKS © 2004
Invitation to 3D vision