Napoli, December 2006
Alex Zelinsky
motivation
our approach
Smart Cars: An Ideal Applications Platform for
Robotics and Vision Technologies
technology
co-pilot
conclusions
out take!
acknowledgments
Alexander Zelinsky
CSIRO ICT Centre,
Canberra, Australia
[email protected]
Motivation: Motor Car Accidents
Napoli, December 2006
Alex Zelinsky
motivation
motivation
our approach
technology
Car accidents are horrific.
WHO estimates that every year on roads
•
750,000+ people are killed
•
25+ million of people are injured
co-pilot
conclusions
out take!
acknowledgments
Road safety success stories:
• Set belts.
• Speed cameras
• Alcohol breath testing
Road safety hard cases:
•
•
Fatigue (worst accidents)
78% of all accidents involve
inattention
(100 car survey, Neale et.al. 2005).
Alcohol
Fatigue
….and the inattention problem will become more serious...
We are beginning to full exploit the IT revolution!
Modern technologies can Distract
Napoli, December 2006
Alex Zelinsky
motivation
motivation
our approach
technology
Awash with new in-vehicle technologies.
•
Mobile devices:
phones, organizers,
DVD players, iPods etc.
•
Dangerous driver ‘aids’:
Some route guidance,
web browsers, gadgets etc.
co-pilot
conclusions
out take!
acknowledgments
• There is little doubt recent in-vehicle
devices increase crash risk
(Stutts et.al. 2001).
• Mobile phone users up to 4 times more likely to crash.
Hands-free no safer (Redelmeiser et.al.1997).
Technology to Assist Driving
Napoli, December 2006
Alex Zelinsky
motivation
motivation
Driver Assistance Systems (DAS)
our approach
•
Combine the strengths of machines & humans
and mitigate their short comings.
•
Intuitive, Override-able and Unobtrusive.
•
Integrated think of power steering
technology
co-pilot
conclusions
out take!
acknowledgments
Problem: 90% of all accidents are caused by human error.
Who hasn’t had a near miss while driving?
For Driver Assistance Systems to be effective:
•
Must keep Driver engaged.
•
Must consider a whole system approach to road safety.
The driver and vehicle together in context of road environment.
Solution: Directly monitoring driver, vehicle and road scene.
An automated co–pilot!
An Automated Co-Pilot
Napoli, December 2006
Alex Zelinsky
motivation
our
ourapproach
approach
technology
co-pilot
conclusions
out take!
clippit for cars?
acknowledgments
TREV: Transport Research Experimental Vehicle
Co-pilot:
Driver assistance context:
• Do the tedious jobs.
• e.g. ACC, lane keeping.
• Share cognitive workload.
• e.g. Workload managers.
• Warn of unexpected threat.
• e.g. Blind spot, missed sign warning
• Validate Pilot’s decisions.
• If important raise alarm.
•..can take control in emergency.
•..precrash / autonomy.
Safety technologies that we need
Napoli, December 2006
Alex Zelinsky
Lane Departure
Road Sign Recognition
Road Hazard Detection
Pedestrian Detection
Obstacle Detection
Vehicle Detection
motivation
motivation
our approach
technology
co-pilot
Blind-spot Sensor
Parking Sensor
Vehicle Detection
Side View Monitor
Occupant Sensor
Adjacent Lane Monitor
conclusions
out take!
acknowledgments
Active Safety Systems
Driver State Sensor • Smart Airbags
•
•
•
•
Head Pose
Gaze Direction
Eye Closure
Blink Detection
•
•
•
•
•
Advanced Driver Assistance
Systems
•
Collision Avoidance (Front, Side, Rear)
•
Pedestrian Collision
•
Lane Keeping
•
Lane Changing
•
Intelligent ACC
•
Night Vision HUD
Distraction / Inattention Mitigation
Fatigue Detection
Workload Management
Driver Identification
Comfort & Parking Assistance
Monitoring driver, vehicle & road
Napoli, December 2006
Alex Zelinsky
motivation
our
ourapproach
approach
Gaze
Driver
Environment
technology
Driver
observation
monitoring
co-pilot
conclusions
out take!
acknowledgments
Pedals
Steering
Driver
action
monitoring
Driver
Assistance
System
Vehicle
Vehicle
dynamics
DAS Components
Napoli, December 2006
Alex Zelinsky
motivation
our
ourapproach
approach
technology
co-pilot
Linking road scene state
with vehicle and driver
state we can create new
DAS applications.
conclusions
out take!
acknowledgments
Driver Assistance Systems
(DAS) needs to use a
distributed multi-functional
approach. Context is
created by “where” and
“what” the driver is looking.
DAS Components
Napoli, December 2006
Alex Zelinsky
motivation
DAS Logic:
- pedestrians
- obstacles
- lane departure
- signs
- inattention
- monotony
our
ourapproach
approach
technology
co-pilot
conclusions
out take!
acknowledgments
Pedestrian detector
Driver
State
Obstacle detector
Vehicle
state
Lane tracking
Sign reading
Inattention detector
Monotony detector
Head pose & Eye gaze tracking
Napoli, December 2006
Alex Zelinsky
motivation
our approach
technology
technology
co-pilot
conclusions
Seeing Machines:
•
•
•
•
•
Canberra based ANU start-up
Founded July 2000
IPO Dec 2006
32 Employees
Revenues > $4m pa
out take!
acknowledgments
FaceLAB:
• Head Pose measurement
•
•
•
•
•
•
- translation: ± 1mm rotation: ±1°
Eye Gaze measurement
- direction: ±3°
Blink detection
.
Saccade detection
Full 3D modelling
Integrated with a scene camera
FaceLAB v4
ANU 1998
Multi-cue Lane Tracking
Napoli, December 2006
Alex Zelinsky
Adaptive cue fusion
motivation
Particle filter state estimation
our approach
technology
technology
co-pilot
conclusions
Road scene
Video input
Vehicle State
Speed, heading
out take!
acknowledgments
Apply generic lane model
Compute cue probabilities
Cue computation
Edges, lane markings
and road colour
P ( h | e) =
P (e | h ) P ( h )
P (e)
Road Sign Detection
Napoli, December 2006
Alex Zelinsky
motivation
our approach
technology
technology
co-pilot
conclusions
out take!
acknowledgments
Detection:
•
Fast Symmetry Transform - radial
•
Used for eye detection developed in faceLAB
Classification:
•
Image cropping & scaling
•
Online image enhancement – super resolution
•
Image correlation - ZMCC
Pedestrian Detection
Napoli, December 2006
Alex Zelinsky
motivation
Detection:
our approach
Reconstruct scene in 3D:
- Stereo Vision
technology
technology
- Ground plane detection
co-pilot A 3D approach to the problem
- 3D depth map construction
conclusions
- Look for rectangular objects
out take!
- height [1m - 2m ]
acknowledgments
Classification:
Use an implicit shape based
classification approach
Support Vector Machines,
as suggested by Poggio et al.
Temporal information improves
results
85.3% Detection rates
0.4% False detection rates
At ranges up to 20 metres
Multi-task Co-pilot
Napoli, December 2006
Alex Zelinsky
motivation
our approach
technology
co-pilot
co-pilot
conclusions
out take!
acknowledgments
Visual Monotony Detection
Napoli, December 2006
Alex Zelinsky
MPEG compression ratio is used
as a metric of visual monotony.
motivation
our approach
technology
technology
•
Correlates well with human judged
monotony.
•
Augment this function with lane tracking
for ambiguous cases (fog, gravel roads).
•
Validated on 1600km trip.
co-pilot
conclusions
out take!
acknowledgments
MPEG size
Fatigue model [Thiffault 03]
Internal factors
• Time of day
• Sleepiness
• Time on task
• Stimulants
Monotony vs MPEG
External factors
• Task complexity
• Task monotony:
- task variance
- Distractions
Alertness
Vigilance
Driving
performance
gravel
road
Monotony
detector
Gaze and Road-Scene Correlation
Napoli, December 2006
Alex Zelinsky
motivation
Correlating Gaze with Road Scene:
our approach
•
not trivial.
technology
•
stereo camera problem.
•
unknown object depth.
co-pilot
co-pilot
object
yobj
z
40
camera y
conclusions
out take!
Unknown object depth:
acknowledgments
•
bounded disparity along
epipolar line.
•
use upper bound on
disparity in error margin:
upper bound (±2°)
xobj
φ
zsign
x θ
eye
zgaze
ygaze
•
faceLAB precision (±1°)
•
FOV of fovea (±2.6°)
xgaze
⇒ Find difference between gaze and sign angle.
⇒ IF (difference > 2+3+2.6) THEN sign not seen.
Gaze and Road-Scene Correlation
Napoli, December 2006
Alex Zelinsky
motivation
our approach
Object
recognized
technology
co-pilot
co-pilot
conclusions
out take!
acknowledgments
Verification:
•
3 distances, 8 objects, 10 trials
•
Prove that driver saw the object. (impossible!)
•
Prove driver would not have recognized the object.
Object
missed
Integrated Systems
motivation
our approach
technology
co-pilot
conclusions
out take!
acknowledgments
Napoli, December 2006
Alex Zelinsky
Driver Monitoring
Napoli, December 2006
Alex Zelinsky
motivation
our approach
technology
co-pilot
conclusions
out take!
acknowledgments
Comparison between the driver’s gaze and the yaw of the vehicle
Conclusions
Napoli, December 2006
Alex Zelinsky
motivation
our approach
technology
co-pilot
conclusions
conclusions
out take!
acknowledgments
Mobile technologies
are flooding into cars.
We must only adopt
technologies that make
driving safer.
We have demonstrated driver assistive technology:
• Modelled on a human co-pilot used in airlines.
• Able to monitor
•the driver’s observations as well as actions.
Enabling:
• Redundant / repetitive warnings to be suppressed.
• Important warnings to be highlighted.
• Warnings acknowledged at a glance.
OUR GOAL
To combine the tireless vigilance of machines with the flexibility and
authority of the human driver.
Autonomous Driving!
Napoli, December 2006
Alex Zelinsky
motivation
our approach
technology
co-pilot
conclusions
outtake!
take!
out
acknowledgments
Alex Zelinsky, June 2004
Acknowledgments
Napoli, December 2006
Alex Zelinsky
motivation
our approach
technology
co-pilot
conclusions
out take!
acknowledgments
•
•
•
•
Luke Fletcher
Andrew Dankers
Luke Cole
Gareth Loy
• Jochen Heinzmann
• Rochelle O’Hagan
• Tim Edwards
• Nick Apostoloff
• Leanne Matuszyk
• James Ashton
•
•
•
•
• Rowel Artienza
Harley Truong
Sebastien Rougeaux
Jason Chen
Yoshio Matsumoto
• Simon Thompson
• Peter Brown
• Lars Petersson