Research Background:
Depth Exam Presentation
Susan Kolakowski
Committee:
Juan Cockburn, Chair
Jeff Pelz, Adviser
Andrew Herbert
Mitchell Rosen
Carl Salvaggio
March 20, 2006
Research Background
•
•
•
•
Introduction
Human Visual System
Eye Movements
Eye Trackers
– RIT Wearable Eye Tracker
• My Research
Introduction
• Why are eye trackers used?
– Objective measure of where people look
– Interest in Human Visual System
• Examples:
– Understanding Behaviors: How do humans read?
– Improving Skill: Train people to move their eyes as
an expert would.
– Improving Quality: What parts of an image are
important to the image’s overall quality?
The Human Eye
Iris
Pupil
Cornea
Ciliary Muscle
Retina
Eyelens
Optic Axis
Optic Nerve
Fovea
Human Visual System
• What we see is determined by
– How the photoreceptors in our retina are
connected and distributed
– How our brain processes this information
– What we already accept as truth (previous
knowledge)
– How we move our eyes throughout a scene
The Retina
• Contains two types of photoreceptors
– Rods that offer wide field of view (and night vision)
– Cones that provide high acuity (and color vision)
The Craik-O’Brien Illusion
Lateral Inhibition
Affect of Previous Knowledge
Rotating Mask
Affect of Previous Knowledge
The Fovea
• At its center: contains only cones (no rods)
• Perceive greatest detail and color vision
– To get the most detailed representation of a
scene, must move your eyes rapidly so that
different areas of the scene fall on your fovea
• Along visual axis - lowest potential for
aberrations
Serial Execution
(fovea covers <0.1% of the field)
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Serial Execution
Eye Movements…
•
•
•
•
Saccades
Smooth Pursuit
Optokinesis (OKN)
Vestibular-Ocular Reflex (VOR)
… and lack thereof
• Fixations
Fixations
• Stabilizations of the eye for higher acuity
at a given point
• Drifts and tremors of the eye occur during
fixations such that the view is always
changing slightly
Eye Movements
Saccades
• Rapid ballistic movement of eye from one
position to another
• Shift point of gaze such that a new region
falls on the fovea
X
X
X
Eye Movements
Smooth Pursuit
• Smooth eye movement to track a moving
target
• Involuntary - can’t be produced without a
moving object
X
X
Eye Movements
Optokinesis
• Invoked to stabilize an image on the retina
• Eye rotates with large object or with its
field-of-view
Vestibular-Ocular Reflex
• Invoked to stabilize an image on the retina
• Stabilizes an image as the head or body
moves relative to the image
X
Eye Trackers
• Invasive
– Painful devices which discomfort subject’s eye
• Restrictive
– Devices that require strict stabilization of
subject’s head, not allowing for natural
movement
• Modern Video-Based Trackers
– Remote - constrained to 2D stimuli
– Head-mounted - allows natural movement
Intrusive Eye Trackers
• Delabarre 1898
• Yarbus 1965
Mechanical stalk
Intrusive Eye Trackers
• Robinson 1963, Search Coils
Video-based Eye Trackers
• Early 1970’s, Limbus
Video-based Eye Trackers
• Cornsweet and Crane 1973, Dual Purkinje
Video-based Eye Trackers
Early 1970’s
• Dark-Pupil
• Bright Pupil
Video-based Eye Trackers
• Head-Mounted
• Remote
Video-based Eye Trackers
R.I.T. Wearable Eye Tracker
SCENE CAMERA
IR LED
EYE CAMERA
R.I.T. Wearable Eye Tracker
How it works
• Off-axis illumination
• Off-line processing
Example Video
My Research
• Objective: Improve the performance of
video-based eye trackers in the
processing stage.
– Compensate for camera movement with
respect to the subject’s head
– Reduce noise
R.I.T. Wearable Eye Tracker
• Advantage:
– Subject is less
constrained, can
perform more natural
tasks
• Disadvantage:
– Camera (eye tracker)
not stabilized - need to
account for any
movement of camera
relative to head
LOWER PRECISION
Analysis of Disadvantages
Lower Precision
• Need to account for movement of camera
with respect to the head requires
additional data: corneal reflection
• Corneal Reflection data is not as precise
as Pupil data.
Too bad we can’t just use the Pupil data
Analysis of Disadvantages
Oversimplifying Assumption
• Assumption: When the camera moves
with respect to the head, the pupil and
corneal reflection move the same amount.
• To account for camera movement:
P  CR
Why this assumption is wrong
• Corneal Reflection data comes from the
center of the reflection off the curved outer
surface of the eye
• Pupil data comes from the center of the
flat virtual image of the pupil inside the
eye.
DON’T MOVE THE SAME AMOUNT
WHEN THE CAMERA MOVES
Result of Oversimplification
• P-CR vector difference changes with
camera movement
– Artifacts in final data
X
X
X
The Solution
• Determine the actual relationship between
the pupil and corneal reflection during
BOTH:
– Camera movements
– Eye movements
• Use these relationships to develop a new
equation in terms of pupil and corneal
reflection position
Eye Movements
Camera Movements
Camera and Eye Gains
• Eye Gain: amount corneal reflection moves
when pupil moves 1 degree during an eye
movement
CR 
eye _ gain  

 P eye
• Camera Gain: amount corneal reflection moves
 pupil moves 1 degree during a camera
when
movement
CR 
cam _ gain  

 P camera
The Equations
4 Initial Equations
(1) Ptrack  Pcam  Peye
(2) CRtrack  CRcam  CReye
CReye
(3) E 
Peye
CRcam
(4) C 
Pcam
4 Unknowns: Peye , Pcam , CReye , CRcam
Pcam Ptrack  Peye

Pcam
 E  Ptrack  E  CReye


Pcam  E  CRcam  Ptrack  E  CReye  CRcam

Pcam (E  C)  Ptrack  E  CRtrack
Ptrack  E  CRtrack
Pcam 
E C
Added Benefit
• Can smooth Camera array without loss of
information from Pupil array:
Peye  Ptrack  Pcam
• Assuming camera moves smoothly
• Result is on same level as Pupil only data

Added Benefit
• Can smooth Camera array without loss of
information from Pupil array:
• Assuming camera moves smoothly
• Result is on same level as Pupil only data
Determining the Gains
• Eye Gain: (Instruct subject to…)
– Look at center of field-of-view.
– Keep camera and head perfectly still.
– Look through calibration points.
• Cam Gain: (Instruct subject to…)
– Look at center of field-of-view.
– Keep eye fixated while moving the camera on
nose.
Eye Gain Results
Eye Gain Results
Eye Gain Results
y = 0.5161x + 0.3322
R2 = 0.9878
Camera Gain Results
Camera Gain Results
Camera Gain Results
y = 0.8143x + 4.5981
R2 = 0.9768
Camera Gain Results
y = 0.8143x + 4.5981
R2 = 0.9768
Camera Gain Results
slope
average
gain = 0.8524
y ==0.8143x
+ 4.5981
of 5 subjects
2
R = 0.9768
Testing the Algorithm
• Collect data:
– 5 subjects look through 9 calibration points
while moving the eye tracker’s headgear
• Extract eye movements:
– Use average gains to calculate Camera array
– Smoothed Camera array
– Subtracted smoothed Camera array from
Pupil array
Eye array
Results
Horizontal Results
X
X
X
X
X
X
X
X
X
Results Continued
Horizontal Results
X
X
X
X
X
X
X
X
X
Results Continued
Horizontal Results
X
X
X
X
X
X
X
X
X
Results Continued
Vertical Results
X
X
X
X
X
X
X
X
X
Results Continued
Vertical Results
X
X
X
X
X
X
X
X
X
Results Continued
Vertical Results
X
X
X
X
X
X
X
X
X
Results Continued
Vertical Results
X
X
X
X
X
X
X
X
X
Results Continued
Vertical Results
X
X
X
X
X
X
X
X
X
Results Continued
Noise Reduction
Conclusions
• Successful application to head-mounted
video-based eye trackers
– Use same gain values for all subjects
• Final Eye array precision is on the order of
the Pupil array precision
– Noise due to Corneal Reflection data is
reduced
Next Steps
• Calibration - Eye array represents
eye movement in head - need to map
this to the world (via scene camera)
Next Steps
• Calibration - Eye array represents
eye movement in head - need to map
this to the world (via scene camera)
• Investigate realistic camera movements and
alternative smoothing options for Camera array
Next Steps
• Calibration - Eye array represents
eye movement in head - need to map
this to the world (via scene camera)
• Investigate realistic camera movements and
alternative smoothing options for Camera array
• Obtain gain values for larger group of subjects
Next Steps
• Calibration - Eye array represents
eye movement in head - need to map
this to the world (via scene camera)
• Investigate realistic camera movements and
alternative smoothing options for Camera array
• Obtain gain values for larger group of subjects
• Test on larger eye movements
Next Steps
• Calibration - Eye array represents
eye movement in head - need to map
this to the world (via scene camera)
• Investigate realistic camera movements and
alternative smoothing options for Camera array
• Obtain gain values for larger group of subjects
• Test on larger eye movements
• Revision for remote trackers
Questions, Suggestions…