Challenges in Virtual Reality
Steve LaValle (UIUC), Ming Lin (UNC), & Paul MacNeilage (LMU Munich)
Workshop support provided by Facebook/Oculus
May 30, 2015
S. LaValle
ICRA VR Workshop 2015 – 1 / 22
We Animals
Virtual Reality: Inducing targeted, interactive behavior in an organism by
using artificial sensory stimulation, while the organism has little or no
awareness of the inferference.
S. LaValle
ICRA VR Workshop 2015 – 2 / 22
How “Real” Should VR Be?
It depends on what you want to do...
Take a university course
Long-distance relationship
Play a first-person shooter game
Virtual travel
Watch a live theater performance
Write software
However, it is important to maintain a baseline:
Comfort, safety, belief (immersion, presence)
S. LaValle
ICRA VR Workshop 2015 – 3 / 22
Some History
1968: Ivan Sutherland, Sword of Damocles
1980s: Jaron Lanier, VPL Research, goggles and gloves
1992: University of Illinois, CAVE
1992: Lawnmower Man
1995: Nintendo Virtual Boy
Virtual reality became one of the most over-hyped technologies ever.
Problem: The component technology was not ready yet.
S. LaValle
ICRA VR Workshop 2015 – 4 / 22
My Oculus Experience
Timeline:
S. LaValle
07/12 Oculus VR founded by Palmer Luckey, 19
08/12 Kickstarter, $2.4 million
09/12 I joined as consultant for software: math, head tracking
03/13 Development Kit shipped; I joined as full-time head scientist
09/13 Headed positional tracking, sensor fusion
10/13 Started perceptual psychology team
03/14 Facebook acquisition, approx. $2 billion
07/14 Development Kit 2 shipping; I returned to UIUC
ICRA VR Workshop 2015 – 5 / 22
A Growing Ecosystem
S. LaValle
Oculus DK2
Sony Morpheus
Microsoft Hololens
Google Cardboard
Samsung Gear VR
HTV/Valve Vive
ICRA VR Workshop 2015 – 6 / 22
For People With Limited Mobility
90-year-old grandma in virtual Tuscany
S. LaValle
ICRA VR Workshop 2015 – 7 / 22
Socializing in Virtual Spaces
Open Sim, Second Life, ...
S. LaValle
ICRA VR Workshop 2015 – 8 / 22
Architecture and Real Estate
Do you wish your home were bigger?
S. LaValle
ICRA VR Workshop 2015 – 9 / 22
Movies
S. LaValle
A private virtual movie theater
ICRA VR Workshop 2015 – 10 / 22
Panoramas
Pick your favorite street views and have a look around.
S. LaValle
ICRA VR Workshop 2015 – 11 / 22
VR + Robots
Connect omnidirectional cameras to mobile robots, humanoids, quadrotors
S. LaValle
ICRA VR Workshop 2015 – 12 / 22
First-Person Shooter Games
Team Fortress 2, Valve Inc.
S. LaValle
ICRA VR Workshop 2015 – 13 / 22
VR Game Jam
What could you do with an elephant trunk?
S. LaValle
ICRA VR Workshop 2015 – 14 / 22
Thrill Seekers
Virtual amusement park rides!
S. LaValle
ICRA VR Workshop 2015 – 15 / 22
Brilliantly Disturbing
Ever wonder how Louis XVI must have felt?
S. LaValle
ICRA VR Workshop 2015 – 16 / 22
Alternative Body Experience
S. LaValle
ICRA VR Workshop 2015 – 17 / 22
Flying Like in Your Dreams
Zurich University of the Arts
S. LaValle
ICRA VR Workshop 2015 – 18 / 22
How Roboticists Can Contribute
Develop better tracking systems (heads, eyes, hands, bodies)
Telepresence using mobile robots, humanoids, quad rotors
Predictive tracking
Image stabilization
Brain-machine interfaces
Develop and incorporate more sensing technologies
Use motion planning
S. LaValle
Applied computer vision
Experience with IMUs
Expertise in dynamical system and error modeling
Utilizing kinematic constraints
To move telepresence robots
To automate virtual worlds
ICRA VR Workshop 2015 – 19 / 22
Sensation and Perception
VR Problems:
Latency
Brightness
Tearing
Textures
S. LaValle
Tracking jitter
Focus
Screen door
Virtual egomotion
Optical distortion
IPD
Frame rate
Head translation
Adverse effects: disbelief, fatigue, simulator sickness
Complications: Adaptation and variance across people
ICRA VR Workshop 2015 – 20 / 22
The Grand Meta-Challenge
Getting engineers to pay attention to the human sensation and
perception side!
Need to understand:
Eye movements
Vestibular organ and vection
Photoreceptor density, operation
Neural pathways
Human optical system
Design of experiments
Without perceptual training, you cannot evaluate your own VR system.
S. LaValle
ICRA VR Workshop 2015 – 21 / 22
Schedule
Robotics Perspective
8:30-8:40
8:40-9:10
9:10-9:40
9:40-10:10
10:10-10:30
Industry Perspective
10:30-11:00
11:00-11:30
11:30-12:00
12:00-1:30
Perception Perspective
1:30-2:00
2:00-2:30
2:30-3:00
3:00-3:30
3:30-4:00
VR Demo Session
4:00-5:30
S. LaValle
Steve LaValle, UIUC
Ming Lin, UNC Chapel Hill
Allison Okamura, Stanford
Juan D. Tardos, U. Zaragoza
Break
Introduction, overview
Real-time Multimodal In
Augmented Reality with
Real-time visual SLAM
Richard Yao, Oculus/Facebook
David Kasik, Boeing
Abe Bachrach, Skydio
Lunch
Perception is a Lazy Sc
The Visualization Busin
Unlocking the potential
Marty Banks, UC Berkeley
Dana Ballard, UT Austin
Paul MacNeilage, LMU Munich
Break
John Stowers, IMP Vienna
The importance of focu
Why do we look where
Deconstructing self-mo
Reverse Engineering A
Socialize, discuss, have fun!
ICRA VR Workshop 2015 – 22 / 22