1
Physical I/O devices
Part 2: haptic output and other
non-visual displays
SGN-5406 Virtual Reality 2012
Atanas Boev
based on material by
Stanislav Stankovic and Ismo Rakkolainen
Department of Signal Processing
SGN-5406 Virtual Reality 2012
2
Outline
Physical I/O devices
Part 1: Input (haptic sensors)
Passive
No
feedback
3D “fly” mouse
Wiimote
PS Move
Kinect
VR Gloves
3D Probes
Accelerometers
Etc…
Movement
support
Keyboard
Mouse
Joystick
Touch screens
Steering wheel
3D “desk” mouse
(SpaceNavigator)
Etc…
Department of Signal Processing
Part 2: Output (non-visual displays)
Vestibular
Haptic
Active
Intentional
feedback
Tactile
Combination of
input+output
Surgery simulator
Novint Falcon
CyberGrasp glove
Virtual keyboard +
force feedback
SGN-5406 Virtual Reality 2012
Kinesthetic
Endeffector
Force
feedback
Olfactory
3
Output
DISPLAYS IN GENERAL
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Output devices
• Present the artificial computer generated stimuli to the user.
• Provide the information about the state of VR environment
to the user.
• Provide feedback about the results of user’s actions.
• Work with human sensory organs:
•
•
•
•
Visual system
Auditory apparatus
Haptic, (tactile, kinesthetic)
Rarely with other senses – olfactory (smell), taste, etc.
Department of Signal Processing
Displays
Display = Output Device
Video display:
•
•
•
•
Computer Screen
Projector
HMD – Head Mounted Display
3D Screens
Audio display:
• Speakers
• Headphones
Haptic display:
• Tactile displays
• Force Feedback Devices
Department of Signal Processing
6
Displays
HAPTIC DISPLAYS
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Skin senses (reminder)
• Tactile – sense of touch
• Kinesthesis – sense of joint position and effort
•
E.g. carrying heavy load affects both tactile (pressure
on hands) and kinesthetic (muscle effort) senses.
• The tip of the finger
•
•
The most sensitive
Even 20 nm movement
• Termoception – sense of hot and cold (15-45°
C, 0.001°/s)
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Types of haptic displays
• Tactile - provides artificial stimuli (touch,
vibration) to receptors in our skin
• Kinesthetic – provides artificial sense of
force
•
•
Department of Signal Processing
End-effector – limits the natural movement, thus
providing feeling of solid objects / passive resistance
Force feedback – applies force, gives sense of active
movement / impact
SGN-5406 Virtual Reality 2012
Multimodality
• Haptic output needs to be correlated with video and
audio stimuli.
• Vibrations of gamepad paired with events in the
game (character getting hit).
• Vibration of a mobile phone while phone is ringing.
• Vibration of the device when a virtual key is pressed
on the touch screen.
Department of Signal Processing
Haptic display controller
Department of Signal Processing
SGN-5406 Virtual Reality 2012
11
Output devices
TACTILE
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Actuators
• Actuators, devices by which artificial stimuli is
presented to human skin.
• Actuators present some force to a region skin.
• Types of actuators:
•
•
•
•
•
•
Blader actuators – pneumatic, hydraulic
Vibrator actuators – electromechanical
Pin actuators – electromechanical
Piezoelectric devices.
Electro-active polymers.
Etc.
Department of Signal Processing
Bladder Actuators
•
Pockets that can be expanded and contracted:
• By controlling the flow of air (pneumatic)
• By controlling the flow of liquid (hydraulic)
•
Strategic placement of the pockets creates the
sensation of pressure on different areas of the
participant’s hand and body
Department of Signal Processing
Vibrator Actuator
•
Most often found in mass market products:
•
•
•
•
•
Game controllers
Mobile phones
Portable consoles
Data Gloves
Simulator seats
•
•
•
•
Easier to control then other types of devices.
Very robust and easy to implement.
Can’t convey the sense of surface texture or shape of the object.
Offer a limited range of effects.
•
Low-frequency speakers (subwoofers) can also be used as a vibratory
display.
Department of Signal Processing
Case study - Haptic Compass
• Belt with 12 vibration devices.
• At any given moment the device facing north vibrates.
• Augmentation of human senses. Humans do not have an explicit compass
sense. This device gives acute awareness of sense of direction.
• See
•
•
Udo Wächter, University of Osnabrück
http://feelspace.cogsci.uni-osnabrueck.de/
Department of Signal Processing
Pin Actuators
•
Small pin arrays placed on each finger.
•
Height of pins controlled electronically.
•
Textures are detected by pressure variations
across the fingertip over time.
•
Good for displays for blind people
Department of Signal Processing
Case study - interactive terrain
Northrop Grumman’s TerrainTable.
Array of 4600 pins push up touchsensitive silicone screen.
Overhead projector projects 2D map.
Military applications.
Department of Signal Processing
Temperature Actuators
•
Can very rapidly present temperature fluctuations,
typically to finger tips.
•
Danger of tissue damage.
•
Limits must be adjusted to the safe range of temperatures tolerated by
human skin.
•
Not the ”real” temperature of the simulated objects.
•
Use scenarios rare, not many practical implementations.
Department of Signal Processing
Senseg E-Sense
•
Tactile feedback on touch screens.
•
Screen is covered by a grid of
electrically activated elements.
•
Tixel a tactile pixel.
•
•
Tixels generate a controlled electric field which
extends several millimeters above the surface.
Exploits electro-sensory phenomenon.
See: http://senseg.com/technology/system-architecture
Department of Signal Processing
Senseg E-Sense
•
Ultra-low electrical current is
passed into the insulated
electrode – the tixel.
•
Electrical charge creates a small
attractive Coulomb force to finger
skin.
•
By modulating this attractive
force, any number of touch
sensations can be generated
from vibrations, clicks, textured
surfaces, etc.
Department of Signal Processing
Aurborne tactile display
• SIGGRAPH 2008 Emerging
Technologies: Airborne Ultrasound
Tactile Display
•
Mid-air tactile sensations by means of
airstreams generated by ultrasonic
actuators
• SIGGRAPH 2009 Emerging Tech:
Touchable Holography
•
See: http://www.youtube.com/watch?v=YP1zZAcPuw
Department of Signal Processing
SGN-5406 Virtual Reality 2012
22
Output devices
KINESTHESIC
Department of Signal Processing
SGN-5406 Virtual Reality 2012
End-effector Devices
•
Force Feedback – End-effector Devices.
•
End-effector devices are a special class of force
feedback devices.
Users limbs, hands, arms, legs are in contact with
machinery.
•
•
•
•
Input and output device.
Movements on hands serve as input.
Device provides feedback through the active force.
•
Generally linked to mechanical tracking sensors.
Department of Signal Processing
Telesurgery devices
•
Stanford Research Institute,
SRI International.
•
M7 Surgical Robot.
•
Developed for NASA.
•
Can perform operations in
Zero gravity.
•
Compensates for unwanted
movements in Zero-G.
Department of Signal Processing
End-effector Devices
•
UNC Nanomanipulator.
•
Microscop image on screen.
•
Mechanical arm with 6DoF.
•
Forcefeed back.
•
See:
http://cismm.cs.unc.edu/tag/nanomanipulator/
Department of Signal Processing
Force Feedback Data Glove
CyberForce system by CyberGlove
Systems.
Mechanical system that exerts force
on hand and arm.
Sense of weight and inertia while
picking up a "heavy" virtual object
Feel resistance of a simulated
objects.
http://www.cyberglovesystems.com/
products/cyberforce/overview
Department of Signal Processing
Components of End-effector
Displays
Mechanical trackers.
Force generating device.
Technologies:
• Electronic motors – one for each DoF,
• Hydraulics,
• Pneumatics
Department of Signal Processing
Features of End-effector Devices
•
Often operates also as an input device, potentially providing
resistance to input controls.
•
Mechanical movement sensors are generally incorporated directly
into the system. Mechanical tracking is generally very fast and
accurate.
•
Typically operate with respect to single point in the virtual world.
•
Number of DOFs: 1-6.
•
Tactile displays e.g. vibrator actuators can be mounted within the
end-effector.
•
Can also be constructed to enhance the user’s force.
Department of Signal Processing
Full Body Devices
End-effector devices which affect user’s
whole body.
Two mayor types:
• Exoskeletons
(wearable robotics)
• Surround platforms
Department of Signal Processing
Exoskeletons
•
Exoskeletons – Meaning external skeleton.
•
Can work with whole body or just with some parts.
•
•
Exoskeletons are not just I/O device.
Applications:
• Military:
• Enhance the power of user’s body or limbs. Increase the load a person can
carry or prologue the time a person can endure the load.
• Medical:
• Restore mobility in paralyzed limbs.
• Telepresence:
• Operate machinery at a distance, in environments dangerous for humans.
Department of Signal Processing
Exoskeleton Examples
• Novint XIO game controller.
• Partial exoskeleton.
• Arm exoskeleton.
• Mass market product.
Department of Signal Processing
Exoskeleton Examples
• Hybrid Assistive Limb.
• Cyberdine Inc. And Tsukuba Uni.
Japan.
• HAL detects bioelectrical impulses
sent from brain to limbs, using a set of
electrodes on the skin.
•
•
•
•
•
•
Possible applications:
Rehabilitation,
Physical training,
Help for disabled people,
Rescue services,
Entertainment (as input device)
Department of Signal Processing
33
Output devices
VESTIBULAR
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Vestibular Displays
•
Work with human vestibular apparatus (sense of equilibrium).
•
Work by physically moving the user.
•
Artificial sense of equilibrium, acceleration and orientation that user is
expected to feel during a real motion.
•
Strong relationship between the vestibular and visual systems of humans.
•
Prevents (or causes) simulator sickness.
•
Makes better immersion.
•
Typically used in specialized simulators (flight simulators).
Department of Signal Processing
Vestibular displays
• D-BOX Motion Code movie seat.
• Moving seat platform – 4 to 6DoF
depending on model.
• D-BOX Motion controller device.
• Movie theaters and amusement parks.
• http://www.d-box.com
Department of Signal Processing
Galvanic Vestibular Stimulation
•
Vestibular apparatus works on the same
principle as human auditory system.
•
Works on the same principle as Cochlear
implant.
•
Sending specific electric messages to a
nerve in the ear that transmits balance
information.
•
Can alter one’s perception of balance.
•
•
See also: lecture VR4.3
Side effect are still not known.
Department of Signal Processing
http://www.youtube.com/watch?v=ZlNfBrXYYTc
Locomotion Platforms
•
•
•
•
User needs to remain within certain limited physical space.
User needs to have an impression of physical movement in
virtual space.
Simulate the physical movement of the user in VR environment,
while keeping user in the same physical position.
Examples:
•
•
Omni-directional treadmills.
CirculaFloor.
Department of Signal Processing
Omni-directional treadmill
• Two treadmills placed on top of
each other.
• Movement along X and Y axes.
• Platform provides the sensation
of moving ground.
• Suspension system to hold the
user in place.
Department of Signal Processing
CirculaFloor
• Several movable tiles.
• With each step a new
segment moves infront of
user’s feet.
• Movement of the segment
counters the movement of
user while maintaining the
illusion of motion.
See: http://youtu.be/rYsvB2y2Ero
Department of Signal Processing
Haptic Rendering
• Rendering – process of generating of artificial
stimuli, based on the current state of VR
environment.
• Needs to be fast, up to 1000 Hz.
1. 3D objects & properties loaded from the database.
2. Collision detection. Only colliding objects are
passed on.
3. Compute collision forces, smoothing, mapping.
4. Haptic texturing (vibrations, temperature, etc.).
5. Present output to the user through haptic display.
Department of Signal Processing
41
Output devices
OLFACTORY
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Olfactory displays
Digital signals in software code trigger the generator to emit
precise amounts of the appropriate aroma.
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Olfactory displays
•
•
Artificial smell simulation
Mostly at research, prototype phase, some niche products
•
•
Film: Scent of Mystery (1960), with accompanying smells
Sensorama 1960, Smellizer 1984
•
Enhanced cosmetics, perfume, food advertising, aromatherapy
•
•
Aromas can be used to enhance the experience and trigger fear,
excitement and other emotions
Multi-sensory magazines (Esquire cover, smell etc.)
•
Companies
•
•
•
•
•
http://www.structuralgraphics.com/sensory-effects/
http://www.aerome.com/
http://www.aromajet.com/
http://www.digiscents.com/
http://www.trisenx.com/intro.html
Department of Signal Processing
SGN-5406 Virtual Reality 2012
Case study – ScentScape
• ScentScape™ specifications:
•
•
•
•
Provides 20 basic scents per
cartridge
Scent cartridges last 200 hours or
more in heavy use, depending on
personal settings
ScentScape™ scent cartridges will
be available in standard and
media-specific versions
Separate "volume control" to adjust
overall smell strength for personal
preference
Department of Signal Processing
See: http://www.scentsciences.com/products/scent_scape.html
SGN-5406 Virtual Reality 2012