Virtual Worlds: Input
Virtual Environments
• Importance of immersion, presence,
interaction, engagement, multisensory
• Elements of the world: graphics,
representation, visual and other senses,
interface, navigation, manipulation, story
• Interface: input, output, computer interface
Virtual Environments as a
Medium
Think of the following mediums and how
they communicate: medium, kind of virtual
world, final composition, how experienced
Painting, dance, music, written word, play,
movie, video game, virtual environments
Interaction in the Virtual World: Overview
• User interface
• Manipulation
–
–
–
–
Methods
Properties
Selection
Operations
• Navigation
– Wayfinding
– Travel
• Interaction and communication with others
– Sharing
– Collaboration
VR Worlds: Input Overview
• User input:
– 3D
– Mouse, keyboard
– Position tracking
– Body tracking
– Other physical input, including speech
• World input
User Interface Metaphors
• Familiar user interfaces: mobile phones,
TV controls, remote controls, cars, video
games, microwaves, doorways, cursor
icons, desktop, touchscreens, notions of
right, left and double clicking, mouseover,
tweeting, texting
• Do you use familiar ones in a VR world?
Has limits and shortcuts
User Interface Interactions
• Need to map user input to interactions
• Users must already know how to use a
tool or must be able to learn
• Users bring experience and cultural
knowledge
• Users need familiarity but also want to
extend what is normally possible in the
real world (has limitations)
User Interface Interactions
(con’t)
• Viewpoint navigation (or travel) and object
manipulation (and selection) are the main
types of interaction
• Differences between the two:
– User’s conceptual model (what moves)
– Extent of interaction space (navigation can be
large, object manipulation often small)
– Perceptual cues (navigation generally visual,
object manipulation uses haptics)
3D User Interfaces
• Operate in 3D space
• Natural: measure of how closely the
actions in the virtual environment
correspond to the actions in the real world
• Examples: Wii Remote, Sony Move,
Microsoft Kinect
Bowman, et all Questions about 3D
User Interfaces and Naturalism
• Are 3D inherently more natural than
traditional?
• Is naturalism the ultimate goal for designers?
• Does naturalism lead to better performance,
increased engagement, or better learning?
• When naturalism isn’t possible should the
designer go for traditional user interfaces or
interfaces with some degree of naturalism?
Issues Involving Natural User
Interfaces
• For the tasks of navigation, selection and
manipulation of object, how well do 3D natural
user interfaces perform vs. magic techniques or
some combination of them?
• Issues involving turning, tracking, selecting,
manipulating, steering
• Natural interfaces sometimes out-perform and
are sometimes inferior- depends on task and
context; they need high level of accuracy and
must be familiar
• Definition of natural?
• Lack of guidelines and standards for
gestural interfaces
• Problems raised by Norman and Nielsen
of visibility, feedback, discoverability,
consistency, scalability, reliability
Bowman et al table from Comm. of the ACM, 2012
Problems with Natural User
Interfaces
• How natural are they?
• Who decides what the gestures or other
actions will be?
• Cultural differences
Input: Position Trackers
• Position tracking – gives location and orientation
of user and/or parts of user and feeds it to the
computer - concerned with 6 degrees of
freedom: x,y,z position and orientation, generally
given by pitch, roll and yaw (angles with
orthogonal axes)
• General issues of accuracy (walking in a
landscape vs surgery), latency (conducting an
orchestra vs walking), interference from
surrounding objects (noise, occlusion, monitors,
metal), encumbrance such as cables (dancing,
large-scale movement), cost
Position Trackers: Main Types
•
•
•
•
•
•
Optical
Electromagnetic
Mechanical
Tangible
Neural – brain
Others: 3D, Videometric, Ultrasonic,
Inertial
Position Trackers: Optical
• Visual information is used for tracking
• Generally from video camera(s)
• Generally need more than one unless don’t want
6 DOF; for example, the Kinect
• Need good gesture recognition or visual
recognition techniques (algorithms)
• Sometimes use body sensors (motion capture)
• Main disadvantage is line of sight issues
Electromagnetic Position Trackers
• Very common
• Need a transmitter (sends low-level
magnetic fields from 3 orthogonal coils)
and a receiver- can be in both
configurations: transmitter on user or
receiver on user
• Strength of receiving signal varies with
orientation and position
• Some with cables and some wireless
Electromagnetic (con’t)
• Advantages are that line of sight does not
have to be clear
• Disadvantages: can be interference from
metal and monitors, must be within several
feet of the transmitter
• Eg. Ascension Flock of Birds, Polhemus
Position Trackers: Mechanical
• Mechanical arms and booms that
physically track movement
• Eg. of BOOM by Fakespace
• Advantages: accurate, no interference
problems
• Disadvantages: Can’t move very far
Position Trackers: Tangible
• Uses hands in a more natural way: always been
the case for tools from an axe to a toothbrush;
relationship between design and interaction
•
•
•
•
•
•
•
Physical models (for architecture, urban planning)
Clay
Sand
Blocks
Interactive lighting devices
Use of effectors, LEDs, motors, sensors, react to heat
Boxes for manipulating music
Position Trackers: Neural
• Field of brain-machine interface (BMI)
• Attempts to read brain signals to direct the
computer
• Efforts to use EEGs, VEP (visually evoked
potential), motor imagery
• Neuroprosthetic devices, brain-implantable chips
• Interfaces also benefit from research in
neuroscience
Position Trackers: Others
• Videometric: camera on person and tracks
surroundings via landmarks such as infrared
light sources
• Ultrasonic: high-pitched sounds emitted at
definite intervals – length of time – need multiple
transmitters and receivers – subject to noise in
the environment
• Inertial: gyrocsopes, accelerometers, tracking
device attached to user with cable- generally
only orientation, so used with other devicesinexpensive, good accuracy
• Neural: muscular (monitors electrical impulses in
skin) – some experiments in reading brain wave
User Input: Body Tracking
• Position of joints: Kinect
• Head tracking- usually through orientation
tracking- used to know what to project
visually
• Hand and fingers; position trackers, finger
sensors (gloves), virtual scalpels
• Torso and feet
• Other body tracking such as heart rate,
temperature
• Indirect tracking using props and platforms
Body Input: Eye Tracking
• Tracks movement of pupils – pretty
accurate, requires that the head be still
• eg. of Jeanne Stern’s project
• Used for people with disabilities
• Problem of “Midas Touch”
User Input: Other Physical Devices
• Physical controls: buttons, switches,
joysticks, mouse, steering wheels
• Arduino devices
• Props such as wands, 3D mouse, scalpel,
drill, realistic devices for particular
application (balls, bodies, spiders),
pressure devices, mobile phones
• Platforms: treadmills, locomotion, rings,
kiosks, wheelchairs, cockpits, submarine
control rooms, cars, workbenches,
• CAVEs, DiVE system at Duke
Other Physical Input: Speech
• Speech recognition: system will recognize
commands, natural language interaction, Siri
• Microphone
• Problem of accents and training: speaker
independence or dependence
• How activated: button, command (talk), vision
(need eye tracking)
• Sometimes increases bandwidth by using
separate processor
World Input to Virtual World
• Persistent virtual worlds: exist over many
experiences, multiple users, Web
communication, databases
• Importing data that changes (weather systems)
• Importing real world data such as objects and
obstacles, vision, knowledge, digital images,
satellite info
• Sometimes the info is brought in with
transducers: translates data into electrical
signals for the computer- devices such as
microphones, weather stations, video, sensors,
medical devices (for AR)
Sources
Questioning Naturalism in 3D User Interfaces, Bowman,
McMahan, and Ragan, Comm. Of the ACM, 2012
The Artificiality of Natural User Interfaces, Malizia and
Bellucci, Comm of the ACM, 2012
Gestural Interfaces: A Step Backward In Usability, Norman
and Nielsen, Interactions, 2010
Understanding Virtual Reality, Sherman & Craig, Morgan
Kaufman, 2003
The Tangible User Interface and its Evolution, Ishi, Comm.
Of the ACM, 2008
Neuroscience and the Future of Human-Computer
Interface, Minnery & Fine, Interactions, Mar-Apr 2009
Building on Realism and Magic for Designing 3D
Interaction Techniques by Kulik, IEEE Computer
Graphics and Applications, Nov/Dec 2009
Sources – con’t
Tangible Interaction=Form+Computing, Baskinger & Gross,
ACM Interactions, Jan-Feb 2010
3D input devices, Frohlich et al, CG&A, Mar-Apr 2006
3D User Interfaces: New Perspectives and Directions,
Bowman et al, Comp Graphics and Apps, Nov-Dec 2008
Usability of Multiple Degree-of-Freedom Input Devices and
Virtual Reality: Displays for Interactive Visual Data
Analysis, Moritz and Wischgoll, VRST (Virtual Reality
Software and Technology) 2007