You Are Player One
User Interfaces in Interactive Entertainment
Richard Marks
Manager, Special Projects
An Interface Revolution is beginning
 Graphics have improved dramatically
 Interface is limiting the interactivity
 Interface is limiting the fun
Describe the ultimate interface…






Fun
Intuitive
Interactive
Flexible
Enabling
Fun!
 PS9 commercial shows what people think is “futuristic”
 Blurring of distinction between real and virtual
 Sensory enhancement
 Gaming anywhere and everywhere
 Experiential
 Describe the system by describing its interface
Background: Games are big busine$$
 For the last 2 years, video game sales revenue has

exceeded box-office receipts
FY 2004, Electronic Arts reports…
 Revenue of $2.96 billion
 Net income of $577 million
 In less than 4 years worldwide….
 70 million PlayStation2 hardware units
 572 million PlayStation2 software units
 1 in 4 American homes has PlayStation2
Accelerating adoption rate
Million units
10
PS2
9
8
7
32/64bit
Game
6
5
CD
Player
PC
4
3
Walkman
2
1
0
0
1
2
3
4
5
6
7
8
9
years
PlayStation R&D (SCE America)
 Mission
 Catalyze new ideas for interactive entertainment
 Focus
 0 to 5 years
 Groups




Advanced Graphics
Distributed Processing
Developer Tools (compilers, optimizers)
Special Projects
 Animation and physical simulation
 Man-machine interface
R&D job description
 Improve existing experience
 Enable new experiences
 “We don’t make games, we make them better”
 Give game developers more tools to use for creation
 Create software that game developers can use
 Prototype tech demos to inspire game developers
R&D cycle
 Matched to PlayStation platform cycle (5-6 years)
 2-3 years research
 Focused on about a 5 year look-ahead
 Intended to raise the bar of computer entertainment
 2 years development
 Hardware specification, proving workloads
 Advanced libraries, technology demos, samples
 1 year production
 Maintenance, support of our releases
 Push
Research models
 Research group pushes technology out of lab and into
product groups
 Tough because product groups have schedules
 Example: EyeToy
 Pull
 Tends to be evolutionary, not revolutionary
 Less freedom, but certainty of usefulness
 Example: PlayStation Voice Recognition
 Stockpile
 What do we do with this? Don’t know, save it for later.
 Most common model for large research groups
Level of interaction
 Games now….
“L1 o ”
PS2
 Future games….
Cameras
Microphones
Gamepad
Inertial
Etc.
PS?
“42”
Interface Modalities
 Inertial (accelerometers, rate gyros)
 Touch (haptics, force-feedback)
 Audio
 Speech (speech recognition, speaker id)
 Waveform (frequency, amplitude, rhythm)
 Video





Familiar
Fast
Physical
Personal
Multipurpose
Voice Input
 Voice recognition
 Command/control, user-independent, isolated word, text authored
 Used in NFL GameDay, NCAA Gamebreaker, NBA Shootout
 Online adaptation to user’s voice
 Far-microphone voice input in noisy environments
 Acoustic Echo Cancellation
 Multi-microphone techniques
 Speaker identification
 Voice fonts
Video interface categories
 Video As Input (VAI)
 joystick replacement
 user does not see video
 applicable to existing games
 Enhanced Reality (or augmented reality)
 live video mixed with computer graphics
 movie-like special effects
 new entertainment genre
 Real-time Motion Capture
 direct interaction
 enables physical interaction
Video As Input
 User does not see the video
 Video quality only important for processing
 Only need to capture/process useful info
 Frame rate and latency are critical
 Immediate and obvious visual
feedback is important
Enhanced Reality
 Augmented reality, but with
an entertainment focus




Minimize encumbrance
Utilize common hardware
Simplify setup/calibration
Create an enjoyable
user experience
Video interface design goals
 Accessible
 Affordable
 Easy-to-use
 Unobtrusive
 Targeted
 Lens good for low light
 Compression specific to PS2 hardware
 Video size appropriate for PS2 processing power
 Multi-purpose
 Communication (e.g. EyeToy: Chat)
 Snapshots
EyeToy: research to product
 1999-2000
 initial research and prototyping
 2001
 join the London game team for 3 months
 learn about their world
 L transfer my technology, but more importantly, they absorb how
I think, so they can extend the ideas
 2002
 experienced game team creates games, not technologist
 2003
 EyeToy launches globally
 2004
 5 million EyeToys shipped to date
EyeToy: Play




Easy to jump in and play
Not too serious
Very social
Amazing demographics
 Young children, parents, girls, grandparents
Mature video technology
 Motion detection
 Color segmentation
 Simple-object pose recovery
 Sphere, position only
 Cylinder, position+orientation
 Gesture recognition
 Pattern matching
 Basic video/graphics compositing
Recent video technology
 Freeform mouse-like input (e.g. Minority Report)
 Retro-reflection
 The “Clam” deformable input device
 Face detection
 Face tracking
 Better color tracking
 Demos (clam, view tracking)
Just around the corner
 PSP + camera
 PSP is your window to an
enhanced world
 Mobile augmented reality
 Mobile video communication
 Face tracking for animation communication
 Next generation console + camera




Much more CPU
Much higher input bandwidth
Camera should be higher res, faster, more sensitive
New interfaces an important design criterion
Future video technology
 Per-pixel distance measurement
 RGBZ video
 Allows true mix of graphics with video using Z buffer
 Easy per-pixel background/foreground (user)
segmentation
 Video: Enhanced Reality special effects – play movie
 Video: Real-time motion capture – play movie
Concluding remarks
 Interface a key element of design
 Blend of technology and psychology
 Impacted by real-world considerations
 Entertainment is a great way to try new technology
 Especially the game industry
 A revolution is already underway
 Now, my 4-year-old waves his hand in front of the TV,
expecting it to turn on
 Next year, he’ll be talking to it
 Soon, all this will fit in his pocket
 And he thinks nothing of it….
Thank you!
 Questions?