Large Display Research
Overview
Mary Czerwinski, George Robertson,
Brian Meyers, Greg Smith, Daniel
Robbins & Desney Tan
Introduction


The increasing graphical processing
power of the PC has fueled a
powerful demand for larger displays
Despite the increasing affordability
and availability of larger displays,
most users’ display space
represents less than 10% of their
physical workspace area
Microsoft Research
2
Introduction continued




Current interfaces are designed around the
assumption of a relatively small display providing
access to a larger virtual world
How might users cope with and benefit from
display devices that provide 25% to 35% of their
physical desk area or perhaps one day cover
entire office walls?
We evaluated usability issues for large displays
and developed a series of research prototypes
that address various issues we discovered
Also, large displays should and can present
beautiful visualizations
Microsoft Research
3
Harris Poll responses (7/02, N=1197)
Mutiple PCs and Displays
Percent Respondants
80%
70%
60%
50%
40%
All
30%
20%
10%
0%
None
Multiple monitors
attached to
multiple
computers.
Laptop and
Dualmon or higher
desktop monitor
connected
together.
Config
Microsoft Research
4
Why a Larger Display Surface?



Productivity benefits
10-30% (despite
sw usability issues)
Users prefer more
display surface
Prices dropping fast
$1,200
$1,000
$1,089
$905
$800
$US

Projected LCD Pricing 2002-2005
$600
$400
$597
$437
15" -13.5%
$752
$699
$378
$510
$327
$200
$625
17" -14.6%
$436
18" -16.9%
$283
$0
2002
2003
2004
2005
Footprints getting
smaller
Microsoft Research
5
Productivity Study w/dSharp Display






Triple
projection
Matrox
parhelia card
3028 x764
resolution
42 in. across
Slightly
curved
120 degree
FOV
Microsoft Research
6
Task Times – Significant Benefits
Effects of Display Size on Task Times
Average Task Time (Seconds)
160
140
120
100
Small
80
Large
60
40
20
0
DISPLAY
Microsoft Research
7
User Satisfaction - Significant
Average Rating (1=Disagree,
5=Agree)
the tasks were easy to perform
5
4
3
2
1
0
Small
Large
Display Size
Microsoft Research
8
Windows Layout - Significant
Average Rating
(1=Disagree,
5=Agree)
I was satisfied with the ease of windows
layout
5
4
3
2
1
0
Small
Large
Display Size
Microsoft Research
9
Cognitive Benefits of Large Displays


Czerwinski et al. document results showing that
larger displays lead to improved recognition
memory and peripheral awareness
Tan et al. demonstrate the advantages of large
displays on 3D navigation in virtual worlds.


Wider fields of view lead to increased ability to
process optical flow cues during navigation, cues
that females are more reliant upon than males
Tan et al. also found that large displays provide for
a more immersive experience when performing
spatial tasks, building better cognitive maps of the
virtual world
Microsoft Research
10
But…Usability Issues






Why click to bring a
clearly visible window
into focus? caused
many errors
Where is my cursor?
Where is my start
button?
Where is my taskbar?
Where are my dialogs?
The software doesn’t
know where the
bezel is…
Microsoft Research
11
Vibelog: How Users Interact with Displays

1st activity repository for
studying windows usage
in aggregate




can’t fix what you can’t
measure
can profile users
can be extended
Single user: capture task
contexts to surface
pertinent ui or provide
reminders
Microsoft Research
12
Multitasking Visualization




Colored block for each time point and app
Amount of shading indicates percentage of
visibility of the window
Tasks
Subtasks
Microsoft Research
13
Task Switching Visualization


Switching tasks (red to blue)
How are email windows arranged and
used?

compare to...
Microsoft Research
14
Windows and Task Management
Issues Emerge


More open
windows
Users arrange
windows spatially
Taskbar does not
scale:


Relationship between # of Monitors and # of
Windows Left Open
18.00
16.00
aggregation model
not task-based
users can’t operate
on groups of related
windows
Avg. # of Windows Left Open

14.00
12.00
Single Monitor
10.00
DualMon
8.00
TripleMon
6.00
4.00
2.00
0.00
No. of Monitors
Microsoft Research
15
Changes in Window Access Patterns
100
90
Percentage of Access Technique
80
70
60
Win
Taskbar
50
40
30
20
10
0
1
2
3
Number of Monitors
Microsoft Research
16
Basic Usability Issues
Seven broad categories:
1.
2.
3.
4.
5.
6.
7.
Input: Losing track of the cursor
Input: Distal access to information
Window management problems
Task management problems
Configuration problems
Failure to leverage the periphery
Failure to use displays artistically
In this overview, research prototypes will be described
that address many of these problems across the
research community
Microsoft Research
17
Input: High Density Cursor (Baudisch et al.)
problem
• at high mouse speeds, the mouse cursor
seems to jump from one position to the
next
use
mo ion
t
mo
regular mouse cursor
f
ursors p ill-in c
c
n
i
l
l
fi
e revio urso
t fram
us f rs
curren
ram
e
high-density cursor
use
mo ion
t
mo
solution
• high-density cursor inserts additional
cursor images between actual cursor
positions
• the mouse cursor appear more continuous
the windows mouse trail…
• makes mouse trail last longer
• drawback: cursor images lag behind
Windows mouse trail
...is not high-density cursor
• hd cursor makes mouse trail denser
• lag-free: mouse stops => cursor stops
high-density cursor
Microsoft Research
18
Input: Drag-and-Pop (Baudisch et al.)
Problem
 Large displays create long
distance mouse movement
 Touch & pen input has problems
moving between screen units
Solution
 Drag-and-pop brings proxies of
targets to the user from across
display surfaces
 The user can complete drag
interactions locally—no need to
deal with distances or to cross
display borders
Microsoft Research
19
Input: Continued 1



Vacuum (Bezerianos
& Balakrishnan)
Vision-tracked multifinger gestural input
(Malik, Ranjan &
Balakrishnan)
Handheld projector
(Forlines et al.)
Microsoft Research
20
Input: Continued 2

Vogel & Balakrishnan--Distant
freehand pointing and clicking

Hand controls pointer position and makes
click selection with finger or thumb
Microsoft Research
21
Input: Continued 3


Kahn et al.—”Frisbee”, a
remote control UI
Grossman et al.—3D
modeling techniques
Microsoft Research
22
FlowMenus and Zoomscapes

Gruimbretiere et al.


Zoomscapes allowed currently unused windows to hang
around, but at 25% of their normal size
Flowmenus were pen-based, fluid interaction techniques for
invoking commands at the user’s point of interest
Microsoft Research
23
Input: TableTop Interaction


MERL’s DiamondTouch
system and twohanded touch gestures
Hinrichs et al.
“Interface Currents”
for collaboration
Microsoft Research
24
Window & Task Management Support
Table
Cloth
Task flasher
LiveBoard
Task Zones
GroupBar
Scalable Fabric
Kimura
WinCuts
Microsoft Research
25
Window Management Challenges


What to do
when you
have your
information
spread out in
physically
large display
surfaces?
Real display at
Georgia Tech
(Hutchings et
al.)
Microsoft Research
26
Window Management: Task Flasher



A more visual alt + tab
Uses 3d scaling and zooming animation to
show selected window
Windows stay on the monitor on which
they are positioned
Microsoft Research
27
Window Management: Table Cloth

Problem:


User wants to
access content
physically far
away
Solution:



Pan the desktop
to user
Compress
content to the
right of focus
Grab content
you need and
snap back
Microsoft Research
28
Task Management: LiveBoard (Elrod et
al.)




Pen-based group
interaction around a large
surface
Supported drawing, popup menus, selection and
annotations
Boardwalk software
provides “planks” or
tasks, from which the
user would choose
A plank automatically
opened up a set of
applications (e.g.,
meeting, scoreboard,
slideshow, games, etc.)
Microsoft Research
29
TaskZones: Virtual Multimon Desktops
(Hutchings et al.)

Problem: user
has multiple
desktops




wall of
monitors
how to switch
focus?
might be using
a phone or
remote control
Solution:
TaskZones
Microsoft Research
30
Task Management: GroupBar





Taskbar for lightweight grouping of
windows into tasks
Can have multiple bars for large
displays
Download at
http://research.microsoft.com/research
/downloads/default.aspx (search for
GroupBar)
~40,000 downloads
Desktop snapshots
Microsoft Research
31
GroupBar Usage Study
Microsoft Research
32
Task Management: Scalable Fabric



Scaled down versions of grouped windows in periphery
Supports task switching and task reacquisition
http://research.microsoft.com/research/downloads/de
fault.aspx (search for fabric); over 20,000 downloads
Microsoft Research
33
Scalable Fabric Usage Study
Survey Question
(1=Disagree, 5=Agree)
TaskBar
Scalable
Fabric
Task Times
20.00
2.95
It was hard to go back
and forth between my
various windows and
applications using…..
3.32
I was satisfied with the
functionality of the ….
2.68
The TaskBar/Scalable
Fabric is an attractive
innovation for Windows.
3.16
4.26
1.84
3.78
Average Task Time (Sec)
Task switching was easy
to perform using the…
15.00
TaskBar
10.00
Scalable Fabric
5.00
0.00
4.47
Task Management Tool
Tool
Figure 9: Average task times +/- one standard error of the
mean for TaskBar and Scalable Fabric.
Table 1: Average satisfaction ratings for the TaskBar and Scalable Fabric.
All ratings were significantly in favour of Scalable Fabric at the p<.05 level.
Microsoft Research
34
Task Management: Kimura (MacIntyre
et al.)


Supported
multitasking and
background
awareness using
interactive
peripheral displays
“Montages” or
activities based on
desktop interaction
Microsoft Research
35
WinCuts: Initial Motivation

Problems:


Sharing live windows/information is hard
Screen space is scarce and laying out information
optimally is hard
Microsoft Research
36
WinCuts Video
Microsoft Research
37
Specify Region of Interest
Animated
advertisement
Region of interest
Seldom used
interface buttons
Scrolling ticker
Microsoft Research
38
Organize Content
Rescale WinCut so
graph scales are comparable
Relevant
content
Microsoft Research
39
Reconfigure Interfaces
Relevant
interface
elements
Relevant
content
Microsoft Research
40
Sharing WinCuts across Machines
Click on “Share”
Specify destination (also running WinCuts)
WinCut appears on destination machine
1.
2.
3.
Remote WinCuts work just like local
WinCuts


Except input redirection disabled
Microsoft Research
41
Share Content when Collaborating
Microsoft Research
42
Now we’re Thinking…

With remote input redirection working:


Create ad hoc remote controls and interfaces
Work across displays and devices
Cell Phone
Laptop or
TabletPC
Desktop
PDA
Microsoft Research
43
Huang & Mynatt’s Design Space for
Peripheral Awareness Display
Research
Microsoft Research
44
Interaction based on Distance


Vogel &
Balakrishnan’s notion
of an interactive,
ambient, public
display
Different functionality
based on distance

Ambient, Implicit,
Subtle and Personal
spaces
Microsoft Research
45
Large Displays as Peripheral
Awareness Surfaces




Brignull and Rogers’
Opinionizer
McCarthy’s et al.’s
Unicast, Outcast &
GroupCast
Izadi et al’s Dynamo
….etc.
Microsoft Research
46
And, Large Displays as Art and Info




Blinkenlights 2.0
in Berlin
Interactive
waterfall display
in children’s
hospital
Weather patterns
window in an art
gallery at night
Etc….
Microsoft Research
47
Conclusions







There is a clear trend toward larger displays
Large displays increase user productivity, aid user
recognition memory, and in some cases can eliminate
gender bias
User studies have identified numerous usability
problems
Research prototypes were presented that outline
techniques for solving many of these problems
The work of integrating these prototype solutions into
one system remains to be done
Correcting these problems significantly improves the
user experience on large displays
Large displays also useful tools for peripheral
awareness and can be aesthetically pleasing
Microsoft Research
48
Acknowledgements






Gary Starkweather
Patrick Baudisch
Ed Cutrell
Eric Horvitz
Jonathan Grudin
All of our colleagues doing large
display research and kindly granted
me permission to show their work
Microsoft Research
49
Thanks for your Attention!


Questions?
More information at:
http://research.microsoft.com/research/vibe
Microsoft Research
50
References











Ballagas, R., Rohs, M. & Sheridan, J. (2005). Sweep and Point & Shoot: Phonecam-based interactions for
large, public displays. In Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing
Systems, pp. 1-4.
Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B., and Zierlinger, A. Drag-andPop and Drag-and-Pick: techniques for accessing remote screen content on touch- and pen-operated
systems. In Proceedings of Interact 2003, pp.57-64.
Baudisch, P. Cutrell, E, and Robertson, G. High-density cursor: a visualization technique that helps users
keep track of fast-moving mouse cursors. In Proceedings of Interact 2003, pp. 236-243.
Bezerianos, A.& Balakrishan, R. (2005). Canvas Portals: View and space management on large displays.
IEEE Computer Graphics and Applications, 25(4). pp. 34-43
Bezerianos, A. & Balakrishnan, R. (2005). The Vacuum: Facilitating the manipulation of distant objects.
Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems. pp. 361-370.
Biehl, J.T. & Bailey, B.P. (2004). ARIS: An interface for application relocation in an interactive space. In
Proceedings of Graphics Interface 2004, pp. 107-116.
Brignull, H. & Rogers, Y. (2003). Enticing people to interact with large public displays in public places. In
Proceedings of Interact 2003, pp. 17-24.
Buxton, W., Fitzmaurice, G., Balakrishnan, R., Kurtenbach, G. (2000). Large displays in automotive
design. IEEE Computer Graphics & Applications, July 2000.
Cao, X. & Blakrishnan, R. (2003). VisionWand: Interaction techniques for large displays using a passive
wand tracked in 3D. Proceedings of UIST 2003 – the ACM Symposium on User Interface Software and
Technology. pp. 173-182.
Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G. and Starkweather, G. (2003). Toward
characterizing the productivity benefits of very large displays. In Proceedings of Interact 2003, pp. 9-16.
Elrod, S., Bruce, R., Gold, R., Goldberg, D., Halasz, F., Janssen, W., Lee, D., McCall, K., Pedersen, E., Pier,
K., Tang, J. & Welch, B. (1992). LiveBoard: A large interactive display supporting group meetings,
presentations and remote collaboration. In Proceedings of CHI 1992 – the ACM Conference on Human
Factors in Computing Systems, pp. 599-607.
Microsoft Research
51
References Continued 2















Grossman, T., Balakrishnan, R., Kurtenbach, G., Fitzmaurice, G., Khan, A. & Buxton, W.. (2001). Interaction techniques for
3D modeling on large displays. Proceedings of the ACM Symposium on Interactive 3D Graphics (I3DG2001), pp. 17-23. New
York: ACM.
Grudin, J. (2002). Partitioning digital worlds: Focal and peripheral awareness in multiple monitor use. In Proceedings of CHI
2002 – the ACM Conference on Human Factors in Computing Systems, pp. 458-465.
Guimbretiere, F., Stone, M., and Winograd, T. (2001). Fluid interaction with high-resolution wall-size displays. In
Proceedings of UIST 2001, pp. 21-30.
Huang, E.M., Russell, D.M. & Sue, A.E. (2004). IM Here: Public instant messaging on large, shared displays for workgroup
interactions. In Proceedings of CHI 2004 – the ACM Conference on Human Factors in Computing Systems, pp. 279-286.
Hutchings, D., Czerwinski, M., Smith, G., Meyers, B., Robertson, G Display space usage and window management operation
comparisons between single monitor and multiple monitor users. In Proceedings of AVI 2004, pp. 32-39.
Khan, A., Fitzmaurice, G., Almeida, D., Burtnyk, N. & Kurtenbach, G. (2003). A remote control interface for large displays.
ACM UIST 2003 Symposium on User Interface Software & Technology, pp. 127-136.
Khan, A. Matejka, J. Fitzmaurice, G. & Kurtenbach, G. (2005). Spotlight: Directing users' visual attention on large displays.
Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems pp. 791 – 798.
MacIntyre, B., Mynatt, E., Voida, S., Hansen, K., Tullio, J., Corso, G. (2001). Support for multitasking and background
awareness using interactive peripheral displays. In ACM UIST 2001 Symposium on User Interface Software & Technology,
pp. 41-50.
Malik, S., Ranjan, A. & Balakrishnan, R. (2005). Interacting with large displays from a distance with vision-tracked multifinger gestural input. Proceedings of UIST 2005 - the ACM Symposium on User Interface Software and Technology. pp. 4352.
McCarthy, J.F., Costa, T.J. & Liongosari, E.S. (2001). UniCast, OutCast & GroupCast: Three steps toward ubiquitous,
peripheral displays. In Lecture Notes in Computer Science, Vol. 2201, pp. 332-*, Germany: Springer-Verlag.
Mynatt, E., Igarashi, T., Edwards, W., and LaMarca, A. (1999). Flatland: new dimensions in office whiteboards. In
Proceedings of CHI 1999 – the ACM Conference on Human Factors in Computing Systems, pp. 346-353.
Paradiso, J.A., Leo, C.K., Checka, N. & Hsiao, K. (2002). Passive acoustic sensing for tracking knocks atop large interactive
displays. In Proceedings of the IEEE Sensors 2002 Conference, pp. 1-6.
Pedersen, E., McCall, K., Moran, T., Halasz, F.G. (1993). Tivoli: An electronic whiteboard for informal workgroup meetings.
In Proceedings of CHI 1993 – the ACM Conference on Human Factors in Computing Systems, pp. 391-398.
Robertson, G.G., Czerwinski, M., Baudisch, P., Meyers, B., Robbins, D., Smith, G., and Tan, D. (2005). Large display user
experience. In IEEE CG&A special issue on large displays, 25(4), pp. 44-51.
Robertson, G., Horvitz, E., Czerwinski, M., Baudisch, P., Hutchings, D., Meyers, B., Robbins, D., and Smith, G. (2004),
Scalable Fabric: Flexible Task Management. In Proceedings of AVI 2004 pp. 85-89.
Microsoft Research
52
References Continued 3











Russell, D. M., Drews, C. & Sue, A. (2002). Social aspects of using large interactive displays for
collaboration. In Proceedings of UbiComp 2002: Ubiquitous Computing: 4th International
Conference, pp. 229-236.
Smith, G., Baudisch, P., Robertson, G., Czerwinski, M., Meyers, B., Robbins, D., and Andrews, D.
(2003). GroupBar: The TaskBar Evolved. In Proceedings of OZCHI 2003, pp. 34-43.
Streitz, N.A., Geiler, J. & Holmer, T. (1998). Roomware for cooperative buildings: Integrated design
of architectural spaces and information spaces. In Lecture Notes in Computer Science, Vol. 1370,
pp. 4-*, Germany: Springer-Verlag.
Swaminathan, K. & Sato, K. (1997). Interaction design for large displays. In ACM’s Interactions
(4), pp. 15-24.
Tan, D., Czerwinski, M., and Robertson, G. (2003). Women Go With the (Optical) Flow. In
Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems, pp. 209215.
Tan, D.S. and Czerwinski, M. (2003). Information Voyeurism: Social impact of physically large
displays on information privacy. Proceedings of CHI 2003 – the ACM Conference on Human Factors
in Computing Systems. pp. 748-749.
Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2003). With similar viewing angles, larger displays
improve spatial performance. In Proceedings of CHI 2003 – the ACM Conference on Human Factors
in Computing Systems, pp. 217-224.
Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2004). Physically large displays improve path
integration in 3D virtual navigation tasks. In Proceedings of CHI 2004 – the ACM Conference on
Human Factors in Computing Systems, pp. 439-446.
Tan, D.S., Meyers, B. & Czerwinski, M. (2004). WinCuts: Manipulating arbitrary window regions for
more effective use of screen space. In Extended Abstracts of Proceedings of CHI 2004, pp. 15251528.
Vogel, D. & Balakrishnan, R. (2005). Distant freehand pointing and clicking on very large high
resolution displays. Proceedings of UIST 2005 - the ACM Symposium on User Interface Software
and Technology. pp. 33-42.
Vogel, D. & Balakrishnan, R. (2004). Interactive public ambient displays: transitioning from implicit
to explicit, public to personal, interaction with multiple users. Proceedings of UIST 2004 – the ACM
Symposium on User Interface Software and Technology. pp. 137-146.
Microsoft Research
53