CHI 2006 · Work-in-Progress
April 22-27, 2006 · Montréal, Québec, Canada
Swordfish: User Tailored Workspaces
in Multi-Display Environments
bindings, each user can choose their own display
connections and create a personalized MDE. This
approach also helps manage changes in the
environment as devices enter, move, or leave.
Jim Wallace
Vicki Ha
Dalhousie University
Dalhousie University
6050 University Ave.
6050 University Ave.
Halifax, NS, Canada
Halifax, NS, Canada
[email protected]
[email protected]
Ryder Ziola
Kori Inkpen
Dalhousie University
Dalhousie University
6050 University Ave.
6050 University Ave.
Halifax, NS, Canada
Halifax, NS, Canada
Introduction
[email protected]
[email protected]
The tight coupling between monitors and processors
characteristic of personal computers over the past two
decades is eroding. It is becoming more common for
people to utilize multiple display environments (MDEs)
for their desktop work [7]. In addition, conference
rooms and public venues are starting to provide access
to a myriad of public displays. As displays become
ubiquitous in our environment, we need to think
carefully about how information is managed across
these displays and how users (in particular, collocated
collaborators) will interact with these displays.
ACM Classification Keywords
Abstract
This paper presents a novel interaction metaphor for
Multiple Display Environments (MDEs) called
lightweight personal bindings. This approach enables
users to easily bind edges from one display to another
and move seamlessly between displays. The goal of this
work is to support collocated collaboration in a dynamic
multi-display environment while accommodating users’
personal preferences. With lightweight personal
Copyright is held by the author/owner(s).
CHI 2006, April 22–27, 2006, Montréal, Québec, Canada.
ACM 1-59593-298-4/06/0004.
H.5.3 Group and Organization Interfaces: Collaborative
Computing.
Recent research in the area of MDEs has stressed the
importance of connecting personal devices (i.e. laptops)
with shared displays to support collaboration [12]. This
not only necessitates an underlying architecture that
allows multiple, heterogeneous devices to be
1487
CHI 2006 · Work-in-Progress
April 22-27, 2006 · Montréal, Québec, Canada
personal bindings. Finally, we reflect on initial usage of
Swordfish and directions for future work.
Navigation in Multi-Display Environments
The benefits of multi-display systems for single users
have been well established in the literature [5, 7].
Research on collaborative multi-display systems has
stemmed from previous work on public and semi-public
displays [7, 8] as well as the interconnection of multiple
heterogeneous devices [3, 10, 11].
Figure 1. A sample MDE with bindings
between displays
connected, it also requires that users understand how
to work seamlessly in this larger virtual workspace.
Given that many people choose to configure their
desktop environments differently [7], it is likely that
there will also be differences in how people want to
configure their virtual workspace environment. We
therefore need to examine how collaborative MDEs are
organized and how they are interpreted by users.
The goal of our research is to develop a lightweight
approach that allows users to easily create and then
navigate across multiple displays through virtual
connections. Lightweight personal bindings are user
defined connections from the edge of one display to the
edge of another display in a MDE. These connections
can be created and altered quickly, allowing users to
define personalized workspaces that support their
individual preferences and perspective of the space.
This paper first discusses related work in the area of
MDEs. We then describe Swordfish, an application we
developed to explore the concept of lightweight
Various approaches have been investigated to support
navigation between displays within an MDE. PointRight
[9] and TeamSpot [15] utilize a spatial metaphor to
encode connections between displays in which the user
moves their cursor off the side of one screen and it
appears at the border of the display adjacent to it.
Another approach is the use of labels for each display in
the environment. For example, Mighty Mouse [3]
maintains a list of all displays in the environment and
users can select the display they want to control by
clicking on an icon associated with that display.
More recently, Biehl and Bailey [2] developed a hybrid
approach that utilizes an iconic representation in which
the icons are laid out to reflect the physical position of
the displays in the room. Head tracking approaches
have also been applied to control cursors in multidisplay environments [1]. The use of head tracking
allows users to effortlessly transfer control to any
display in their line of sight, essentially creating fully
flexible bindings.
Several open source and commercial systems are
available for network control of multiple systems (e.g.
Synergy [13] and Desktop Rover [4]). However, these
1488
CHI 2006 · Work-in-Progress
April 22-27, 2006 · Montréal, Québec, Canada
systems are primarily designed for individual users
connecting to multiple machines and only allow one
person to utilize a display at a time.
A key challenge in designing collaborative MDEs is
providing support for individual as well as group
processes. It is naïve to think that every user will view
the MDE in the same way and want to interact with it in
the same manner. Just as people like to configure how
their windows are laid out on their desktop, people will
likely have preferences about the layout of their multidisplay workspace, dictating how they move between
different displays [7].
Lightweight Personal bindings
The idea of lightweight personal bindings is that users
dynamically ‘bind’ specific edges on their personal
device to other displays in the MDE (i.e. laptop to
various public displays) to provide a virtual connection
between the displays (see Figure 1). Once a binding
has been created, moving the cursor off the screen in
the direction of the binding moves the cursor to the
connected display. Any display can have bindings to
several other displays and multiple users can bind to,
and actively interact with, a display simultaneously.
The main advantage of this approach is that users can
freely alter the shape of their workspace to meet their
personal needs. In addition, users are likely more
cognisant of the virtual connections between displays
since they have chosen which bindings to create and
have manually instantiated them. Only displays that are
being used require bindings, thereby reducing the
complexity of a user’s workspace.
User created bindings have previously been
investigated for tablet PCs. For example, ConnecTables
placed in close proximity to each other automatically
form a connection between the adjacent edges [14].
Stitching [6] is another technique that allows users to
connect mobile devices on the fly. This approach allows
users to create bindings using a pen-based gesture
across the devices. Our lightweight personal bindings
extend these concepts to more diverse environments
and permit more connections per device.
Swordfish Testbed
As part of this work, we developed Swordfish
(Spontaneous Workspace Organization Designed For
Interactive SHared environments), a testbed
application to explore lightweight personal bindings. It
was developed in C# using Microsoft .NET. The ability
to create arbitrary bindings between displays in a MDE
is available in other research and commercial systems
[9,15], however, development of our own suite enabled
us to explore many underlying components of display
management systems.
The Swordfish framework is composed of three basic
elements: network hosts, user input devices, and
displays. Each network host has a number of different
input devices and displays associated with it, and
maintains a database of all displays and associated
security permissions on the network. The network hosts
communicate with each other using a peer-to-peer
protocol which automatically detects any other
available public displays on the subnet. This approach
allows users to view all available displays and easily
create a binding to any display.
1489
CHI 2006 · Work-in-Progress
April 22-27, 2006 · Montréal, Québec, Canada
Swordfish renders its own cursors, enabling multiple
users to each have a fully functional cursor on the same
screen. When a user’s cursor leaves their local screen,
input is forwarded directly to the receiving host via a
UDP stream of mouse and keyboard events. The
receiving host draws the cursors and places the events
on the system’s global event queue. The remote host
then treats these events as if they were local input.
Swordfish’s ability to simulate a multi-user environment
is limited, of course, by the input restrictions of the
underlying OS. Specifically, only one open window can
have mouse and keyboard focus and only one user can
drag at a time. Because it is intended for collaboration
among small groups, these limitations can be largely
mitigated through social mediation. This compromise
allows Swordfish to occupy the middle ground between
a single user multi-display environment and true
groupware, without the significant overhead of either
option. As well, custom applications can be developed
to handle mouse input from multiple users to achieve
the full benefit of a multi-user system
to iteratively design, explore, and evaluate various
interface components for lightweight personal bindings.
To date, we have utilized this sandbox application to
examine various interface approaches for managing
(creating and modifying) and visualizing lightweight
personal bindings.
Managing Bindings
We have investigated three main approaches for
managing lightweight personal bindings: menu, direct
manipulation and graph based interactions. All three
approaches use standard interaction techniques which
make them easy and intuitive to use.
The menu-based interface provides users with a dropdown contextual menu for the creation of bindings. The
menu is obtained by clicking on the border of the
screen where the binding will originate and provides
both a textual label and a thumbnail image for each
available display (see Figure 2a). Bindings are created
by selecting the menu item corresponding to the target
screen. The same technique is used to de-select a
screen and delete the binding.
One of the main purposes of Swordfish is to enable us
Figure 2. a) A border interaction technique b) a Palette-based interaction technique and c) a
conceptual map-based interaction technique
1490
CHI 2006 · Work-in-Progress
April 22-27, 2006 · Montréal, Québec, Canada
The direct manipulation approach uses a drag-and-drop
(or drag-and-throw) technique with thumbnail images
of each display organized in a palette for quick and
easy access (see Figure 2b). Users select the desired
display from the pool of thumbnail images and drag (or
throw) the representative icon to the border of the
screen where the binding is to be created. Bindings are
deleted by clicking on the edge of the display where the
binding exists and dragging the icon off towards the
palette. If spatial information is important the icons can
be laid out in a manner similar to ARIS [2], however
this approach requires that the location of the user’s
personal device within the MDE is known in order to
create an appropriate map.
Our third technique manages bindings using a
conceptual map of the user’s workspace. This ‘graph’
view displays a thumbnail for each connected display
with links representing the bindings between displays
(see Figure 2c). All remaining available displays are
shown on an adjacent panel. To create a binding users
click to draw an edge between two thumbnails. To
delete a binding, users grab and drag the edge
representing a binding off the thumbnails.
Visualizing Bindings
Once bindings have been created, moving seamlessly
between displays is possible if users’ remember the
bindings (i.e. which edges of which displays are bound
together). Visualization techniques can be used to help
recall bindings. The most straightforward approach is to
place a small, coloured border at the edge of the screen
(colours can be chosen to uniquely identify particular
screens in the environment). Optionally the border may
also contain a thumbnail image or label as an additional
reminder of which display is connected to the binding.
The conceptual map approach to managing bindings
also provides a visualization of the bindings. The map is
a graph illustrating the virtual workspace (i.e. all of the
connected displays) where the user’s current active
display is enlarged and positioned in the centre of the
visualization. The nodes in the graph are continually
updated as the user moves throughout the workspace
to reflect the current position of the user’s cursor. This
approach provides a reminder of which display the
user’s cursor is currently positioned on and all adjacent,
connected displays.
Visualizing bindings becomes especially important when
several bindings are connected to one edge of a
display. The visualization must show where on an edge
the cursor must cross to move to the connected
display. In addition, users must be able to easily
manipulate these bindings, changing their size, shape
or position. This optional customization supports
personal preference which can aid recall of the binding.
Subtle tradeoffs exist between visual effectiveness and
visual overload. Borders can distract from the desktop
if made too wide, or if too small can be difficult to
acquire. Palettes and maps may be effective, but
require screen real estate. Creating a lightweight
interface that is easy to interact with requires the
designer to walk a fine line between creating a usable
interface and imposing on the user’s day to day work.
Conclusion and Future Work
The concept of lightweight personal bindings adds a
new dimension to previous approaches to collaborative
MDEs. Similar to previous work in this area, our
approach enables multiple users to simultaneously
share and interact with communal displays.
1491
CHI 2006 · Work-in-Progress
April 22-27, 2006 · Montréal, Québec, Canada
Additionally, users can create a set of personal
bindings, thereby defining a distinctive virtual
workspace that integrates their personal devices with
shared resources and reflects their perception of the
space. A further benefit of this approach is that it
effectively manages dynamic MDEs where devices and
people enter, interact with and leave the environment.
Through iterative prototyping and in-lab usage, the
Swordfish testbed has provided us with a great deal of
insight on the design of MDEs. We are currently
incorporating the ability to move artifacts between
displays into Swordfish and examining more in-context
usage by outside users. This will enable us to validate
the necessity and benefit of lightweight personal
bindings. In addition, we are further examining how to
effectively visualize bindings in a MDE.
Acknowledgements
We would like to thank NSERC and NECTAR for
supporting this research, and members of the EDGE
Lab for their assistance with this project.
References
[1] Ashdown, M., Oka, K. and Sato, Y. Combining Head
Tracking and Mouse Input for a Gui on Multiple Monitors.
Ext. Abstracts CHI 2005 (2005), 1188-1191.
[2] Biehl, J.T. and Bailey, B.P. Aris: An Interface for
Application Relocation in an Interactive Space. In Proc. GI
2004, Canadian Human-Computer Communications Society
(2004), 107-116.
[3] Booth, K.S., Fisher, B.D., Lin, C.J.R. and Argue, R. The
"Mighty Mouse" Multi-Screen Collaboration Tool. In Proc.
UIST 2002, ACM Press (2002), 209-212.
[4] DesktopRover.
http://www.neslosoftware.com/.Accessed Sept 2005.
[5] Grudin, J. Partitioning Digital Worlds: Focal and
Peripheral Awareness in Multiple Monitor Use. In Proc.
SIGCHI 2001, ACM Press (2001), 458-465.
[6] Hinckley, K., Ramos, G., Guimbretiere, F., Baudisch, P.
and Smith, M. Stitching: Pen Gestures That Span Multiple
Displays. In Proc. Adv. Visual Interfaces 2004, ACM Press
(2004), 23-31.
[7] Hutchings, D.R., Smith, G., Meyers, B., Czerwinski, M.
and Robertson, G. Display Space Usage and Window
Management Operation Comparisons between Single
Monitor and Multiple Monitor Users. In Proc. Adv. Visual
Interfaces 2004, ACM Press (2004), 32-39.
[8] Izadi, S., Brignull, H., Rodden, T., Rogers, Y. and
Underwood, M. Dynamo: A Public Interactive Surface
Supporting the Cooperative Sharing and Exchange of
Media. In Proc. UIST 2003, ACM Press (2003), 159- 168.
[9] Johanson, B., Hutchins, G., Winograd, T. and Stone,
M. Pointright: Experience with Flexible Input Redirection in
Interactive Workspaces. In Proc. UIST 2002, ACM Press
(2002), 227-234.
[10] Paek, T., Agrawala, M., Basu, S., Drucker, S.,
Kristjansson, T., Logan, R., Toyama, K. and Wilson, A.
Toward Universal Mobile Interaction for Shared Displays. In
Proc. CSCW 2004, ACM Press (2004), 266-269.
[11] Rekimoto, J. and Saitoh, M. Augmented Surfaces: A
Spatially Continuous Work Space for Hybrid Computing
Environments. In Proc. CHI 1999, ACM Press (1999), 378385.
[12] Shen, C., Everitt, K., and Ryall, K. Ubitable:
Impromptu Face-to-Face Collaboration on Horizontal
Interactive Surfaces. In Proc. UbiComp 2003 (2003), 281288.
[13] Synergy. http://synergy2.sourceforge.net/index.html.
Accessed Sept 2005.
[14] Tandler, P, Prante, T, Müller-Tomfelde, C, Streitz, N
and Steinmetz, R Connectables: Dynamic Coupling of
Displays for the Flexible Creation of Shared Workspaces. In
Proc. UIST 2001, ACM Press(200 1), 20.
[15] TeamSpot. http://tidebreak.com. Acccessed January
2006.
1492