Mobile User Interfaces
Jasper Soetendal
student Bedrijfsinformatica (1097008) - Vrije Universiteit Amsterdam
Abstract
The Mobile User Interface is not just the result of adjusting and fine-tuning the currenct Fixed
User Interfaces. The needs, level of attention and the context of a mobile user differ greatly from
those of a fixed user. In this document I'll describe the current mobile devices and their interfaces
and take a look in the future.
Context-awareness and augmented reality are the way to go to develop suitable Mobile User
Interfaces. Although a lot of research is done already, still a lot of problems exist. Some of these
problems won't be solved with the increase of computer power and will require changes in the
MUIs.
After describing the research is done for near and distant future, some 'crystal ball' future telling
will be done by sketching a scenario.
Synopsis
"Since the introduction of the mouse and graphical user interfaces, no big shifts have
taken place in user interaction. Still most devices have a crt, keyboard and mouse.
But are we now on the edge of a new era? In the society of today, mobility is a hot
topic. Why should we be bound to a desk on a fixed place?
In my paper I want to explore the possibility of 'Mobile User Interfaces'. With GSM, a tiny
display and even tinier keyboards are the standard, but what will happen with these
interfaces as UMTS will be available? Just for communication-purposes, the phones won't
really change, but what if we want a mobile office?
Virtual and Augmented Reality have a lot of possibilities, but is the world (and the
technology) ready for people wearing Heads Up Displays? Another topic of mobile user
interfaces is the use of wearable devices.
In my paper I'll give an overview of the current status of mobile user interfaces and try
to forecast the future with one (or more) scenario(s)."
Contents
Synopsis .............................................................................................................. 2
Contents .............................................................................................................. 2
Introduction .......................................................................................................... 3
1. Current Mobile User Interfaces ............................................................................ 4
1.1 Mobile Internet ............................................................................................. 4
1.2 Usability improvements on WAP ...................................................................... 6
1.3 Usability improvements on i-Mode ................................................................... 6
1.4 PDAs (Palm vs Pocket PC) .............................................................................. 7
2. Near-Future Mobile User Interfaces .....................................................................10
2.1 Agents and context-awareness ......................................................................10
2.2 User attention metaphors ..............................................................................11
2.3 Near-future Devices......................................................................................12
3. Distant-Future Mobile User Interfaces..................................................................13
3.1 User Interfaces with AR.................................................................................13
3.2 Collaboration Interfaces ................................................................................15
Pictures ............................................................................................................16
4. A Scenario .......................................................................................................17
5. Conclusion .......................................................................................................19
6. Literature ........................................................................................................20
Mobile User Interfaces
August 2002
2
Introduction
Since the introduction of the mouse and graphic user interfaces (GUI's) no big shifts have
taken place in user-interaction. Still most of us are bound to a desktop with a monitor, a
keyboard and a mouse. Off course millions of people do use a cellular phone, but the era
of mobile computing has yet to come. Nowadays bandwidth is too small and the displays
are too tiny, but the third generation GPRS and UMTS-networks are available very soon.
These techniques do have a great potential, but the question is: will they influence the
Human-Computer Interaction? Will this mobile broadband network just be another way of
connecting your laptop with the Internet, or will it be the enabler of complete new ways
to work and communicate?
In this paper we will discuss three groups of mobile user-interfaces (MUI's). The currently
available, those that will be available very soon and those that will (or will not) be
available in the distant future. The current available MUI's are those on GSM, i-mode or
PDA. With soon available MUI's we'll discuss agents and context-awareness and in the
last chapter we'll look at the current status and some future telling at Augmented Reality.
As often, the real shift towards a complete new user interface is said to take place in the
distant future. The question we'll try to answer at the end of this document is whether
this shift will take place at all. Will we be able to ignore the legacy of the "2D-flatscreenmouse-and-keyboard-user-interaction" and create a brand new way of user interaction,
or will we continue to evolve the current user interfaces with tiny improvements?
Mobile User Interfaces
August 2002
3
1. Current Mobile User Interfaces
1.1 Mobile Internet
The face of the Internet is changing. Within just a few years, more people will be
accessing the Internet from mobile phones, PDAs, pagers, wristwatches and a variety of
information appliances others than desktop PCs. We are quickly approaching the mark of
1 billion mobile phone users and, while today only a fraction of existing mobile phones
are Internet-enabled, the situation is fast changing [Sadeh, 2002].
Although the network standards differ from continent to continent and even from country
to country, the restrictions of hardware and network are equal: small memory capacity,
tiny displays, less powerful CPU, limited input method, slow speed and unstable
connections.
Unlike PC user interface, keyboard and mouse are not suitable for mobile devices.
Rather, intuitive operation like a four-button interface (cursor forward, cursor back,
select, and back buttons) is required. So, some kind of accessibility considerations,
should also be taken into account.
All these limitations request a different presentation of the accessed websites. Websites
for the mobile Internet have to be redesigned and rewritten. Just copy and pasting the
current content to a mobile website should be avoided. Even more then on the wired
Internet, it's important to quickly provide personal and locally relevant information to
users. A mobile Internetsite should easy to mine and information will have to be clearly
presented.
Even on a higher level, the Internet is changed. On a mobile phone it's impossible to try
to find a site by Google.com, browsing trough tens or hundreds of results.
Fixed Internet-users will access most of their pages by a search-engine, mobile
Internetusers have to rely on mobile portals, mostly compiled by mobile operators. In
this sense, the open-platform of the common Internet is somewhat decreased, giving the
mobile portals the power to decide what sites will be visited by the mobile users.
Although this is not strictly related to the mobile user interface, it is a result of the
restrictions of it.
All these restrictions are guilty to the fact that the mobile Interface haven't had it's real
break-trough yet. In Europe WAP is available for a couple of years now, but the number
of users is dissapointing. Even a better, packet-switched network like i-mode, launched
in The Netherlands april 2002, hasn't more than 11.000 users right now.
According to a study of the Nielsen Norman Group, carried out in late 2000, even
mundane tasks such as retrieving newsheadlines, checking the local weather forecast or
accessing TV program listings were taking significantly longer than most users could
tolerate. Worse the times failed to significantly improve even after a week of daily use,
as users continued to confront slow connections and poorly designed sites. Overall, 70
percent of the study participants indicated that they did not see themselves use WAP
within the next year.
By and large, the problems uncovered by this and other similar studies of early WAP
users fall into four categories: Technical limitations, poor site design, poor content and
device limitations.
Example
We will discuss these problems on the basis of a case study of an i-mode site,
wehkamp.nl, the biggest mail-order company of the Netherlands. There's only one thing
to do at Wehkamp, so let's say an i-mode user wants to shop ('winkelen') at the
Wehkamp site. He (or she) will have to go trough the next four screens:
Mobile User Interfaces
August 2002
4
As you can see, two of these screens are totally unnecessary. Screen 2 does introduce
the user to Wehkamp with a nice picture, but the only effect is unnecessary waiting time
and consuming unnecessary bandwidth. Even the third screen is redundant; 90% of all
Wehkamp-visitors will visit the site to shop (option "1. Winkelen"). All options of the third
screen could be part of one new option on the fourth screen, like "Search" or
"Consumerdesk".
Another example will show a better design, the ANP site, the Dutch Press Agency, wich
most users will visit to view the latest news.
To fulfil this task, the user only have to press one button: "2. ANP Nieuws". The user
directly enters a screen where the current headlines are scrolling at the top of the page.
If the user wants some specific news, he can choose a topic (Sports, Showbiz, Financial,
etc.)
Identity
As we can see, it's very important to reduce the number of screens a user has to go
through. Also the number of graphics and images has to be reduced, because they
consume bandwith and cause more waiting time.
But, how it is possible to create a company-identity on the mobile Internet? On i-mode
most news services look-a-like and all banking sites are the same. Can you really
differentiate sites on such a small screen?
The answer is that there needs to be differentiation between services. But the
differentiation cannot come through graphic design; it has to come from a better writing
style, a style that's optimised for very, very short content. This will lead to a renaissance
of appreciation for writers who are good at creating small pearls of content.
Mobile User Interfaces
August 2002
5
1.2 Usability improvements on WAP
One protocol to access the mobile Internet discussed in the previous paragraph is WAP,
the Wireless Application Protocol. This protocol is independent from the network it runs
on, but unfortunately it's currently much confused with WAP over GSM, just one way to
implement WAP.
WAP content was originally required to be written using what is now referred to as the
legacy Wireless Mark-up Language (WML) and the WML Scripting Language, WMLScript.
With the release of WAP 2.0 in August 2001, content can now also be written in XHTML
Basic, a subset of the XHTML language recommended by W3C.
Deck of cards
A new concept introduced with WML is the deck of cards metaphor. This concept is
developed to meet the restrictions of the mobile Internet. Many user applications can be
broken down into small sets of interactions with the user. Each interaction involves just a
fairly small number of options and is presented as one card. The deck of cards concept
consists of bundling together these sets of interactions into a deck of cards and sending
these cards all at once to the mobile device.
Earlier we discussed that waiting times are one of the most frustrating experiences over
mobile networks. With this concept it is possible to significantly reduce the number of
times the user will have to wait for data to be transferred over the network.
Wap 2.0
With version 2.0, WAP moved toward adopting widely accepted Internet standards. The
W3C-defined XHTML Basic standard has been adopted as the basis for WAP 2.0. XHTML
Basic is the mobile version of XHTML 1.0, on which the WAP Forum based its XHTML
Mobile Profile.
WAP CSS is the mobile version of cascading style sheets (CSS) that has only those
features of CSS that are relevant to the mobile environment. XHTML and CSS put more
formatting power in the developer's command. Using XHTML and CSS, you could even
display the same document on different devices using distinct presentation capabilities.
1.3 Usability improvements on i-Mode
Another, somewhat newer, protocol used to access the mobile Internet is i-mode. i-mode
was launched in 1999 by NTT DoCoMo in Japan and gained a 30 million user-base in only
3 years. Nowadays 50 million Japanese people do use i-mode, in a land with 120
inhabitants. i-Mode requires an packet-switched network, allowing an always-on data
connection. The user is only charged for the number of packets sent and received over
the Internet time. i-Mode is launched in Europa in april 2002, starting with operator KPN
in the Netherlands. In Europe GPRS (General Packet Radio Services) is used as carrier for
i-mode.
cHTML
i-Mode content is required to be written using cHTML, a subset of the common HTMLstandard. Any site on the Internet, written in cHTML (or cHTML-compatible HTML) can be
viewed on an i-mode handset.
cHTML is a subset of HTML, many HTML-tags are not allowed in cHTML. But to meet the
characteristics of the i-mode handsets, some new options are introduced.
Every link on a page can be linked to an accesskey on the keypad. As you can see at
the pictures from the Wehkamp and ANP-site, by just pushing, for example, the '1'button, the user can activate a link.
An icon of the accesskey accompanies these links. These icons are called "Emoji Icons"
which can be used to enhance an i-mode site without using a lot of additional bandwith.
There are about 150 emoji icons embedded on the handsets, representing accesskeys,
weather and horoscope symbols and all sorts of other icons.
Mobile User Interfaces
August 2002
6
Another improvement is, that it's possible to link a link to a telephone number instead of
a webpage. So with the click of one key, it's possible to phone the contact person of a
website.
1.4 PDAs (Palm vs Pocket PC)
Mobile Internet is not the only application for mobile devices. You don't have to be
connected to the Internet when you're on the move! For the use of off-line applications
PDAs have been out there for years. Millions of people do use there PDAs daily for the
function there named for: personal digital assistant.
The PDA-market is divided into 2 segments. Although the PDA-functionality is
implemented on a wide range of hardware devices, but most devices currently run one of
two operating systems, and consequently present the user with one of two user
interfaces. The two dominant operating systems in the marketplace are Microsoft
Corporation’s Pocket PC (Windows CE 3.0) operating system and Palm Computing’s
PalmOS v3.5. Above you'll see a picture from two devices running on these operating
systems.
PDAs have bigger screens than GSMs, some even have colour screens, and offers better
user input possibilities. Both Palms and Pocket PCs use a touch screen interface and a
stylus for handwritten input. With these devices the applications can be more advanced
than those on cellular phones. Keep track of your appointments on a date book, checking
your email and create some notes are maybe very complex on a cellular, with an PDA it's
very easy.
With screensize up to 240x320 it's even possible to display simple webpages. There are
different webbrowsers available for both PDA platforms.
As long as the display of mobile phones won't enlarge, a PDA is the most convenient way
to surf the mobile Internet today. Most PDA hardware providers can deliver expansion
packs to turn your PDA into a wireless Internet device. Even some products like the Palm
i705 and the Handspring Treo are a PDA with integrated cellular phone (or vice versa).
Usability study
In this paragraph we'll take a look at these two different operating systems, because it
turned out that the differences between these to systems are very interesting.
One of the major differents is that the Palm OS was designed from scratch, especially for
handheld devices, while the Pocket PC interface attempts to provide users of Windows on
desktops with a reasonably familiar, or at least recognizable interface.
According to a study at the Auburn University [Hübscher-Younger et al, 2001]:
"While we found that devices running the two operating systems were physically very
similar and provided very similar sets of built-in PIM functionality, the design
philosophies behind the two interfaces turned out to be completely different. While the
Palm OS generally provides a single way, or perhaps two ways to perform a given task,
Mobile User Interfaces
August 2002
7
or to navigate from one point in the interface to another, the Pocket PC devices possess
many different ways of accomplishing most tasks."
The approach taken by the Palm OS is very consistent, based on a very simple
conceptual model. Because of the simplicity of this interaction model, it's very easy for a
user to predict the result of any action he'll take. So, for the user it's very easy to find
one of the very few the right ways to perform a task. This approach leads to a very
usable interface.
The approach of the Pocket PC, which is quite contrasting, is what called the permissive
interface. Permissive interfaces are based on the concept that every reasonable user
action should lead to a reasonable result. For instance, when add a new meeting to your
calendar, you could first create add a new event and drag it to the right date, or you
could search the right date, select the right time and add the meeting.
The design of a permissive interface is based upon less stricter conceptual models then
the Palm OS is. Such an interface may even support several conceptual models, but it's
important that too much different conceptual models can disorient the user.
With these differents in minds [Hübscher-Younger et al, 2001] measured the time it took
for doing a set of task on a Palm OS device and a Pocket PC device. The results are very
surprising:
These results imply that the participants are significantly more efficient with the Palm OS
than with the Pocket PC operating system. This is very surprising, because a significant
part of the participants were used to the common Windows desktop. Although the design
philosophy of Pocket PC was to create an interface common to the common Window
Interface, users still have difficulties performing trivial tasks.
The Palm OS, which is designed for mobile use seems to be much easier. One answer
could be the fact that the context of a mobile user differs completely from the context of
a fixed user. In the next paragraph we will try to answer this question.
New Usage scenarios
Mobile devices as cellular phones and PDAs open the door to a number of services and
applications that would be simply inconceivable from a desktop PC. Combined with
mobile Internet the number of new possible services is even bigger.
But it's very important to understand that the user requirements and context of a mobile
user differs completely from a fixed user. A mobile user isn't fixed on his screen al the
time, he's attracted by events in his surrounding, like along racing cars, traffic lights,
(coincidental) meetings, etc. So many of the services for mobile use will be revolve timecritical needs that require short, point-to-point interactions. A mobile user would not
want to use its phone for complex Internet session aimed at composing a new computer
at Dell.com. But a mobile user would much more readily use its device for arranging a
meeting, checking the news headlines or a favourite stock.
The approach Palm has chosen for the design of it's user interface is much more
associated with this new usage scenario than the Pocket PC is. Simple interfaces, to the
point, easy interaction. Although the Windows Desktop Interface is perfect to do more
Mobile User Interfaces
August 2002
8
complex tasks, but downgrading this interface to the Pocket PC OS is not the way to go,
because of the surroundings and the new usage scenarios of the mobile user.
According to [Sadeh, 2002]:
"Many experts believe that we only started to scratch the surface of m-commerce
(or: mobile Internet, or: mobile devices) and that many more services and usage
scenarios remain to be invented. The more futuristically inclined among them
envision pervasive computing scenarios where new technologies for
personalization and context-awareness and new modes of interaction will make it
possible to identify every wish and present us with adaptive services aimed at
facilitating all aspects of our daily lives."
With this vision of the future we will end this chapter, and take a look at the near-future
Mobile User Interfaces.
Mobile User Interfaces
August 2002
9
2. Near-Future Mobile User Interfaces
Wearable computer with HUD from Xybernaut.
It's very difficult to decide what's near future and what's distant-future. In this chapter we will
discuss context-awareness and agents. There's a lot of study about this topic right now. But no one
can tell whether this will be implemented earlier than the topics I'll discuss in the next chapter. I
believe that context-awareness is more near future than augmented reality (chapter 3) is. But
maybe this order will, about a year of 5, turn out to be unchronological after all.
In this chapter we will discuss the near-future mobile user interfaces. We assume that in
the near-future mobile broadband is rolled out with at least 2 Mbps speeds and we have
hardware, with capacities like current pc's, small enough to wear everyday. In other
words, we don't have any network or hardware problems. Bandwidth is broad enough,
hardware is powerful enough. The only problem we still have, is the usability problem.
We'll have to understand that performing tasks on a computer isn't the users primary
task. While he's performing these tasks he's at work, driving through the traffic, waiting
for the subway train, or walking through a park.
2.1 Agents and context-awareness
Mobile use of applications and services requires a new way of user interaction. Important
keyword in exploring this new way is 'context-awareness', your mobile device should be
aware of the context of the user. Context-awareness is about capturing a broad range of
contextual attributes to better understand what it is that the user is trying to accomplish,
and what services or applications he or she might possibly interested in. For example,
when a user is looking for a restaurant, the device should offer information about
restaurants that are nearby the user, satisfy the wishes of the user (price, sort of food)
and are open at that moment.
Using the Internet on a fixed PC, the user would search Google or the Yellow pages for
'restaurants' and manually browse through hundreds of results until he/she finds the
right place for dinner. Standing on the corner of a street, the user doesn't have time or
equipment to do this.
The solution to deliver this context-awareness is the use of agents. These agents should
be aware of the preferences of the user (what does he/she likes), the calendar of the
user (what is he/she doing, what will he/she do within an hour), the current location of
the user, social context information (who are the users friends, colleagues, classmates)
and other context information such as current time and weather.
When all the information about the users context is available, it can be accessed by a
collection of personal (virtual) agents, each in charge of assisting the user with different
tasks, while locating and invoking relevant Internet services identified through service
registries. One of these agents, let's call him the restaurant agent, could offer the user
suggestions about where to have it's dinner. This agent will look up the date book of the
Mobile User Interfaces
August 2002
10
user to see how much time is available, and depending on the time and user preferences
this agent will select only a few restaurants nearby the current location of the user. If it's
raining, the distance the user will have to walk will be minimized, when it's sunny
restaurants an outdoor café can be chosen, etc.
It's clear that the challenge of developing an interface like these agents is not only about
collecting all these preferences and information about the context. The difficulty lies on
deriving the user current objectives and preferences out of these contextual attributes.
This won't be implemented overnight.
But agents can be used for much less complex tasks. When a user is busy, he won't be
continuously confronted with new (email) messages. A mail agent could measure the
importance of every incoming message and decide to confront the user with it
immediately, or save it for a later moment when the user has more time for reading
his/her mail.
2.2 User attention metaphors
With the devices currently available, there is not much 'space' for these agents. The
current devices use the 'windows-icons-menus-pointers' interface, the WIMP-interface for
short.
Assumptions inherent in this metaphor make it inappropriate for mobile computers. First,
WIMP interfaces see interaction with the computer as the user’s primary task. With
mobile computers, interaction with the real world is the primary task. Second, WIMP
interfaces assume the user has plenty of screen real estate and a pointing device.
Although we assumed that mobile computers are powerful, they still may have limited
screen space, and their input devices may have to be used with one hand, when out of
the user’s view, and at an arbitrary orientation—making pointing devices generally
unsuitable [Billinghurst, 1999]. [Rhodes, 1998] introduces more fitting metaphors, the
ambient and agent interfaces. The concept of this interfaced is based on the fact that
every (mobile) application requires a different level of user attention.
The figure above shows two continuums that measure the computer user’s (a) input and
(b) output attention. Sensors like microphones or cameras require very few user
attention, where direct manipulation (like the conventional WIMP-interface) enforces a
high cognitive and perceptual load to the user. Full visual and audio multimedia output
demands a lot of user attention, but agent technologies automatically act on the user’s
behalf and so require little.
In between these extremes are ambient interfaces, changing from very attentionconsuming to independent agent processes, which use low attention output modalities
such as background noise and augmented reality interfaces that overlay information on
the real world.
The chosen interface is dependent on the level of attention an application requires, and
the level of attention a user can give to an application on that moment. When driving on
a highway, an agent (like the mail agent we met the previous paragraph) or any other
Mobile User Interfaces
August 2002
11
low-level ambient output would be most appropriate for reading/filtering new incoming
(email) messages. But when the user leaves the highway, and enters an unknown city,
searching for restaurant or hotel, a more high level ambient output will take over, like an
augmented reality overlay of navigation information on the real world.
2.3 Near-future Devices
In the previous 2 paragraphs we've introduced some concepts on mobile user interfaces.
These are very conceptual, and do not result in physical devices. In this paragraph we
will quickly discuss some hardware for near-future mobile devices.
Xybernaut (www.xybernaut.com)
Since its founding in 1990, Xybernaut has pioneered the research, development and
commercialization of wearable computer technology, hardware and related software. This
exploration results in very futuristic hardware that can actually be bought at Xybernaut
Website. For example, the Poma at the picture at the start of this chapter, is a 128 Mhz,
32 Mb Ram computer with a headmount display. This display is built for optimum comfort
and functionality. The one-inch, full colour, 640 x 480 SVGA viewing screen sits below
your eye, weighs approximately 3 ounces and provides a viewing area resolution similar
to that of a desktop monitor from two feet away.
Fieldmouse
[Masui & Siio, 2000] introduce the Fieldmouse: part of a proposed new interaction
technique called the Real-world GUI, where users can control real-world appliances just
like performing GUI operations on graphical computer terminals. A Fieldmouse is a
combination of two devices: an ID-scanner (like a barcode scanner) and a 3D-motion
sensing device. Using a Fieldmouse various GUIs of various devices can be controlled,
just like a computer can be controlled with a conventional mouse. A FieldMouse can use
the barcode reader to tell what and where it is pointing at, and measure the relative
movement of the device after detecting the barcode. Fortunately, many of the GUI
widgets are based on point-and-drag operations and require only these information for
interaction, and they can easily be simulated by the FieldMouse. For example, a barcode
symbol can be used like a pulldown menu, by using the amount of the movement for
selecting items. If the system interprets the amount of the relative movement as an
analog value, it works just like a slider or a scroll bar.
UMTS Devices
Within years, or maybe months, 3G mobile networks will be rolled out in Europe. With
UMTS data transfers up to 2 Mbps are possible. This opens the door for total new
applications, as we mentioned before.
One of the critical success factors of UMTS is the availability of handhelds for these
UMTS-networks. Hardware providers like Nokia, Siemens, Sony-Ericsson, Motorola,
etc. are busy designing new devices.
The picture at the right shows a concept model of a Nokia
phone. With UMTS different devices will integrate into
one new communication devices. PDAs, videoplayers,
webcams and phones will combine in one 3G-device.
In Germany, Mercedes-Benz and Siemens have
even introduced their first car with full UMTS
Internet access.
With these devices, combined with the 3G networks and user interface concepts as user
agents and the ambient interfaces, the future is near, a future with an immense number
of new possible services. Let your personal agents do the work, videoconference with
your colleagues or explore an unknown town by augmented reality navigation: The future
is near.
Mobile User Interfaces
August 2002
12
3. Distant-Future Mobile User Interfaces
I graded augmented reality as distant future. However, in experimental studies and even in some
maybe futuristic, but really existing applications, augmented reality is already implemented.
But before augmented reality is widely accepted and has overcome the problems mentioned in this
chapter, many years will pass.
Forecasting the near future is very difficult. Distant Future Telling is almost impossible.
This is not the place to play Jules Verne or Nostradamus and foresee some very futuristic
user scenarios. To concentrate on one development instead of summing up some vague
scenarios, I'll discuss augmented reality in this chapter.
Augmented reality (AR) is an intermediate between reality and virtuality. The basic goal
of an AR system is to enhance the user's perception of and interaction with the real world
through supplementing the real world with 3D virtual objects that appear to coexist in
the same space as the real world.
Although AR is not restricted to particular display technologies as Head-Mounted Displays
(HMD), most current concepts or demonstrations of AR use these HMDs. We've shown a
HMD earlier in chapter 2, as part of the Xybernaut Poma device. There is a lot of research
about AR, and some tests with real AR-systems are been done already.
3.1 User Interfaces with AR
Until recently most interface of AR-systems were based upon the same interaction design
as the WIMP-interface, the desktop metaphor. One main trend in interaction research
specifically for AR systems is the use of heterogeneous designs and tangible interfaces.
Heterogeneous approaches blur the boundaries between real and virtual, taking parts
from both worlds. Tangible interfaces emphasize the use of real, physical objects and
tools. Since in AR systems the user sees the real world and often desires to interact with
real objects, it is appropriate for the AR interface to have a real component instead of
remaining entirely virtual [Azuma, 2001].
In such an interface, the user can interact with virtual object as if they were real.
Touching, pushing, tilting and dropping are possible and have results like these objects
are real.
This concept of interaction allows multiple scenarios and applications. One example is
that a user or a group of users holds a physical board upon which virtual controls are
drawn. These controls can be controlled as if they are tangible. Another way of testing
these examples are games. In the AR 2 Hockey Game, two users are playing a air hockey
game by moving a real object, representing the paddle.
Another trend in this design philosophy is the possibility of collaboration amongst several
simultaneous users. When collaboration is possible, a group of users can control a
tangible interface together. When one users moves an object, this object will move in all
the users augmented reality.
The Human Factor
OK, now we know what's possible technical. But how do human users react on
augmented reality? Experimental results from human factors, perceptual studies and
cognitive science can help guide the design of effective AR systems in many areas. In a
study by [Drascic, 1996] 18 design issues, current limitations and (future) problems are
discussed. These are divided in implementation errors, current technological limitations
and the hard problems. We won't discuss them all 18, but we'll take a snapshot of some
of them:
 Calibration errors & mismatches
In order for a graphic or video image to be scaled properly in space, the calibration
parameters, which determine the visual angle, perspective, and binocular parallax of that
Mobile User Interfaces
August 2002
13
image relative to the viewer, must be accurately specified. This is extremely difficult, and
when it's incorrectly done, errors like depth distortion can appear; Objects appear in
augmented reality on another place then they really are. These depth distortions can
affect performance in many different ways. In situations where the viewer must switch
back and forth between the real and virtual environments, these mismatches in coordinate systems can lead to disorientation and other performance deficits.
 Restricted field of view
When walking around in the real world, we have a very broad field of view. As long as we
can see important parts of our world, like the floor and our own body, we can solidify our
perception. When these parts are invisible due to a restricted field of view, as result of
hardware restrictions, we lose a great deal of confidence in our understanding of the
world. Even very simple actions, like walking around can be very difficult in augmented
reality with a head mounted display with a limited field of view.
 Size & Distance Mismatches
Differences in image resolution and clarity of virtual objects can be interpreted as
accommodation and texture perspective depth cues. Likewise, image brightness and
contrast mismatches can be inappropriately interpreted as luminance and aerial
perspective depth cues. These false can cause otherwise identical video, graphical, and
directly viewed objects to appear to have different sizes or to be at different distances,
even when the geometry of the images suggests they this should not be so.
 Absence of shadow cues
Even the most well-developped system won't be able to automatically capture the
environmental illumination information and calculate realistic shadow of virtual objects to
fall on real objects. This has a impact on user perception, it can impair user performance.
Besides these problems of [Drascic, 1996], [Rolland and Fuchs, 2000, quoted in [Azuma,
2001]] performed a detailed analysis of the human factors in connection with head
mounted displays. These factors can reduce the user performance when working wit
these displays:
 Latency
Delay causes more errors than all the other factors described in this document. Delays as
small as 10 milliseconds can make a significant differences on the users performing tasks
that require (some) precision.
 Depth perception
Rolland and Fuchs sum up factors mentioned earlier studied by [Drascic, 1996]. They
conclude that accurate depth perception is arguably the most difficult type of registration
to achieve in an AR display because many factors are involved. They're quite right,
because all 18 factors described by Drascic have influence on the users depth perception.
 Adaptation
Frequently use of augmented reality systems can lead to adaptation that has negative
influence of the users performance when the AR is 'turned off'. In a study with HMDusers, the subjects exhibited a large overshoot in a depth-pointing task after the HMD
was removed.
 Long-term use
Long use of AR displays causes discomfort, both in eye strain and fatigue. AR displays
that are uncomfortable may not be suitable for long-term use.
So, now we've summed up quite some problems with the 'human factor' we can conclude
that there's still a lot of work to do. Beside form these technical related human problems,
there remain a lot of social problems. Will users accept a world where real objects and
virtual objects are difficult to distinguish? For some professional work AR can be really
helpful, but will it be used in peoples everyday life? And, will it socially be accepted that
Mobile User Interfaces
August 2002
14
you're walking around with a (bulky) head mounted display? Although these devices will
be more and more sophisticated, well-designed and tiny, they are still visible.
I can't answer these questions right now. The future will tell.
3.2 Collaboration Interfaces
As quickly mentioned before, collaboration on AR is a very interesting topic. A group of
users in one room can collaborate on a shared augmented reality environment. A change
to this environment from one user will result to changes in this shared environment. But
the concept of collaboration isn't limited to co-located users. The collaborating users can
be miles away from each other. Think about distant operating, a group of surgeons all
over the world can help one physical present surgeon with a difficult operation. Or, less
complex and risky, two player can play the mentioned AR 2-Airhockey game even when
they are not on the same location.
These collaboration possibilities demand for very specific interfaces. These interfaces
consist of two parts, a shared part and an individual part. The shared part displays the
shared environment, where the individual part can also present different information to
each user, tailoring the graphics to each users interests and skills and supporting privacy.
Where this individual part can be relatively easy implemented, there will occur some
problems with the shared environment. Since all virtual objects are displays dependent
on each users view on the world, it may be quite difficult to ensure that each user
understands where the other users are pointing or referring to. In some cases, the
designers attempt to overcome this problem (and possible registration problems) by
rendering virtual representations of the physical pointers, which are visible to all
participants. However, this does not help when users gesture with untracked hands or
refer to objects descriptively (e.g., The lower left part of the object.).
Still a lot of work needs to be done to implement well-performing augmented reality. In
the first two chapters mainly solutions were discussed. This chapter is more an
enumeration of problems. This is because there are still a lot of problems that aren't
solved yet. Although there's are a lot of applications that offer a solution to a small
subset of these problems, an application that offers a solution to all, or at least most of
the problems, is still unavailable. Again, future will tell if there will be applications or
services that create an augmented environment where virtual and real objects are
indistinguishable.
In chapter 4 I'll create such a system, regardless how the problems are solved. Chapter 4
gives a future scenario, a futuristic Verne or Nostradamus like vision of the future.
But first... some pictures.
Mobile User Interfaces
August 2002
15
Pictures
By reading all those books, papers and websites, I saw an immense number of very
futuristic pictures of future devices or scenarios. I'll end this chapter with some of these
pictures, just for futuristic fun.
Mobile User Interfaces
August 2002
16
4. A Scenario
The year is 2012, ten years from now. How will we interact with our computer and how
will we interact with the real world. In this chapter I'll describe a scenario of mobile user
interaction in the year 2012. It isn't a strong, scientifically built theory, but after days of
reading a pile of papers, a glimpse of the future can be caught. It's a personal vision,
based on the findings of the papers I've read.
Today, July 11th 2012, the size of great computer power can be reduced to the size of a
matchbox. Computers with gigantic processor power and memory (related to the current
hardware) can be worn in a jacket or in a backpack. For back-up, synchronization and
information, this mobile device is connected by 4G Networks to the Internet and the
users home desktop computer.
The interface of the mobile computer is presented as augmented reality by a
headmounted display. This display is worn almost invisible on (sun)glasses. (In fact, a
real fashion of futuristic shaped glasses for HMDs exists today.)
The Mobile User Interface (MUI) of this wearable device consist of three parts:
1. A gigantic collection of independent, individually acting user-agents.
2. A classic see-through WIMP-interface, overlaying reality.
3. Augmented reality, virtual objects integrate indistinguishable with reality.
Highly context-aware user-agents
Most of the work is done by the highly context-aware user-agents. We discussed these
agents in paragraph 2.1. These agents are working behind the screens on tasks, enforced
by the user or not, only appearing to the user to report the results. These agents appear
as characters, real or cartoon-a-like persons, to the user. Interaction with these agents is
done normal speech. (OK, this may be weird in a full subway-compartment, but hey! it's
2012. Besides, even nowadays people on the street are talking to an invisible person by
phone.) Because these agents appear as characters, know (almost) everything about you
and your environment and are interacting with speech, they seem like virtual friends.
Whenever you want a nearby restaurant you call "James!" to search for it. And when
walking through the street, Jane, your shopping-agent, may point out a very interesting
summer-sale offer at the shop you're passing.
Classic interface
Of course, not all tasks can't or don't want to be performed by agents. For these tasks a
more conventional user interface is available. A see-through screen and a virtual
keyboard are visible on the HMD. Now you're able to use a computer or the Internet like
your used to on your fixed computer. Instead of using a mouse you can just point at the
screen and instead of a real keyboard you're using a virtual one. This classic interface
isn't as futuristic as the other two, but will turn out to be as useful as them.
Augmented reality
With augmented reality, the real world itself
becomes the user interface. Objects, virtual or
real, can be touched for information or
performing a task. The user, tracked by a
centimeter-level real-time–kinematic global
positioning system(GPS) and an
inertial/magnetometer orientation sensor,
experiences the world augmented by multimedia
material displayed on his HMD. The movement
of the hands and fingers are measured with
Mobile User Interfaces
August 2002
17
almost invisible hand gloves, rings or (implemented) chips. With this equipment virtual
objects will appear indistinguishable from real objects in the users environment. Which
virtual objects are inserted is dependent from what the user wants. If he's on the way to
a new, unknown location, navigational information, like shown on the picture can be
presented. Or when walking through a historical site like Rome, buildings will be
presented as the existed in the history.
But for more everyday tasks, buildings or objects can be overlaid with labels giving
information about them. The possibilities are almost infinite and every user will choose
his own way of composing the Augmented Reality.
Possible Applications
Think about all the possible applications with an MUI like this! User-agents like friends
will inform you about everything you want to know and augmented reality can display
everything you want to see.
 Navigation
Get lost in an unknown city or area is impossible.
With AR navigational info can be presented to guide
you to your destination. And when you want an
understanding of the surroundings, you'll ask for a
virtual, 3D map. You can tilt, rotate and inspect this
map as much as you want.
Interested about the social or historical context of
your surroundings? Just ask an agent to be your
agent telling you "at your right hand you can see the
famous Van Gogh Museum. A huge collection of...".
 Social info
Being part of a community like MSN, your personal information and interest can be made
public to the world. Because of the positioning systems, you can make your location
public to the world or a selected subset of people. Looking for a friend, an arrow or
navigational info can tell you how the find him/her.
Or think about a party, where most of the party people are part of this community. Let's
tell your AR-interface to highlight all 20-year old, single girls, with the similar interests or
profile.
 Shopping
We've introduced Jane, our shopping assistant, but imagine the collection and prizes of
most stores in a shopping-centre are available on the Internet. Jane can select the most
interesting products of every store, and display them above or near every shop. Just
looking at a store will inform you about the prize of the hardware your looking for.
 Business
Although the examples above are nice, most applications will have a more businesslike
character. Technicians will never have to browse through manuals again, policemen can
identify all by-passers, etc. Again we'll realize that the possibilities are infinite.
The real-time information of the environment, user-context and access to the Internet
opens the door to commercial applications and services that can't be even thought of
right now.
When I was writing this scenario, I was astound of the immense number of possible
applications can be thought. The combination of the three described interfaces seems
perfect to me; Agents are your best (virtual) friends and the real world is the MUI.
Mobile User Interfaces
August 2002
18
5. Conclusion
Early mobile marketing campaigns trumpeting the arrival of mobile computing in
everyone's pocket led many to believe that the fixed Internet experience they had grown
accustomed to could simply be recreated on their mobile devices. But nowadays all these
device have tiny screens, low memory and processor capabilities, let alone the awkward
user input possibilities.
We have to realise that designing a Mobile User Interface isn't about just adapting and
fine-tuning the current Fixed User Interfaces. The context and the attention of the mobile
user differs greatly from the fixed user.
Current mobile (internet) devices are nice, but they are mostly a downgraded version of
the fixed equivalent. The real shift to Mobile User Interfaces will still have to take place.
I've browsed the websites of many hardware producers; If it's up to them, this shift
won't take place in the near-future. Upcoming UMTS-devices and PDA's are still trying to
close the gap to the fixed devices. But in my opinion this gap will have to be broad out.
Highly context-aware agents and augmented reality is the way to go to cope with the
different context of the mobile user.
But the development of these new interfaces did not left the laboratories at this moment.
A lot of research is done, but there still remain a lot of problems to overcome. As
computers increase in power and decrease in size, most of these problems will be solved.
But a scenario like I sketched in the last chapter won't be implemented overnight. In fact
I doubt whether 2012 wasn't a too optimistic estimate.
Most important is that we'll have to understand that a MUI isn't a 'FUI'. The needs and
characteristics of a mobile person differ greatly from these of a 'fixed' person. New
design philosophies are needed, new viewpoints will have to be created.
The question the introduction of this document ended with can now be answered:
One day we will al walking around in our own augmented reality, communicating and
interacting with real and virtual objects and person like they are all real.
Amstelveen, July 11, 2002
Jasper Soetendal
Mobile User Interfaces
August 2002
19
6. Literature
- "Handheld and Ubiquitos Computing", Thomas, Gellersen,
2nd International Symposium, HUC 2000, ISBN 3-540-41093-7
- Real World GUI's, Masui & Siio (p. 72-84)
- "M-Commerce", Norman Sadeh
Wiley Computer Publishing, 2002, ISBN 0-471-13585-2
- "Mixed Reality - Merging Real and Virtual Worlds", (Y. Ohta and H. Tamura, eds.),
Springer-Verlag, 1999, ISBN 3-540-65623-5
- "Collaborative Mixed Reality", Billinghurst, Kato
- "Wearing It Out: First steps Toward Mobile Augmented Reality Systems", Feiner, MacIntyre, Höllerer
- Proceedings ISWC ‘97 (Int. Symp. on Wearable Computing),
Cambridge, MA, October 13–14, 1997, ISBN 0-8186-8192-6
- "A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban
Environment", Steven Feiner, Blair MacIntyre, Tobias Höllerer, Anthony Webster (p. 74–81)
- "An Experimental Comparison of Two Popular PDA User Interfaces", Hübscher-Younger, Hübscher, Chapman
Department of Computer Science and Software Engineering, Auburn University, September, 2001
- "Wearable devices: New Ways to Manage Information", M. Billinghurst & T. Starner
Computer (IEEE Computer Society), january 1999
- "Recent Advances in Augmented Reality", Azuma et al.
Computers & Graphics, November 2001
- "Exploring MARS: Developing Indoor and Outdoor User Interfaces to a Mobile Augmented Reality System",
Höllerer, Feiner, Terauchi, Rashid, Hallaway, Computers & Graphics 23(6), pp. 779-785, 1999
- Proc. SPIE Vol. 2653 "Stereoscopic Displays and Virtual Reality Systems III"
1996, ISBN 0-8194-2027-1
- "Perceptual Issues in Augmented Reality", Drascic and Milgram
- "Mobile Multimedia Communication 1996-2000", C.Rodríguez Casal and F. Schoute
MMC Final report, Information and Communication Theory Group, TU-Delft, The Netherland, September 2000
- "De gebruiker en de multimediale belofte", v/d Anker, Warmoeskerken, Arnold
Tijdschrift voor Ergonomie, 25 (3), p. 74-84, 2000.
- “WIMP Interfaces Considered Fatal”, B Rhodes
http://www.hitl.washington.edu/people/grof/VRAIS98/Rhodes.html.
- "3D City Info - a Near-future Application of 4G Services", Ismo Rakkolainen
Most of these papers can be found both on the Internet or in (proceeding) books. Only some of above
papers/books are supplied with an ISBN-number. Most of the other papers will have an ISBN-number, or are
published in a proceeding book with an ISBN-number, but I couldn't find these numbers.
Mobile User Interfaces
August 2002
20