Ambient Intelligence: Physical interaction
Guido van der Zanden
[email protected]
ABSTRACT
Keywords
Ambient Intelligence, physical interaction, tangible interfaces
1.
INTRODUCTION
Ambient Intelligence (AmI) is a paradigm for the vision on digital
systems that was developed in the late 1990s [1]. The term itself
contains the two cornerstones of this paradigm. In short, ‘ambient’ stands for electronics that will move into the background.
Instead of ‘grey boxes’ such as computers and televisions, the
electronics will be embedded and disappear in our environment.
In this way, technology will become an unobtrusive part of our
environment. The other cornerstone, ‘intelligence’, is the second
step in AmI development. If technology can be moved into the
background, interfaces have to be developed that provide “easy,
intelligent and meaningful interaction”. In order to achieve this
goal, the systems needs to be personalised, adaptive to changes,
anticipatory and context-aware. Instead of letting the user adapt
to the technology, technology has to adapt to the user [2].
Although the description above is far from complete, it contains
the two most important ‘key-words’ in this paper, namely ‘environment’ and ‘interaction’. When technology disappears into
our environment as stated above, virtually everything can become
some sort of intelligent object: walls, chairs, tables et cetera can
all be used to embed technology into. Those objects which are in
our environment for hundreds of years suddenly transform from
static into intelligent objects that can react to changes in our environment and can be a part in human-computer interaction.
Those intelligent objects can be smart or intelligent in many ways.
Some may only be used to sense some activity in the environment. This information can then be used by other objects to respond suitable to those sensed circumstances. For example, a
sensor in a lounge chair could determine the direction a user is
looking in order to change the position of a television screen. In
this way, the viewer always has the best viewing angle on his
television screen: The systems adapts to the user. This type of interaction is called implicit interaction: the ‘state’ of the systems
changes while the user did not have the explicit intention to do
so.
Implicit interaction is just on example category of interaction in
AmI environments. Explicit interaction will also play a role in
order to let the user interact with the system. Nowadays, most
interaction takes place by pushing buttons, typing on a keyboard
or using a mouse. It is obvious that is is not very unobtrusive to
equip every object in our environment with a mouse, keyboard
or other sort of input device. To really create an unobtrusive environment, the difference between the old static objects and the
new smart objects should only be noticeable when someone uses
the new ‘smart features’ of those objects.
The requirement stated above roughly brings two others along
with it. Firstly, the technology that has to be embedded into our
environment has to be very small in order to make it disappear in
our environment. Secondly, because common manners of interaction are not available anymore, new ways have to be developed.
The first requirement, the minimisation of technology, contains
the minimisation of computational power to the size that it can
be embedded seamlessly in our environment. Furthermore, technology has to be developed that can detect user input and is also
embedded in the environment. ‘The environment’ could be anything in our surroundings; chairs, walls, tables or newly designed
objects which can be unnoticeable places in our surroundings.
Because interaction takes place on the outside of technology, in
general this means that the surface or skin of the ‘old common
objects’ will become the point of interface with users [3].
The concept of ‘common objects’ that become the point of interface with users will require a different approach regarding to
interaction design. Nowadays, computer screens are filled with
so called metaphors to give some intuitive meaning to what is
displayed on the screen [4]. For example, we can open and close
a window on our PC by clicking the mouse, which is a metaphor
to opening a book or writing pad for instance. But when technology is embedded into our environment, we have got many more
options. Grabbing, moving, trowing et cetera will become part of
the available options, something that is not recommended to do
with our mouse or computer screen.
Because the options with physical objects are greater than with
common technology, it is interesting to investigate the manners
in which we will utilise those possibilities. Therefore, this paper
will discuss interaction in the physical AmI environment along
with supporting technologies in order to create an overview of
the developments, possibilities and emerging interaction styles in
AmI environments.
2.
RESEARCH
In order to perform to the research, it will be carried out on the
basis of three main questions:
• How will technology be embedded into the surface of our
environment in order to make them ‘the point of interface’
with users?
• Which theories and interaction styles can be found that are
important regarding interaction with physical objects?
• Which practical applications or developments can be found
regarding interaction with our environment?
The first two question are related to each other: Without the
proper technology, interaction cannot take place. Vice verse, if
technology is available for interaction, suitable interaction styles
have to be developed to utilise the technology properly. Therefore, it can occur that technologies will be discussed that have
not got some practical use yet or interaction styles that are not yet
realisable. The goal of the last question is to show some practical
developments today to give one some idea about what AmI has
to offer in the future.
The paper will be structured according to the sequence of questions. Thus first technological developments will be discussed
that will have influence on the way we will interact with AmI
systems. Because technology will be embedded into common
objects, the section will mainly discuss smart materials and intelligent fibres. Smart materials and intelligent fibres have the
ability to for example transport information, change characteristics and for that reason are suitable to replace normal materials
unobtrusively.
The second section will discuss briefly some general notions about
human interaction to outline the possibilities of communication,
for example human sight, hearing and touch. With this common
knowledge in mind we will see how that knowledge can be applied to interaction with physical parts of AmI systems. Firstly,
this discussion will contain some theory about affordances in interaction. Furthermore, because AmI system parts can be everywhere in our environment, it is important that the system does
not draw too much attention to itself because it would confuse
the user. Therefore some theory about how a system can present
itself to the user in a intelligent manner will be discussed. Intelligent in this context means only drawing attention when necessary
and when not, disappearing to the background where it will not
bother the user.
technologies such a nanotechnology or organic LED technology.
In general, the purpose of smart materials is to “meet ambitious
requirement such as miniaturisation, cost or power efficiency”
[5]. For example, organic LED technology can be used to turn
virtually any sort of surface into a display and nanotechnology
can be used to produce very small memories or integrated circuits
in order to embed them into our environment. It is important to
understand that those technologies are needed to fulfil the AmI
vision, but are not within the scope of this paper. The fact that
smart materials are also materials that can change their characteristics, they are interesting to discuss because new AmI interaction
styles can emerge from them. The next section will discus how
they can become part of interaction with AmI systems, this subsection will only discus the possibilities that this kind of materials
have.
Smart materials in the context of this paper can be defined better
as “materials that change appearance or shape as a function of
an external stimulus” [6]. Changing appearance or shape could
simply be the way a user sees an object by change properties like
light absorption or scattering, but it could also refer to the way
a user perception of its environment by for instance changing a
materials roughness. The other important part of the definition,
the external stimulus, could be the input of a sensor that senses
the presence of people in a room or changes in temperature. But a
sensor is not necessarily needed, some materials can also change
characteristics more autonomous by changing directly as a result
of changes in the environment.
TECHNOLOGICAL DEVELOPMENTS
With regards to interaction, it is not important to know what those
kinds of materials are made of or to exactly know specific advantages or disadvantages of using them. Important is how they
can change and to which stimulus they can respond. Therefore,
only the possibilities that smart materials can offer as described
by Broer et al. [6] will shortly be discussed. One must keep in
mind that this will not be a complete overview of all possibilities,
but it will give some insight into them.
The introduction stated that ‘common objects’ will become the
point of interface in AmI systems. When looking at emerging
technology that is related to AmI, two technologies are found that
can enable us to make common objects the point of interface:
Smart materials and Intelligent fibres. In short, smart materials
are materials that can change their characteristics as response to
certain input, like changing temperature or light. Intelligent fibres
are fibres with mainly the possibility to embed electronic circuits
into them. Both technologies can be used to replace old materials
without changing their original purpose but with the possibility
to make them the point of interface in an AmI System.
To start with the stimulus materials could respond to. The first
possibility is that materials respond to input from an external sensor. As mentioned before, sensors will be discussed later on. Direct response, thus without the intervention of a sensor, is actually not that exceptional. We can see the effect when looking at
for instance a thermometer or frozen water, both indicating some
temperature condition. Next to temperature, materials react to
stimulus in the form of electric fields, humid conditions, light
etc. How to stimulate materials to achieve a controlled response
is work for physicists, so with regards to the subject of this paper
only the outcome is important.
Both smart materials and intelligent fibres will be discussed in
this section. But next to those two technologies, sensoring technology will become important in order to make common objects
the point of interface. As stated in the introduction, common
objects can be grabbed, moved, thrown et cetera. In order for
those objects to ‘know’ their are grabbed or moved they have to
be equipped with sensors. Due to this requirement, a technology
called smart dust will be discussed to show how this can be done
without making the technology obtrusive to users.
One of those outcomes is a change of form of solid materials. The
so-called Shape Memory Allows (SMA) is a class of responsive
metals that can “return to some memorised shape or size when
they are heated above a certain termperature” [6]. This change in
shape actually is a phase change. Instead of changing from for
instance solid to liquid, the metals changes within the solid state
[7]. Because the disadvantages like poor fatigue properties SMA
have, researcher looking for other materials, especially polymers,
that also have a mechanical response. Results have been found
using electricity, temperature and light to create a mechanical response, but most of the work is in the exploring phase and thus
not applicable [6]. However, it shows that changing mechanical
properties of solid objects is achievable and therefore something
to keep in mind with regards to AmI.
Finally, some concrete applications that are in development or
already developed will be discussed to give some insight in how
the theory is applied nowadays and what to expect in the future.
3.
3.1
Smart Materials
In the introduction of this section, smart materials were refereed
to as “materials that can change their characteristics”. Actually,
the term smart materials refers to a much broader spectrum of
Other developments regarding smart materials ‘only’ have a visual effect, for example materials that change color or change between being reflective, dark or transparent. Furthermore, developments are made regarding displays made out of materials with
interesting properties, which are flexible, can have any shape and
because of their size can be embedded into i.e. clothing. Those
displays are applicable in a way we nowadays use displays, but
more ubiquitous. The other smart materials are much more experimental and therefore have the opportunity to enable completely
new interaction styles.
3.2
Intelligent Fibres
In contrast to smart materials, intelligent or electronic fibres have
a much more clear application perspective. The goal of developing intelligent fibres is to embed technology into clothing and
is basically build around two parts: wires and connections [8].
The wires, which can be integrated with normal fibres in different ways, provide a manner to transport information the way this
normally goes. The connections mentioned can be for instance a
woven keyboard with one connecting for each button or a small
device that can be attached to clothing using press studs. Although there are some difficulties, for instance regarding electronic resistance or the need to make electronics washable, the
concept of electronic fibres is clear and not only applicable in
clothing but everywhere textiles are used.
3.3
Smart Dust
The two technology areas described above in some way make it
possible to make common objects the point of interface: electronics in clothing enable users to make more use of their clothing by
embedding a MP3 player or phone, smart materials enables systems to interact at more places by having more displays available
and interact in new ways by changing characteristics. However,
by only using these technologies, many interacting styles will not
be possible. Many ways of ‘doing things with common objects’
contain physical encounter with these objects. To enable interaction in this way, those actions need to be sensed by for example
movement, pressure or visual sensors. Next to that, those sensors
need to be small in order to prevent that they become obtrusive
towards users.
A technology that is build around sensing is ‘smart dust’. In short,
smart dust devices are about one cubic millimetre in size and
equipped with a sensor, some hardware to do something with the
sensed data, some communication capabilities and a very small
solar cell. In theory, because of the low power consumption of
the different components (between pico and nanojoule) a smart
dust device could sense, process and transmit the sensed data every second, based on power supply from the solar cell of one to
ten millijoule [9]. While not very useful on their own, in greater
numbers they can provide useful information and because of their
size, many could be deployed without bothering the user. Although the hardest part is to process the information gathered by
the smart dust devices, the technology can enable us to detect
much of our actions and use this information to create more manner of interacting with an AmI system.
4.
INTERACTION STYLES
The means humans have to communicate are limited. Limited
in this context means that humans have a finite set of senses
which can be used to either receive or send ‘messages’. With
AmI, new ways of addressing those senses will become available
and furthermore, it is likely that the computers that can interact
Figure 1: Ponzo illusion, which yellow line is longer?
with humans will be in our environment in greater numbers then
nowadays. Because the change in numbers and interaction capabilities, it is useful to look at how humans can communicate and
in which way interaction designer can make use of this knowledge. Therefore, first the human ‘input-output channels’ will be
discussed. Afterwards, some theorem and examples will be discussed to show how this knowledge can be applied to interaction
design.
To start with human capabilities to interact, we have five senses
to receive signal from the outside world: vision, hearing, touch,
smell and taste. However, only three of them are used to communicate, namely vision, hearing and touch. With regards to the
subject of this paper, hearing will not be discussed. One could argue that we can hear our environment, which is true, but the actual
communication, hearing and talking, is not a physically changeable. We cannot throw, grab or move sound and will therefore not
be discussed.
Vision is our primary source of information [4] and is nowadays
the most important sense for interaction designers. When considering human vision in interaction design, one must take into
account the difference between actual observation and the human
perception of the observation, due to processing in our brains.
Because of this processing, human sight has some capabilities
and limitations. Without going into details, those mainly are the
result of perceiving size and depth, brightness and color [4]. Result of our perceived vision is that some things attract attention
while others do not, some thing are clear to see while others are
not and the eye can be fooled, for example the Ponzo illusion
shown in figure 1. Both yellow lines are the same length, but
we perceive it differently. The second and last sense important
to physical interaction is touch. Normally, we use touch to feel
if something is hot or cold, which is providing a warning, or it
provides feedback when lifting something [4]. For example, lifting a glass without feeling it would be rather difficult, or at least
speed down the process of doing it. Touch is important in real
life, in contrast to touch in human-computer interaction. This
is logically, the interesting part of using computers happens on
the screen, not when you are typing or using the mouse. When,
as stated before, common objects become the point of interface
with users, this could change. The interesting question is, how
will we interact with an AmI system using touch?
The questions asked above will not be answered in this paper. We
can only show some guidelines or theorem that outline the contours of physical interaction in AmI systems. In order to do this,
firstly the term ‘affordance’ will be introduced and after that an
approach in utilizing the knowledge of affordances will be shown.
Together, this can give some insight in how physical interaction
will be in the future and, maybe more important, how it will not.
The approach stated above is just a vision. However, in normal
desktop computers new interaction styles could be introduced,
like windows, buttons and the mouse, because nothing comparable was available. In AmI, this computer will disappear into the
environment where it has to adapt to existing human conventions
to be unobtrusive and invisible. From this point of view, using
‘old’ affordances seems very plausible.
To start with affordances, this term was introduced by J.J. Gibson
to refer “to the actionable properties between the world and an
actor” [10]. Those affordances are natural, meaning that they are
just their: “they do not have to be visible, known, or desirable”
[10]. To show this natural character of affordances, one could
think of a playing child, using everything within its grasp to build
castles, cars etc. For older people, the objects the child uses has a
completely different affordances, but neither the affordances the
child or the older person know are better or worse.
5.
Don Norman ‘borrowed’ the term affordances in order to apply
it in interaction design. However, because affordances are natural, they cannot be changed or manipulated. Therefore the term
perceived affordances is introduced [10]. The subtle difference
between the two is best explained by an example. On a computer
system, clicking with the mouse can be done at every pixel on the
screen. Therefore, clicking on the screen is an affordance, just
like physically touching the screen, moving the mouse, looking
at it etc. However, many of those affordances are useless when
using a computer. A useful affordance is clicking on a red cross of
a window, which closes the window. When adding such a cross
to a window, an interaction designer did not add an affordance,
he or she only used the convention of a red cross to link an affordance to the desired reaction of closing a window. In other
words, the affordance is already there, the task of an interaction
designer is to find ways to let the user know the affordance is useful. When doing so, the goal is to design it in such a way that the
perceived affordance by a user is the same as the actual response
of the system.
In current interaction design, many ways of creating the right perceived affordance exist. Those interaction styles are culturally
dependant and rely on certain conventions. The red cross in the
right corner of a window is an example of a convention, but there
is no reason it could not be in the left corner. For physical interaction, those interaction styles do not exist because of the lacking
of applications, but work is done to change this. For example,
in [11] a couple of experimental systems were shown. These systems were mainly physical because they had a touch screen, so no
big breakthrough in physical interaction styles. However, their
approach seems a good starting point: “Our approach is based
on exploiting the affordances of real objects by augmenting their
physical properties with the potential of computer-based support”
[11]. What this approach says, is that we have to look at objects
that are in our surroundings for sometimes hundreds of years (e.g.
paper or pencils), learn how we use those objects and ‘copy’ those
‘interaction styles’ into our vision of AmI. For several reasons
this seems a good idea. First of all, using regular or new AmI
objects would make no difference, except for the additional features of the new objects. Everything learned when growing up is
applicable when using computers. Furthermore, Norman states
in [12] that conventions are cultural constraints that evolves over
time and are slow to be adopted. Introducing ‘new’ ways of interaction with our physical surroundings could be a slow and long
process, forcing users to adapt while in AmI computers should
adapt to humans.
PRACTICAL DEVELOPMENTS
Technological developments and theories about interaction is only
a basis to build upon. This section will discuss three developed
examples of physical interaction applications. Furthermore, those
applications will be linked to affordances and, if applicable, technological developments mentioned earlier.
At the Tangible Media Group, part of the media lab of the Massachusetts institute of technology, multiple tangible interfaces have
been created. For example, in Figure 2 the SandScape application can be found. The SandScape is a tangible interface for
“designing and understanding landscapes through a variety of
computational simulations using sand” [13]. In other words, the
SandScape is a sandbox in which users can shape the sand. The
computer captures the shape of the sand and performs different
calculations on the shape. For instance, it can calculate water
flows, shadows and drainage. It interacts the results of those calculations by projecting different colours on the sandscape in real
time. Shaping the sand is something very intuitive and choosing
the type of calculation the is also by simply laying down a block
on the edge of the sandbox.
Figure 2: SandScape application [13]
Another project, the Multiple Intimate Media Environments (MIME)
project is focusing “on the relationship between computer technology and people’s experience of their intimate media collections around the home” [14]. With intimate media the groups
means for example pictures, souvenirs and diaries. One of the
MIME concepts is GlowTags. GlowTags are tags that can be attached to objects and contain meta data. The GlowTags will pulse
or glow, hence their name, “in certain circumstances, when they
notice a connection, either a correlation of dates or an interaction
with a certain person or related to a certain location” [15]. The
typical example is a tagged birthday present which will glow a
year later, reminding the user of the person who gave the present.
Although not directly serving a certain task, the concept reveals
a very simple idea about subtle interaction in an environment.
Glowing or fibrating are probably the most obvious manner to
notify a user, but others could be used. It shows a concept how
artifacts in a user environment can draw attention in various ways
without obstructing a user to much.
The last project we will mention is smart dust. The concept of
smart dust is explained before, an application has not been given.
The acceleration sensing glove [16] (see Figure 3) is a hand glove
with ‘accelerometers‘ attached to the fingertips and palm of the
hand. It demonstrates that with the use of those accelerometers,
which are little smart dust devices attached to the glove, “hand
gestures can effectively be translated into computer interpreted
signals“ [16]. In this particular case, those signals are translated
into the different characters in the alphabet, including the space
characters and the delete action. The use of the glove for inputting characters is probably not ideal and smart dust devices
are better used in wireless sensor networks. However, the use
case shows another physical interaction style; movement. If adequately sensed, movement or gestures can well be used as input
as the Nintendo Wii has shown.
Finally, we have seen some examples or concepts of interfaces
which fit in the AmI vision. What we can conclude is that the
vision leads the technique; there are plenty of ideas, the technique
has to follow to keep up. For example, tangible interfaces such as
the SandScape needs a projector which can not be hidden in the
environment and sensing movement with a sensing glove is not
very comfortable.
7.
References
[1] E. Aarts and J. Encarnacao, “Into ambient intelligence,”
in True Visions-The Emergence of Ambient Intelligence,
E. Aarts and J. Encarnacao, Eds. Springer-Verlag, 2006,
ch. 1, pp. 1–16.
[2] E. Aarts, “Technological issues in ambient intelligence,” in
The new everyday, E. Aarts and M. Stefano, Eds. 010
publishers, 2003, pp. 12–17.
[3] M. Verbcken, “Towards a new sensoriality,” in The new everyday, E. Aarts and M. Stefano, Eds. 010 publishers,
2003, pp. 54–59.
[4] A. Dix, J. Finlay, G. Abowd, and R. Beale, HumanComputer interaction, 3rd ed. Pearson Education Limited,
2004.
[5] S. Grabowski and H. Nikol, “Smart materials,” in The new
everyday, E. Aarts and M. Stefano, Eds. 010 publishers,
2003, pp. 84–89.
[6] D. Broer, H. Houten, M. Ouwerkerk, J. Toonder, P. Sluis,
S. Klink, R. Hikmet, and R. Balkenende, “Smart materials,” in True Visions-The Emergence of Ambient Intelligence, E. Aarts and J. L. Encarnacao, Eds.
SpringerVerlag, 2006, ch. 4, pp. 53–82.
[7] S. R. Group. (2008, April 23) Shape memory allows.
[Online]. Available: http://www.cs.ualberta.ca/∼database/
MEMS/sma mems/sma.html
Figure 3: Acceleration Sensing Glove [16]
6.
CONCLUSIONS
We have seen some technical developments, interaction styles
and concrete examples related to interaction with our physical
environment. In general those examples can be divided into three
groups, namely user input, system output and the interface.
In the AmI vision, common objects will become the point of interface. These common objects can be clothes, CDs and books
with small tags attached et cetera. The most important characteristic of this point of interface is that it is unobtrusive and hidden
in the environment. System output in an AmI environment can
at one hand be characterized as subtle and non-disturbing. By
giving subtle visual signals or changing characteristic of objects
a system can draw attention from a user. At the other hand, in
AmI systems those manners of interaction can also be used in
normal application. Last and probably most interesting are the
new ways of user input. User input by touch, the tangible interfaces. An interesting vision on tangible interface is they have to
be designed based on ‘normal’ affordances. In other words, tangible interfaces in AmI should be used in the same way we use
objects which are in our surroundings for hundreds of years.
[8] H. Reichl, C. Kallmayer, and T. Linz, “Electronic textiles,”
in True Visions-The Emergence of Ambient Intelligence,
E. Aarts and J. L. Encarnacao, Eds. Springer-Verlag, 2006,
ch. 4, pp. 113–130.
[9] B. Warneke, M. Last, B. Liebowitz, and K. S. J. Pister,
“Smart dust: communicating with a cubic-millimeter computer,” Computer, vol. 34, no. 1, pp. 44–51, 2001.
[10] D. Norman. (2007) Affordances and design. [Online].
Available:
http://www.jnd.org/dn.mss/affordances and.
html
[11] N. Streitz, “Designing interaction for smart environments:
ambilent intelligence and the disappearing computer,”
vol. 1, 2006, pp. 3–8.
[12] D. Norman. (2008, April 23) Affordances, conventions
and design. [Online]. Available: http://www.jnd.org/dn.
mss/affordance conv.html
[13] Y. Wang, A. Biderman, B. Piper, C. Ratti, and
H. Ishii. (2008, April 23) Mit tangible media group
— projects - sandscape. [Online]. Available: http:
//tangible.media.mit.edu/projects/sandscape/
[14] Koniklijke Philips Electronics NV. (2008, April 23) Mime
project. [Online]. Available: http://www.mimeproject.org/
[15] ——. (2008, April 23) Mime : Concepts : Glowtags. [Online]. Available: http://mime.cs.nott.ac.uk/content/
concept3.html
[16] J. K. Perng, B. Fisher, S. Hollar, and K. S. J. Pister, “Accel-
eration sensing glove,” in Proc. ISWC ’99: Proceedings of
the 3rd IEEE International Symposium on Wearable Computers. Washington, DC, USA: IEEE Computer Society,
1999, p. 178.