05
may
2014
Augmented Reality, art and technology
How will we do it?
A look into the future
of 3D animation and AR
Wim van Eck
Towards Hybrid Disciplines
in a Postdigital World
Isjah Koppejan
The Great Pig in the Sky
Interview with Theo Botschuijver
AR[t]
Magazine about Augmented
Reality, art and technology
May 2014
Interview with Theo Botschuijver, Page 12

2
Documentary of Jeroen Visser: Theo Botschuyver: art, science, recreation (1986), photo by Roel Bazen
3
Colophon
Table of contents
welcome
to AR[t]
Virtual IllusionS
What Psychedelics,
the Counterculture
and Virtual Technologies
have in common
Dorien Zandbergen
Contributors
ISSN
The great pig in the sky
Interview with Theo Botschuijver
Hanna Schraffenberger
& Jouke Verlinden
06
08
12
2213-2481
AR Lab & Partners
Contact
Wim van Eck, Edwin van der Heide, Yolande Kolstee,
The Augmented Reality Lab (AR Lab)
Maarten H. Lamers, Stephan Lukosch, Robert Prevel,
Royal Academy of Art, The Hague
Hanna Schraffenberger, Jouke Verlinden
(Koninklijke Academie van Beeldende Kunsten)
Illuminating Shadows
Intuitive Interaction with
Projected Augmentations
Marcello Gómez Maureira
& Carolien Teunisse
20
Guest Contributors
Prinsessegracht 4
Rosen Bogdanov, Florian Heller, Isjah Koppejan,
2514 AN The Hague
Heide Lukosch, Marcello Gómez Maureira,
The Netherlands
Diego Montoya, Melo Montoya, Pawel Pokutycki,
+31 (0)70 3154795
Peter van der Putten, Dirk Schart, Carolien Teunisse,
www.arlab.nl
Tamiko Thiel, Alexander Verbraeck, Dorien Zandbergen
[email protected]
Graphic design
Editor-in-chief
Esmé Vahrmeijer
Hanna Schraffenberger
printing
Editorial board
Media Krachtcentrale, Rotterdam
Mariana Kniveton, Laetitia Kolstee,
Yolande Kolstee, Jouke Verlinden,
CoVer
Reba Wesdorp
Boooooom
Photography: David Joosten
Designer: Manouk Hasebos
Make-up & Hair: Joëlle Romita
Model: Wouter @ Republic Models
www.arlab.nl
4
How will we do it?
A look into the future and
the past of 3D animation
and augmented reality
Wim van Eck
Augmented Self
Some thoughts on augmenting
ourselves
Yolande Kolstee
The AR Curse
Maarten H. Lamers
Hello Plant!
Deepening Human Connections
to Plants by Sonic Augmentation
Rosen Bogdanov
& Peter van der Putten
30
36
40
42
Towards hybrid disciplines
in a postdigital world
Reflection on the future of art,
science and tech in arts education
Isjah Koppejan
48
Site Venice Site Biennale:
The Manifest.AR Augmented
Reality Intervention into the
2011 Venice Biennial
Tamiko Thiel
56
Hitting imaginary walls,
pulling virtual strings
What augmented reality can learn
from urban dance
Hanna Schraffenberger
66
There is more than meets the
eye in Augmented Reality
Game Environments
Stephan Lukosch, Heide Lukosch
& Alexander Verbraeck
72
Corona: Audio AR for historic sites
Florian Heller
80
Developing augmented reality
applications: A future perspective
Robert Prevel
86
Hyper-Mondrian
Dirk Schart, Diego Montoya
& Melo Montoya
88
The so-called
Augmented Reality
Pawel Pokutycki
Radioscape, Wormhole
Dordrecht and Beyond
On the Sensation of Presence
Edwin van der Heide
92
96
5
WelcomE...
to the fifth edition of AR[t],
the magazine about Augmented
Reality, art and technology!
Two and a half years ago the AR Lab, a collabo-
do it’ focusing on the future and past of 3D ani-
ration between The Royal Academy of Art, Delft
mation and Augmented Reality. He looks at how
University of Technology and Leiden University,
software has developed throughout the years and
started the AR[t] magazine series with the ambi-
uses this expansion to see where we are heading.
tion to compile and design an inspiring magazine
Hanna Schraffenberger continuous with her inter-
for the emerging AR community inside and out-
view articles. For this issue, she has interviewed,
side the Netherlands. Our experience and influ-
together with Jouke Verlinden, designer Theo
ence in the field grew over time, and our various
Botschuijver. Two former US presidents are also
contributors have written thought provoking and
featured in this issue: John F. Kennedy in Maarten
sometimes outspoken articles about AR. In all our
Lamers’ article ‘The AR Curse’ and Benjamin
issues we have shared our interest in Augmented
Franklin in my article ‘The Augmented Self’.
Reality, discussed its applications in fine arts and
The other articles cover, amongst others, critical
provided insight into the underlying technology.
views on Augmented Reality, artistic approaches,
We are proud of what we have established with
natural phenomena that are combined with the
the magazine in such a short space of time and the
latest technology and we learn what Urban Dance
international audience we have reached.
can teach us about Augmented Reality.
In this fifth issue of AR[t], we look at AR from vari-
Our thanks goes out to all researchers, artists
ous perspectives, however, the articles do have a
and lecturers at the AR Lab (whether based at
common denominator: the future of Augmented
the Royal Academy of Arts, The Hague or Delft
Reality. In some of the articles the future of AR is
University of Technology or Leiden University)
explicitly outlined, in others it is more implicit.
and all other authors from all around the world
We hope that this collection of articles guides
who have contributed in this and previous issues.
you in prospective projects. Stop predicting,
Moreover, a special thanks to you, the reader of
start producing: the future of augmented reality
AR[t] magazine.
is bright!
I hope we keep sharing our experiments and ideas
In random order, I would like to give you a quick
about AR and other intriguing, new techniques.
impression of some of the articles you will find in
We look forward to meeting you somewhere in
this issue.
the near future, in one reality or another!
Wim van Eck continues his series ‘How did we
Yolande Kolstee, Head of AR Lab
do it’, but it is now turned into a ‘How will we
6
7
Virtual Illusions
What Psychedelics, the Counterculture and Virtual Technologies
have in common
by Dorien Zandbergen
In Spike Jonze’s latest movie Her [1], the lonely
Theodore develops a relationship with his Operating System Samantha. After an initial period
of romance, the two alienate from each other.
Yet, different from human-to-human relations,
the cause for this alienation is that Samantha really is – in her very being - alien to the human
Theodore. While for Theodore, Samantha is the
One and Only, Samantha has thousands of simultaneous connections with virtual other entities.
And where Theodore relies on verbal communication to express himself to Samantha, Samantha
is at a loss of words when she wants to explain
to Theodore what happens to her while she is
Joaquin Phoenix in Her (2013), image courtesy of Paradiso Filmed Entertainment
rapidly evolving into something else. Samantha’s
capacity for connecting, learning, developing and
evolving is so big, that she can’t rely on the lim-
people’s living rooms, voice-controlled personal-
“counterculture” generally refers to the post-war
and exploitation, and rebuild a new, holistic sense
ited repertoire humans have available for com-
ized digital assistants that seem able to read peo-
international movements of students, poets, writ-
of awareness. Zen meditation, brought to Califor-
munication. In this she finds companionship with
ple’s minds, and the normalized and seamless use
ers, academics, bohemians and others who felt
nia by Indian guru’s and returning Indian travelers,
the uploaded spirit of a certain Alan Watts, who
of small cameras, earphones and microphones as
united in a desire to reinvent western culture –
as well as psychedelics, yoga, absurdist theatre
will be further discussed later on.
habitual extensions of the human senses.
away from corporate greed, war-related violence
and encounter groups were among this repertoire
and environmental destruction. Within this cultur-
of de- and re-conditioning techniques.
Jonze’s movie is set in a near future city modeled
Yet, the brief appearance of Alan Watts showed
al milieu, Watts represented the so-called mystic
on a combination of Los Angeles and Shanghai.
me something else. Namely, that our ideas about
strand: for him, cultural change had to come from
It is less well-known that electrical, electronic
The story could be interpreted as most people
the potential of digital technology – such as the
a change in personal perception. Watts regarded
and later digital technologies played similar de-
understand the science fiction genre in general:
idea that it can manifest higher forms of aware-
“society’s official version of reality” as “silly and
and reconditioning roles. In his The Electric Kool-
as giving people a taste of the future and particu-
ness - are inspired by cultural domains not com-
inadequate.” For him, another type of reality can
Aid Acid Test (1968), American author Tom Wolfe
larly of the role of advanced technology in this
monly associated with technology. Domains such
be experienced that reveals the “grandeur of the
[3] describes how this was the case for a group
future. In this understanding, a movie like Her
as spirituality and political orientation, for in-
cosmos.” (Watts in [2, p. 54]).
of hippies, The Merry Pranksters. The pranksters
has predictive value; it helps people anticipate
stance.
the strange realities of a world more and more
lived communally in a cottage in a forested area
Those within the counterculture who were drawn
south of Palo Alto, and made cross-country trips
characterized by the non-intrusive, intuitive, in-
Alan Watts (1915-1973) was a British-born, self-
to this type of explanation regarding the causes
in a 1939 International Harvester school bus that
terfaceless presence of forms of artificial intel-
educated theologian, philosopher and lecturer. In
of social alienation, warfare and environmental
they bought from a man with 11 children. They
ligence with emotional and rational intelligence
the 1960s and 70s he was a mystic teacher and a
destruction, embraced a variety of techniques.
had wired their house, the forest around it and the
that exceeds that of people. Jonze shows a world
spiritual leader for the so-called countercultural
These would help them “decondition” from social
bus with speakers, microphones and stroboscopes
of all-immersive games that become one with
movements predominantly in California. The term
narratives that preach competition, dominance
so as to create disorienting environments of sound
8
9
ics, conventionally known as a theory of digital
Theodore and his beloved operating
information systems, formed one of the (semi)
System, Spike Jonze’s movie could be
scientific backbones of this cultural environment
embraced as one such technique.
Dorien Zandbergen
where people tried to imagine the world and the
connection between its parts (nature, humans,
machines, etc.) in holistic terms.
References
It is in this cultural context that the idea of the
1. Jonze, S. (Writer, director,
personal computer, the World Wide Web, Virtual
producer). (2013) Her [Motion
Reality, and today the Internet of Things, Quan-
picture]. United States: Warner
tified Self and the Singularity have found very
Bros Pictures.
positive and hopeful reception. As Richard Barbrook and Andy Cameron (1995) put it [4], this
2. Anderson, W. T. (2004).
Californian Ideology has become the dominant
The upstart spring: Esalen
framework for interpreting digital technologies,
and the human potential move-
outside of California as well. Whether phrased
ment: the first twenty years.
In 1996, Dorien Zandbergen worked for the helpdesk
in explicitly spiritual terms or not, this ideology
iUniverse.
of the Dutch internet provider XS4ALL. Fascinated
by the engaged culture she saw emerge around this
perpetuates the idea that humanity’s merging
with ubiquitous intuitive technologies helps us
3. Wolfe, T. (1968). The Electric
Kool-Aid Acid Test. London: Black
of digital culture as an anthropologist. Her MA thesis
connect with a global humanity, helps us expand
Swan.
discussed gender dynamics and the political structure
of Open Source hackers in a squat in Amsterdam. For
our knowledge of the world and make us more
Biofeedback practitioner. From D.B. Payne and C.T.
4. Barbrook, R., Cameron, A. (1995).
her PhD thesis, she studied the countercultural con-
The Californian Ideology. Alamut:
text in which information technologies developed
Yet, where it was the intention of these 1960s and
Bastion of Peace and Informa-
and popularized in California since the 1960s. Assisted
70s hippies to turn against, what they regarded
tion, Retrieved May 8, 2014, from
by a Fulbright Scholarship, she conducted this research
creative and efficient.
Reitano, BioMeditation: The Scientific Way to Use the
Energy of the Mind (Brookline, Mass.: BFI, Inc., 1977), ii
new thing called the internet, she became a student
empathise more with different kinds of reality,
as, rigid and blinding thought systems – like con-
http://www.alamut.com/subj/
while spending a year in Silicon Valley. Currently, as
and light. The culmination of this cacophony was
ventional religion, cultural convictions and norms
ideologies/pessimism/califIdeo_I.
an independent researcher, she is studying the way in
the three-night Acid Test held in San Francisco in
– today, this digital optimism has become dog-
html
which European cities turn themselves into so-called
1966. Like many other such events at the time, the
matic in its own right. As such, it blinds people to
venue of the Acid Test housed projectors, oscillo-
the very real possibility that the ever-increasing
scopes, music and strobe light. The idea was that
presence of tracing and tracking sensor-based
in this ‘cacophonous’ multi-media environment
technologies creates an orwellian nightmare. And
there was no room for rational contemplation and
to the fact that exploitation, warfare and corpo-
Zandbergen, D. (2011). New Edge.
a critical cross-disciplinary platform on Smart Cities.
attachment to conventional thought systems.
rate greed simply continue to exist in our digital
Technology and Spirituality in the San
The aim of this platform is to understand better, in a
worlds, yet in ways that are perhaps even more
Francisco Bay Area. Leiden University.
hands-on, ethnographic way, what the conditions are
Ideas that came from this cultural background
difficult to discern. It also stops people from
Non-published dissertation, to be
in which information technologies can be empowering,
would later play a role in the Californian enthu-
carefully contemplating how sensor technologies
found at: http://dorienzandbergen.
and when they can be disempowering. In all this work,
siasm for Virtual Reality and Virtual Worlds. Also
may alienate humans from their own human sens-
nl/links/
Dorien likes to maintain a hands-on understanding of
the first non-military experimentations with sen-
es and direct day-to-day environments.
Smart Cities. Following processes of digitization in dif-
Further Reading
ferent spaces in the city, she wants to understand how
this works out in practice. This research will be part
of a documentary and will give way to the founding of
techniques and technologies varying from woodworking
tools to software.
sor-based technologies for motion-tracking or for
biofeedback purposes were informed in part by
It is my opinion that we need new types of de-
a similar motive – to decondition from engrained
conditioning techniques to challenge our collec-
[email protected]
patterns of human thought and to get in touch
tive blind faith in the digital. By beautifully show-
 @DorienZ with a higher form of reality. Similarly, cybernet-
ing the painful yet insurmountable gap between
www.dorienzandbergen.nl
10
11
The great pig in the sky
Interview with Theo Botschuijver
by Hanna Schraffenberger
& Jouke Verlinden
12
13
In my adolescent years, I stumbled upon a documentary on Dutch TV that had
a huge impact on my dreams of the future. It featured a funky designer-computer scientist who was enthusiastically creating a pre-Kinect depth camera.
He did this by hacking a Polaroid ultrasonic sensor and x-y frames, controlled
by an Apple II with a custom-built graphics card. The documentary made my
head spin: computers were cool, technology can “augment” the process of
creating art … Years later, while I was graduating on VR technology in the US, I
found out about this funky designer-computer scientist’s longtime collaboration
with VR artist Jeffrey Shaw. Uncanny! They certainly explored the future in the
past… If anyone should be consulted about the future of Augmented Reality, it
should be him: Theo Botschuijver.
To be honest, I hadn’t even heard of Theo Botschuijver before my colleague Jouke
suggested that we’d interview him. After checking Theo’s website [1], it turned out
that, although I didn’t know him, I knew many of the works that Theo had realized
together with media artist Jeffrey Shaw. More importantly, his oeuvre featured
many Augmented Reality pieces – some of them were realized long before I was
born. Of course, I was intrigued…. What was someone who had been working with
AR in the 1970s doing now and what were his plans for the future?
in Europe. There was a farm close by Eindhoven
Augmented Inflatables –
Inflatable Reality
where all kinds of people could work and experi-
Theo’s augmented inflatables are designed to be
ment. In 1967, we started working together, up
presented in combination with a virtual layer that
until 1983. We called ourselves the Event Struc-
is projected onto them. Theo figured out how he
ture Research Group.”
could combine a physical, inflatable object and a
had studied in Italy and wanted to do some things
virtual layer that is projected onto this object in
Walking on water and a
smoking octopus
his Talking Head series.
“I have several Talking Heads. I project footage of
The Event Structure Research Group is behind
a talking face onto an inflated head. This setup is
many of the early inflatables, such as the ‘Air-
connected to a microphone so that the projected
ground’ (1968), a large inflatable playground and
face moves its mouth when one speaks into the
the ‘Waterwalk’ (1969), an inflatable that allows
microphone. That way, the inflatable head ap-
a person to enter it and walk over water by walk-
pears to speak.” We can imagine that this can
ing in it. “Wait, I can show you!” Theo takes out
be quite entertaining at a party and liven up an
a book in which he keeps track of all his projects.
otherwise straightforward speech. However, we
As the pages turn, we get more and more famil-
are more fascinated by his more conceptual, ar-
iar with his work. There’s commercial work, such
tistic, and sometimes almost romantic, works. For
as amusement park-like experiences designed for
example, Cloud (1970, Event Structure Research
big car companies to promote their latest model.
Group, Theo Botschuijver, Jeffrey Shaw), which
There is political work; an inflatable world-bomb
was exhibited in front of the Stedelijk Museum in
that has circled over the heads of demonstra-
Amsterdam. Like the inflatable heads, the work
We call out a shy “Hello?” and after some seconds
learned about woodworking, metals; we had theo-
tors at the peace demonstration at the Museum
uses projection mapping, and combines virtual and
of uncertainty, we enter the studio. Theo’s work-
retical lectures: they were a fantastic five years.
Square in Amsterdam in 1981. There are sketches
physical elements in the real environment. It con-
place is spacious; it is filled with his old works, big
We could spend entire mornings talking about
for shoes that use piezoelectric materials to shine
sists of a physical, inflatable cloud onto which – at
sawing machines, lasers and materials in all shapes
forms and shapes and structures.” As much as
when you walk. There is an inflatable Octopus.
night – a daytime cloud is projected, accompanied
and sizes. The huge studio is also rather cold. We
Theo enjoyed studying at the university, it was the
Most works come with an accompanying story. “It’s
by mist effects and weather sounds.
find Theo and Hannie - Theo’s partner and busi-
practical aspect of it that he enjoyed the most.
always different how the public reacts. In the case
ness associate - in a small and comfortable little
“One year of the course was a practical year. Here
of the octopus, people were starting to attack it
side-room. Hannie is filling the stove with wood
it all came together. I worked at a furniture fac-
with knifes. But with time, you learn to anticipate
and Theo’s self-made inflatable windows do a good
tory and at the glass division of Philips and at the
things that could go wrong. We hooked the octo-
job at keeping the cold outside, illustrating Theo’s
precision moulding department where they built
pus up to a smoke machine. So that after the at-
expertise when comes to inflatables. We take off
the most accurate and fascinating little parts.”
tack he emitted smoke from all his wounds. Until
our jackets, get acquainted with Theo’s dog and
Despite the great time he had at Philips, he soon
someone run off with the smoke machine….” As
before we know it, we are entangled in a conver-
stopped with his path towards industrial design. “I
diverse as his work is, inflatables - most of which
sation about the beginning of Theo’s career.
ended up in England, where I got in touch with a
Theo made himself - in all sizes, forms and fig-
band called Pink Floyd. They were still unknown
ures form a recurring theme throughout all of his
back then, and I enjoyed their first performances.
work. If there would be a professorship in inflat-
They did not restrict themselves to the stage.
able production, his name would be the first to
“I would probably have pursued a degree in fine
They wanted to play with the whole space and in-
come up. His collection includes figurative ones
arts, hadn’t it been for my art teacher in high
volve the audience as well. That appealed to me.
(the Octupus), useful ones (an inflatable drum-
school who pointed me towards a new school in
Those guys were more approachable back then….”
set), architectural pieces (domes and buildings),
Eindhoven, which had just started and which of-
That was around 1967, the same time when he also
beautiful objects (clouds), some questionable ones
fered a degree in industrial design.“ The combina-
started his long-standing collaboration with Jef-
(huge breasts that can serve as party decoration)
tion of technical classes, difficult theory and ap-
frey Shaw. “Jeffrey and I met around ‘64/65. He
and finally, a collection of augmented inflatables.
Early career
14
 Kim Eshuis, Svp.sfeerbeheer
plied practice immediately appealed to Theo. “We
Talking Head 2 (2002), inflatable human head as a projection screen.
When we arrive, the door to Theo’s studio is open.
15
the city hall while a live band was playing. Then
A few pages of the photobook later, we enter AR
the pig, but they wanted it to look like one of
the plastic structure was inflated with smoke,
territory again. At least, the image Theo shows
those Walt Disney pigs or like the friendly pig
while an array of film and water dye projectors
us looks an awful lot like an AR headset. As it
that stands in front of your butcher. Pink Floyd,
constantly casted movies onto the structure. The
turns out, that’s exactly what the headset on the
however, wanted an aggressive and realistic pig.
audience was very enthusiastic and wanted to
picture is supposed to illustrate. It is a dummy
I made the model pigs for them and a company
participate; some of them even undressed and
prototype of Theo’s idea for a 3D desktop work-
produced three big scale versions.” Unfortunate-
jumped onto the inflatable.
space, which is situated in the real environment.
ly, the team responsible for the production didn’t
In 1986, he extracted the small CRTs from cam-
take the necessary precautions when it came to
One of our favorite works is “Viewpoint” (Event-
corders, and used half-silvered mirrors to project
the photo shoot with the pig. “We let the pig fly
structure Research Group: Theo Botschuijver,
the Apple II’s crude 3D graphics.
in the sky. But even before we started with the
Jeffrey Shaw), an installation from 1975 that was
real shoot, there was a sudden ‘pling!’ - the ring
shown at the 9th Biennale de Paris, in the Mu-
To illustrate that the headset is in contact with
that connected the pig to its anchor broke... and
Cloud (1970), Eventstructure Research Group: Theo
see d’Art Moderne in France [2]. The installation
the computer, Theo has connected the headset
there was no safety rip-panel. The pig drifted
Botschuijver, Jeffrey Shaw.  Pieter Boersma
showed images of several staged events that had
to the computer with a cable. “It turns your own
off! It wasn’t our fault, that was the job of the
taken place in the museum’s space earlier. The
living room into a memory palace. With the envi-
firm we hired.” Now he can laugh about it, but
As we explore Theo’s oeuvre further, it becomes
animated slides of these events were projected
sioned device, you can put your documents into
back then it definitely wasn’t funny. “Everyone
more and more apparent that those inflatables
onto a retro-reflective screen. Because of the
your real surroundings, and store them in real
thought it was a publicity stunt but it wasn’t.
are not his only works that have explored key
retro-reflective properties of the projection sur-
space. It will be much easier to find them back...
The pig was floating into the worst possible di-
concepts of Augmented Reality, such as combin-
face, the events were only visible through a view-
If I wear the glasses, and I look to the real book
rection. As soon as I saw the pig drifting into the
ing the physical and the virtual in real space,
ing console and the screen remained grey from
shelve, I can store files there…”
direction of the airport, I called Heathrow and
we mentioned earlier on. In fact, Theo has real-
all other perspectives. The projected images
ized a variety of AR works long before the term
were aligned perfectly with the real space. Con-
‘Augmented Reality’ was devised. An early per-
sequently, a look through the console revealed
Looking back, looking ahead
formance in 1967 that experimented with archi-
a seamless collage of projected images and real
Theo’s most well-known work is probably the
for the pig. At some point the pig flew so high
tectural structures, live action and projections
surroundings. In passing, unsuspecting visitors
huge inflatable flying pig that is featured on the
that it was not considered a hazard anymore.”
was “MovieMovie”. The Event Structure Research
became part of the viewed scenery, resulting in
cover of Pink Floyd’s album “Animals” from 1977.
Ironically, the pig landed - and got itself injured
Group, which at that time consisted of Theo
a combination of real and virtual events taking
“This wasn’t easy. We hired a company to make
- at a pigsty. Theo and the team involved stayed
Botschuijver, Jeffrey Shaw and Sean Wellesly
place at the same time. “We had staged twelve
Miller, prepared a pitch to bring “something spe-
events. For example, one event showed a visitor
cial” to the Knokke arts festival that year. And
who went to sleep on a museum bench. Another
their proposal was accepted! Theo made a huge
image sequence showed how we built the instal-
inflatable projection dome. Dressed in white lab
lation, the process of setting up the retro-reflec-
coats, Theo, Jeffrey and Sean set up the dome in
tive screen in the museum space.”
warned them. They couldn’t believe their ears.
“Could you spell that sir?’ – P. I. G., a flying PIG!”
– ‘Oh, I see”. They sent a helicopter with a sniper
Early see-though HMD model. Documentary of Jeroen Visser: Theo Botschuyver: art, science, recreation (1986),  Roel Bazen
MovieMovie (1967), Eventstructure Research Group: Theo Botschuijver, Jeffrey Shaw, Sean Wellesly Miller.
An extension of the Corpocinema idea produced for the Experimental Filmfestival in Knokke, Belgium.
 Pieter Boersma (left) and Suzy Embo (right)
16
17
Flying pig made for Pink Floyd (1976). Battersea powerstation, London, England.  Phil Taylor, in: Mason, N. (2005). Inside out: a personal history of Pink Floyd. Chronicle Books.
with the concept of a helium filled balloon with
a minimized weight.” We can see sketches of the
wing on Theo’s walls. We wonder, whether Theo is
still planning on building an actual airwing. “This
is a dream, yes, this is a dream. Unfortunately,
we would need a lot of money to realize it, or
we need to collaborate with a big company. The
airwing has to be filled with helium and that’s exup all night, first picking up the pig from the sty
pensive. In the end, I envision it about 200 meters
and then repairing its holes… and finally shot the
in size. But even to realize a small one, approxi-
images for the album cover the next day.
mately 18 by 20 meters would be fantastic. That
idea is very attractive to me. The airwing would
One more gig in the sky
So far, the flying pig has been one of Theo’s most
be great as a means for transportation. But also
as a place to live... I can envision whole villages
floating in the sky.”
well-known projects. However, when it comes to
the future he dreams about one more great gig in
the sky: the airwing - a combination of airship and
References:
airplane. “We used to have two prevalent ways
of flying, the airship and airplane. I always found
1. Botschuijver, T., & Van den Dop, H. (n.d.).
it weird, that those were so at odds with each
Spatial Effects, http://spatialeffects.nl/, ac-
other… And after the Hindenburg disaster in May
cessed on 6 May, 2014
1937, the airship era ended. But there are also catastrophes with airplanes and we still have those.
2. Shaw, J. (n.d.). Viewpoint. Retrieved from
Anyway, I have always been fascinated by the idea
http://www.jeffrey-shaw.net/html_main/
of a synthesis between the airplane and the air-
show_work.php?record_id=46, accessed
ship…. to combine the aerodynamics of a wing
on 6 May, 2014
Airwing – a synthesis between airplane and the airship (computer model by Wick van Rij)
18
19
Illuminating
Shadows
Intuitive Interaction with
Projected Augmentations
by Marcello Gómez Maureira
& Carolien Teunisse
20
21
Light is an important aspect for all types of visual augmented reality
(AR) installations. This is especially true for projection-based AR,
Bioinformatics research group at Leiden Univer-
With this setup, people can interact with the pro-
sity. With the help of Fons Verbeek who leads
jection content by using the shadows they cre-
or ‘spatial AR’, where video projectors are used to create a virtual
the research group and also supervised our own
ate when occluding the primary projection. One
research project, we were able to get access to
of the first features we implemented was using
internal images of the zebrafish, making it the
shadows to reveal the inside of the zebrafish in
star of our AR installation.
form of X-ray visualization. Creating a shadow on
overlay on a physical canvas. When audience members approach spatial AR installations and enter the path between the projector and
the zebrafish canvas would display the skeleton
the canvas, this usually results in undesired shadows that disturb
of the fish wherever shadows were formed. Sub-
the illusion. This is particularly problematic in the context of inter-
Design and Functionality
active installations. How can the audience physically interact with
The zebrafish model consists of a wood and wire
layers that could be displayed by moving closer
the display without causing disturbing shadows? One approach of
base mesh that has been wrapped in paper-
to or further away from the fish model. To do so,
maché and painted white, serving as projection
we used the depth information reported by the
canvas. One of the two video projectors – the
Kinect sensor to determine which visualization to
primary projector – faces the fish model from
display on the shadows. In its current form, our
the side and is used to project the outer ap-
interactive display can be used by several people
pearance of the zebrafish onto it. The second
at the same time, leaving the choice of which vi-
projector is then positioned at an elevated posi-
sualization is shown to the person closest to the
tion in such a way that it is unlikely to cross its
model. Since the depth of each person is tracked
projection path.
separately, it is also possible to give everyone
dealing with such negative interference is to facilitate the interaction through separate computer input devices. With Illuminating
Shadows we follow another approach: rather than seeing shadows
as an undesired artifact, we embrace interference and explore the
possibilities that it can offer.
sequent experiments added further visualization
their own shadow display. This is an intriguing
Illuminating Shadows is an ongoing research proj-
A working prototype – involving two video projec-
In its regular state – that is without user interac-
possibility that we intend to explore in the near
ect that uses shadows as an intuitive means to
tors, a Kinect motion sensor, and a large zebraf-
tion taking place – the second projector is simply
future.
facilitate interaction in a spatial AR scenario.
ish model - is the test bench for our experiments
inactive. When, however, a user enters the space
Our setup allows the audience to step in front of
in this project. We wanted to design a display
between the zebrafish model and the first projec-
When using AR technology systems such as AR
the projector and to interact with the display by
that could find its place in a museum exhibition
tor, a shadow area is formed on the fish canvas.
glasses or smartphone and tablet apps with AR
creating shadows on the canvas. Their shadows
next to traditional physical exhibits. Without ac-
This area is recognized by the Kinect sensor, which
elements, we are constantly aware of the equip-
are used to reveal additional layers of informa-
cess to a fitting model for our project we decided
passes the information on to a program that con-
ment that enables us to experience the augment-
tion. This is achieved with a secondary projector,
to build our own: the aforementioned zebrafish
trols the output of the second projector. Finally,
ed reality. We feel the weight of the phone we
which fills in the shadows with extra content. As
model. Now the zebrafish is probably not the
the second projector displays an image on the very
hold in our hands when looking through our screen
a result, the audience can use their shadows to,
first specimen that comes to mind when looking
same area that is covered by the shadow of the
at the world and we cannot look away from the de-
look inside an object and reveal additional infor-
for a prototype subject. We learned about ongo-
first projector, creating an ‘illuminated shadow’.
vice mounted on our heads or resting on our ears.
mation by moving their shadow over the surface.
ing zebrafish research done by the Imaging and
22
23
Usually when using projection-based AR, we would
are naturally curious about the possibilities and
like the equipment to disappear from sight to give
the questions that often start with “Wouldn’t it
the illusion that the projections are part of our
be great if ...”. In other words, we want to experi-
reality. In our project, however, we welcome the
ment and play with the technology to see what it
awareness of the projection technology, at least
can do. Essentially, this is the kind of curiosity we
partly. While the second projector remains an ele-
hope to inspire in the audience.
ment that is ideally hidden from the audience, the
Block diagram showing the interaction between hardware, software, and human user.
primary projector is deliberately on full display. Its
In order to do so, we invite them into the space
position marks the source of both light and shad-
between the projector and the physical canvas,
ow, and is therefore an important element for the
causing them to become part of the augmenta-
interactive experience.
tion and ultimately enabling them to experiment
with it. By positioning users in this way, shadows
Seeing Shadows in a New
Light
become an unavoidable component of the interaction. It is highly fitting then, that the creation
of shadows can already in itself encourage play-
In the first issue of AR[t] magazine, Jouke Ver-
ful interactions. This is especially the case when
linden discusses spatial projection-based displays
bringing several people together to create shad-
in his article “Pixels want to be freed!” [1], and
ows on a shared canvas. While the personal space
mentions occlusion and shadows as known prob-
around a user is often reserved for close friends,
lems in this field. This is certainly a problematic
the perceived closeness between each individual
aspect for many installations. Working with shad-
shadow can encourage a playful atmosphere that
ows as intentional elements does not necessar-
wouldn’t normally occur between strangers.
ily solve the issues. And yet, we cannot help but
feel that considering shadows a part of spatial AR
Another way in which we implemented the ele-
could be an interesting opportunity, both as form
ment of ‘play’ is through exploration. While users
of interaction, and as a visual aspect that inher-
are technically able to create shadows on the dis-
ently connects the virtual with the real world.
play that are large enough to influence the complete canvas, most of the time, the shadows only
Schematic illustration of the augmented zebrafish installation.
The Element of Play
take up a small area of it. This causes the shadows
to act like dynamic windows reaching into the ex-
The aspect of ‘play’ has been of great impor-
hibition model. Such windows focus the attention
tance in the concept of our installation. Creating
to the places they originated. A user might, for
space for playful interactions holds the promise
example, hold his or her hand in such a way that a
of captivating audiences in a fundamentally per-
shadow is created on the fin of the zebrafish mod-
sonal and emotional way. This makes the interac-
el. Consequently, the fin becomes the focus point
tion, as well as the elements that are part of it,
of the interaction. Seeing all the possible content
all the more memorable for the audience.
requires active exploration of the different areas
by changing the shadows on the canvas. Here we
When talking about interaction within the con-
should remember that the distance between user
text of augmented reality, there is a certain
and the zebrafish has an impact on which shadow
playfulness present in many of the most exciting
layer is going to be displayed, which further in-
works in AR and especially in spatial AR. That is
creases the amount of exploration space.
not to say that spatial AR cannot be used in more
Early 3D sketch showing the projection of zebrafish skin on the
left, and an X-ray projection on a user’s shadow on the right.
24
utilitarian environments. And yet, the use of
There are, generally speaking, certainly more ef-
spatial AR still seems be enjoying its honeymoon
ficient ways to let users access information. Explo-
phase of technological novelty. In this phase we
ration of content, after all, already indicates that a
25
certain amount of effort is required. Furthermore,
Second, a green animation was projected on the
it hints at the possibility that some aspects might
pedestal of the model when the installation was
be left undiscovered. And yet, we believe this to
not in use, showing a figure holding its arm up in
be an absolutely acceptable element of our instal-
front of a projector.
and student of the Media Technology
that there should not be a right or wrong way in
While we have yet to receive feedback on wheth-
early video and animation work, she
how the model is explored. The user experience
er the animation proved helpful, we do know that
looked for interesting dialogues be-
is what’s most important here. If users are capti-
the addition of zebra stripes has actually caused
tween different kinds of media and
vated and enjoy interacting with the model, there
some confusion. Here it should be said that our
in her first encounter with a projec-
is a high chance of them seeing everything there
initial implementation of the zebra crossing was
tor she created animations that aug-
is to see. More importantly, there is a high chance
done with paper. In retrospect it makes sense,
mented the environment. Nowadays,
that they will remember it.
but as it turns out, people do not like to walk on
she is particularly interested in cre-
paper. The result of this implementation was that
ating compelling experiences that re-
users actively avoided the area towards which
flect on interactions between people,
we wanted to guide them. Even after the setup
media and our physical reality.
Mind Your Step
Zebrafish (Danio rerio). Image by Okinawa Institute of Science and Technology (OIST)
Carolien Teunisse is a media artist
program at Leiden University. In her
lation. In our project, playful interaction means
26
Carolien Teunisse
Throughout our project we went through several
was improved by marking the area with white
iterations in which we modified our augmented
tape, we made the mishap of cross-hatching an
During her Digital Video Design
zebrafish display. Especially when creating a
area that should have indicated the existence
studies at the HKU University of the
prototype, many aspects, whether technical or
of an interactive area. While cross-hatching pat-
Arts in Utrecht, she met like-mind-
conceptual, were changed along the way. While
terns can absolutely be understood as defining an
ed artists who together now form
focusing mainly on the installation itself, it can be
area, they are also often used to mark areas that
DEFRAME collective. They explore
easy to lose sight of the surroundings, referring to
should not be used.
the possibilities of current media
technology by creating immersive
both the environment in which the installation is
presented and to the way in which an audience is
What this small example shows, is that there
experiences in the form of installa-
invited to interact.
were not many issues with the zebrafish display
tions and live visual performances
itself. It rather shows how simple modifications
often using augmented projection
In the case of our display we had to encourage
surrounding the installation can change how us-
technologies. Teunisse is also co-
people to step into the space between the pro-
ers perceive and interact with it.
founder of Fiber, an audio-visual
network platform based in Amster-
jector and the physical model. The difficulty
here is that many people actively avoid occluding
a projection path, as this is usually discouraged
Potential Implementations
in projection displays. A mindful audience might
So far, we’ve discussed the conceptual and physi-
therefore entirely miss the possibility of interact-
cal design as it was realized in our project. How-
ing with our display – a situation that we wanted
ever, in many ways the zebrafish installation is
to remedy as best as possible. We introduced two
very much a prototype, created for the purpose
ways to help users understand that they are wel-
of experimenting with the possibilities of shadow-
come to interrupt the primary projection. First,
based interaction with spatial AR setups. As such,
zebra stripes were added that indicate which
it represents only one of many possible forms of
visualization of the zebrafish corresponds to a
implementation.
dam organizing events and a festival
concentrating on the integration of
media art and electronic music.
certain distance. Here the symbolic nature of zebra stripes is meant to convey the message that
It is easy to imagine the use of other display con-
passing through the projection area is not only
tent. Starting from the visualization layers, many
possible but also required for interaction.
other types of information could have been used

in the zebrafish display. An interesting example
 DEFRAME.nl
would be to visualize the development stages
 FIBER-Space.nl
Pim Graus
27
of the zebrafish, starting from the larvae to its
In this respect, it is also interesting to think of
These examples show that there are many ways
adult appearance. While the physical shape of
different implementation in terms of scale. In
in which the future developments of shadow in-
the canvas would then no longer match the dis-
their paper ‘Shadow Reaching: A New Perspective
teraction in spatial AR might unfold. It is hard to
play content, it could instead serve as a consis-
on Interaction for Large Displays’, Shoemaker et
not get excited about the possibilities and we
tent reminder of the scale between the different
al. [2] describe the interactive possibilities of us-
certainly hope that our zebrafish installation was
development stages.
ing shadows to interact with very large screens.
not the last of its kind.
Since the reach of a shadow can easily be exIt is just as easy to think of completely different
tended by moving closer to the light source from
content. Augmenting a model of the human body
which the shadows originate, users can interact
could let users dive into the different anatomi-
with elements that they would not be able to
cal layers or even animate processes by moving
reach otherwise. On the other end of the spec-
closer to the model. Leaving the realm of organic
trum is the possibility to create small-scale ver-
matter, we can also imagine to augment architec-
sions of our installation for the use in classrooms
A video of Illuminating Shadows can be found at:
tural structures and implement shadow interac-
or in situations where multiple exhibition models
 www.vimeo.com/94137178
tion in such a way that it illustrates the changes
are presented in close proximity to each other.
made through history.
Links
 http://www.illuminating-shadows.com
Acknowledgements
Over the course of this project we have received
the display system as well as in constructing
Marcello Gómez
Maureira
the prototype itself. We would like to thank
Fons Verbeek and Hanna Schraffenberger, who
supervised the development of this project
from the start. We also want to thank Georgios
Marcello Gómez Maureira is an in-
Lampropoulos, Bram Snijders, Livia Teernstra,
teractive media enthusiast who
Meggy Pepelanova and Isabelle Kniestedt for the
is on an ongoing journey through
support they gave us whenever we were in need
different professions and fields
of a helping hand.
of research. Originally trained as
mechanical engineer, video game
artist, and video game designer,
References
he is currently pursuing the Master’s Media Technology at Leiden
1. Verlinden, J. (2012, April). Pixels Want to
University as part of his research
Be Freed! Introducing Augmented Reality
into designing interactive spaces
Enabling Hardware Technologies.
for audiences. Outside the aca-
AR[t], issue 1, 42-59.
demic environment, he develops
video games and works as free-
2. Shoemaker, G., Tang, A., & Booth,
lance game and graphic designer
K. S. (2007, October). Shadow reaching:
through his company ‘Dandy Uni-
a new perspective on interaction for large
corns’.
displays. In Proceedings of the 20th annual
ACM symposium on User interface software
 MARO.in
28
and technology (pp. 53-56). ACM.
Zebrafish (Danio rerio). Image by Okinawa Institute of Science and Technology (OIST)
a lot of help, both in the conceptual design of
29
Image 2. Skammekrogen, photo by Åsmund Sollihøgda with the courtesy of Makropol
Image 1. 3-Sweep, image courtsey of Tao Chen
How will we do it?
A look into the future and the past
of 3D animation and augmented reality
by Wim van Eck
In previous AR[t] magazines, I have used the ‘How
3D modelling and animation
did we do it?’ column to give an insight on how the
ings, better previews, more advanced physics,
fects with existing software. The visual effects
easier character animation tools etc. But these
companies hired to realise these special effects
are all small evolutions; the general workflow re-
have enormous budgets which enable them to
mained pretty much the same. The same goes for
develop add-ons for existing software, or even
most other commercial 3D animation packages.
develop new software to achieve a certain goal.
This actually makes sense though; most profes-
Sometimes, these programs are developed fur-
sional 3D animators don’t have time to constantly
ther and released as commercial software or a
get familiar with significant changes to their soft-
plugin afterwards, a great way to earn back the
ware. Having to learn big changes within your
invested money. The Lord of the Rings trilogy, for
software doesn’t go well with meeting everyday
example, gave birth to two new software pack-
deadlines. Many 3D professionals even work with
ages. Mudbox was developed to digitally sculpt
older versions of their software so they are sure
the fantasy characters, and crowd simulation
all bugs have been removed and their current
software Massive made it possible to realise the
workflow with other programs remains intact. We
spectacular battle scenes featuring thousands of
can be sure that the coming updates of commer-
characters on screen at the same time.
cial 3D animation packages will offer great new
options and possibilities, but we should not ex-
The same goes for the computer game industry.
pect the biggest innovations here.
This rapidly growing industry has been largely re-
AR Lab creates its projects and which software is
The majority of 3D artists use commercially avail-
used there. Since this issue’s main theme is ‘The
able software such as Maya, 3dsMax, Cinema 4d
Future’, I will look into my perfectly ray-traced
and Blender. In 2004, I was using Maxon Cinema
The movie industry, however, is an incubator for
power of personal computers. To comply to the
glass sphere to predict how we will develop our
4D R9 to create my 3D models, while currently
many fresh and innovative technologies. Big Hol-
high expectations of the players, the visuals of
projects in the near future… Or on second thought,
I’m using Maxon Cinema 4D R15. When we look
lywood blockbusters have immense budgets and
new games have increased incredibly. The latest
maybe it is a better idea to have a look at how
at both interfaces we see nothing has radically
their way to lure moviegoers is often to show
game-engines such as Unreal Engine 4, CryEngine
software has developed throughout the years to
changed, the biggest changes are cosmetic. Obvi-
special effects that have never been shown be-
and Unity 5 are able to produce almost photoreal-
see where we are heading, and at the same time,
ously, many improvements were made through-
fore on the silver screen. This forces them to in-
istic visuals in real-time. Since it is almost impos-
find out who drives innovation in this field.
out the years; more realistic and faster render-
novate, since they cannot achieve these new ef-
sible to create the accompanying highly detailed
30
sponsible for the enormous increase of graphical
31
I also foresee exciting times for real-time render-
recently impressed by Metaio’s new 3D objects
ing. While it is currently mostly used for games and
& environments tracking. Besides better tracking
simulations, movies could also benefit. Since com-
performance, we also started using software that
puters, tablets and phones are getting increasingly
was easier to use. ARToolkit still needed a good
powerful processors, animated movies could be
knowledge of C/C++, while nowadays we can use
rendered in real-time instead of having to play
easy to use game engines such as Unity to devel-
back a huge pre-rendered video file. Current real-
op our projects. Applications such as Aurasma/
time render engines already offer sufficient im-
Junaio/Layar do not even require any program-
age quality, and since the upcoming 4K resolution
ming skills at all.
(Ultra HD) format will result in such huge video
Image 3. Project Tango, image courtesy of Google
files it could make sense to render it out in real-
Augmented reality software developed quite
time. This was actually already quite common in
quickly since its introduction, and these develop-
the eighties, when programmers in the demoscene
ments will surely not slow down for the coming
programmed impressive real-time visuals only oc-
years. Google Glass gave a preview of things to
cupying small sizes such as 4096 bytes.
come, with hardware becoming truly wearable in-
environments manually, many game studios make
3D models by sketching simple 2D shapes, with
Rendering movies in real-time would also offer
stead of being bulky and uncomfortable. Instead
usage of procedural content generation, enabling
the software adding the third dimension auto-
the spectator the possibility to virtually walk
of mostly relying on cameras for tracking, we
them to algorithmically generate variations of
matically. A recent effort is 3-Sweep [2] (image
around within the movie, choosing a unique point
see a trend of combining more sensors for more
game assets, instead of creating them manual-
1), a research project by Tao Chen et al. 3-Sweep
of view to see the story unfold. Virtual reality
stable and markerless tracking. A good example is
ly. SpeedTree, for example, can generate whole
can derive a textured 3D model from a single
goggles such as the Oculus Rift or Sony’s Project
Google’s Project Tango (image 3), which maps your
forests of unique trees, and Allegorithmic’s Sub-
photograph, only needing some simple input
Morpheus are ideal to realize such possibilities.
surroundings (and creates a 3D scan) by combining
from the user. I recommend watch-
The Danish short film Skammekrogen by director
data from a motion tracking camera, depth sensors
ing the impressive demo on their
Johan Knattrup Jensen (image 2) is an exciting
and a regular camera. Being able to sense your en-
site since it is quite spectacular to
attempt at such an approach, where you wear an
vironment’s depth gives (among other things) the
see their software in action. These
Oculus Rift to see a family dinner drama unfold
advantage that virtual objects can be occluded by
applications are not often used by
from the successive point of views of the five din-
real objects, creating a much more advanced inte-
professional 3D artists directly, but
nerguests. It is unlikely that traditional cinema
gration of the virtual within the real.
the underlying technologies might
will be replaced soon, but these new technolo-
find their way into commercial 3D
gies invite us to have a fresh new look at how
One of the biggest problems with augmented re-
animation packages at some point.
stories can be experienced, possibly blurring the
ality is that there is still no ‘standard’ application
line between movies and computer games.
to see all the different augmented reality proj-
“While in the seventies games were
developed by small teams or even
individuals, a current blockbuster
game such as Grand Theft Auto 5 is
estimated to have a development
team of over a thousand artists”
are too many of them. The current need to install
am certain procedural content generation will play an even bigger role
stance Designer can generate variations of tex-
ects. There are in fact some standards, but there
So what will the near future bring? I
Augmented reality
and use many different viewers and applications
ruins the whole experience. I don’t have high
in the creation of pre-rendered and real-time 3D.
tures. Such techniques do not only speed up the
It simply will have to, since the consumer con-
Augmented reality software went through quite
hopes this will change in the near future, but a
creation process, but also significantly reduce
tinuously expects larger and more detailed digi-
a development since its introduction. I started
change is certainly needed to elevate augmented
memory requirements.
tal worlds and characters in movies and games.
working with augmented reality in 2006, a time
reality to a next level.
While in the seventies games were developed by
when popular augmented reality software such as
A different type of innovation comes from aca-
small teams or even individuals, a current block-
ARToolkit was using fiducial markers (black and
demic research at universities. These are places
buster game such as Grand Theft Auto 5 is es-
white patterns). While the tracking was already
which allow focus on new technologies and in-
timated to have a development team of over a
quite stable, it was still sensitive to low-light con-
teractions, without having to worry if the results
thousand artists. Instead of increasing the size of
ditions and partially occluded markers. The possi-
Professional 3D scanners have become more af-
are production ready and free of bugs. This can
these teams even further, it would be more fea-
bility to track high contrast images gave exciting
fordable throughout the years, and free software
results in innovative approaches to create 3D con-
sible to give the artists tools which allows them
new possibilities and also added better low-light
such as Autodesk 123D Catch already gives excel-
tent. Teddy [1] for example allows users to create
to automatize even more parts of the process.
performance and more forgiving occlusion. I was
lent results. It is much easier to make a scan of
32
3D scanning
33
Image 4. Get Even, The Farm 51
someone’s face instead of modelling it by hand.
in movies such as The Curious Case of Benjamin
crease and open up exciting new possibilities.
times for developers and consumers. Let’s stop
But did 3D scanning already start to replace 3D
Button, where a digital representation of actor
Imagine 3D scanning a beautiful location with
predicting and start developing!
modelling? While 3D scanning offers exciting pos-
Brad Pitt was created at different ages. Another
your phone while you are on vacation, so your
sibilities it also has some big drawbacks. Firstly,
example are the photorealistic environments of
friends at home can experience it virtually wear-
the object already has to exist (you cannot scan
the computer game Get Even (image 4, expected
ing a device such as the Oculus Rift. This would
something fictional), it should be accessible to
in 2015), which were created by 3D scanning real
surely be an great alternative for postcards or
be scanned and ideally shouldn’t be transparent,
world locations. This innovative approach makes
digital photographs… Or why not 3D scan your ho-
(2007, August). Teddy: a sketching interface
reflective or dark, as this gives scanning errors
it possible to seamlessly switch between live ac-
tel room and place it online as part of a review so
for 3D freeform design. In Acm siggraph 2007
with most scanning methods. And once you have
tion recordings and the virtual game.
people can virtually preview it at home? Sony al-
courses (p. 21). ACM.
1. Igarashi, T., Matsuoka, S., & Tanaka, H.
ready proposed such functionality during the GDC
a 3D scan, it still needs quite some manual work
to clean the model so it has a good topology (dis-
The previously mentioned Project Tango could
2014 (Game Developers Conference) unveiling of
tribution and flow of polygons) and a reasonable
play an important factor by possibly putting a
its Project Morpheus virtual reality headset.
amount of polygons. This doesn’t mean that 3D
3D scanner in everybody’s hands, similar to what
scanning is not widely used with often spectacu-
happened when mobile phones became equipped
This article has undoubtedly only shown a glimpse
lar results. 3D scanning played an important role
with cameras. The quality of the scans will in-
of things to come, and these are truly interesting
34
References
2. Chen, T., Zhu, Z., Shamir, A., Hu, S. M., &
Cohen-Or, D. (2013). 3-Sweep: extracting
editable objects from a single photo. ACM
Transactions on Graphics (TOG), 32(6), 195.
35
Augmented Self
Some thoughts on augmenting ourselves
by Yolande Kolstee
NOW
Integrating certain aspects of the virtual world
network of sensor information and data trans-
into our day-to-day activities is taking place at
port working at top speed.
a slow but unstoppable pace. People are getting
used to on the spot information that is brought to
To hear an exclamation like the one above is
them digitally. Location based information, like
quite common, but at a closer look, it is also a
density of traffic, weather circumstances or is-
bit weird: a person is looking for dynamic infor-
sues in public transport, is a logical addition. We
mation about the location while the person is
are getting used to the idea of meta-data about
actually at that place himself. Static analogue
our environment and we expect this to be avail-
information carriers, such as books, can pro-
able all day, every day. We might consider it a
vide us with information regarding our location
kind of natural right, or even feel entitled to this
and make us more aware of the special char-
extra information. This kind of awareness of the
acteristics it has; whether in historical, ethical,
existence of meta-data can be seen as a special
physical or cultural sense. The availability of
type of augmented reality. We usually consider
analogue information about a certain location is
augmented reality the addition of real time vir-
also available prior to the visit and will be so af-
tual information. However, certain other types
terwards. Location based information produced
of data addition can also be seen as augmented
by sensors, may be locally available or trans-
reality.
ported to the spot via our global positioning
system. Location based information also offers
LOCATION BASED DATA
time dependent, dynamic information – something analogue information lacks.
A location without dynamic digital information
available on the spot is nearly unimaginable.
People checking their smartphone for actual
“Hey, I’m here at the Vanderbilt Square and I
information, are not interested in the situation
don’t have any info on the actual weather here!
from 5 or 10 minutes ago. They could access this
What’s going on?”. Having immediate access
information since it is stored in log files, but those
to data about our physical environment seems
files are most often ignored. An app that reports
natural, and not everybody realizes that this is,
traffic jams that have already been solved, is ir-
in fact, the result of an extremely complicated
ritating and will soon be on the to-replace list.
36
37
LOGGING BODY RELATED
DATA
Benjamin Franklin listed his
thirteen virtues as:
ThirteenVirtues.com brings Franklin's simple an-
In theory, the picture in picture effect continues
alogue system to the information age. You can
deeper into the picture ad infinitum. This self-
track your progress against Franklin's virtues with
monitoring trend, and the fact that we have more
onist of Helen Fielding’s Bridget Jones’ Diary [1],
1.
Temperance (Tem.)
your favourite web browser. Or maybe you don't
images of ourselves than ever before, makes us
you should be familiar with lists like this: Mon-
agree with all of Franklin's original 13 virtues - no
aware of ourselves in an unprecedented way. The
day: 6 units of alcohol, 12 cigarettes. Weight:
2. Silence (Sil.)
problem. Just add, remove, or change them so
psychological, social and health effects of this
75 kilos. The penetration of digital systems that
Speak not but what may benefit others or that you only track what you are interested in.
new type of self-awareness are open for research
measure our own data is increasing every day.
yourself; avoid trifling conversation.
Apps on smartphones give us insight in data con-
3.
Order (Ord.)
If you are familiar with Bridget Jones, the protagEat not to dullness; drink not to elevation.
cerning our body; for example, apps that mea-
Let all your things have their places; let each
sures our heartbeats per minute, our breath fre-
part of your business have its time.
in the coming years.
AUGMENTING OURSELVES THE DROSTE EFFECT
References
1. Fielding, H. (1996). Bridget Jones¹ Diary,
quency, our burned calories based on our activity
4.
Resolution (Res.)
It is not unlikely that in 25 years all kinds of gad-
(measured in footsteps for example), our sleep
Resolve to perform what you ought; perform get-like monitoring equipment, like smartphones
and wake patterns. We can input our weight and
without fail what you resolve.
or smart-watches, are replaced with microchip
Pengiun books.
2. Franklin, B. The 13 Virtues, http://thirteen-
calculate our BMIs (body mass index), which can
5.
Frugality (Fru.)
implants and possibly augmented reality lenses.
be split up in fat, non-fat, bones and fluids. We
Make no expense but to do good to others or As a result, that which has to be monitored and
can even record the sounds we produce by voice
yourself; i.e., waste nothing.
that which is reading the monitored data is one
or other body parts while we sleep. By monitoring
6.
Industry (Ind.)
entity: our self. This creates a strange version of
our body, we are in a way augmenting our sense
Lose no time; be always employ'd in some
what the Dutch call the Droste effect [4]: an end-
org/wiki/Droste_effect, accessed 8 May
of ourselves. The quantified-self movement (see
thing useful; cut off all unnecessary actions.
lessness iteration of watching a picture in a pic-
2014.
http://quantifiedself.com/) expands on this by
7. Sincerity (Sinc.)
ture. We are monitoring ourselves while we are
also logging external data like the composition of
Use no hurtful deceit; think innocently and monitoring ourselves…
the air we breathe.
justly, and, if you speak, speak accordingly.
virtues.com, accessed 8 May 2014.
3. Franklin, B. (2004). The Autobiography of
Benjamin Franklin. Touchstone. In [2].
4. Droste effect wikipedia, http://en.wikipedia.
8. Justice (Jus.)
Mental and/or emotional performance in combina-
Wrong none by doing injuries,
tion with information about temperature or brain
or omitting the benefits that are your duty.
activity via wearable equipment can be added to
9.
Moderation (Mod.)
the image based on our bodies’ data. Although
Avoid extremes; forbear resenting injuries so people have been tracking their activities and
much as you think they deserve.
Sun.
Sil.
•
Ord.
•
feelings in diaries for centuries, life logging is dif-
10.Cleanliness (Clea.)
ferent. It gives an interesting twist to augmented
Tolerate no uncleanliness in body, cloaths, Fru.
reality. Let us consider a very nice example by
or habitation.
Ind.
Benjamin Franklin of what we would nowadays
11.Tranquillity (Tran.)
Sinc.
call mental performance logging as on the website
Be not disturbed at trifles, or at accidents Jus.
http://thirteenvirtues.com/ [2]. In 1726, at the
common or unavoidable.
age of 20, Benjamin Franklin created a system to
12. Chastity (Chas.)
Mod.
develop his character. In his autobiography, Frank-
Rarely use venery but for health or
lin states: “I propos’d to myself, for the sake of
offspring, never to dullness, weakness, or clearness, to use rather more names, with fewer
the injury of your own or another's
ideas annex’d to each, than a few names with
peace or reputation.
more ideas; and I included under thirteen names
13.Humility (Hum.)
of virtues all that at that time occurr’d to me as
necessary or desirable, and annexed to each a
short precept, which fully express’d the extent I
Tue
Wed
Thu
Fri
Sat
Tem.
Res.
Imitate Jesus and Socrates.
Mon
Clea.
•
•
•
•
•
•
•
•
•
•
•
•
•
••
Tran.
Chas.
Hum.
••
•
•
Benjamin Franklin tracked his progress on self-made charts. The days were listed on the top row
and the virtue were listed on the left column. If he violated a virtue on a given day, he placed a
dot in the corresponding cell.
gave to its meaning.” [3]
38
39
The AR Curse
by Maarten H. Lamers
I am not an “AR optimist”. In my opinion, augmented reality offers interesting possibilities, but few exciting
realities. The ultimate killer app still
needs to augment my reading glasses, fingertips or hearing aid. Perhaps
Google Glass will inspire developers
to come up with exciting new AR concepts, although the current outlook of
this happening is bleak, from what has
been announced. Could it be that AR
is suffering from the VR-curse?
In 1997, I visited CeBIT, Europe’s largest computer
expo in Hanover, Germany with a friend. It was
the age of virtual reality, and there we bought the
Forte VFX1, a then state-of-the-art virtual reality
system. It in no way matched our expectations.
In the nineties, virtual reality was hyped through
Hollywood movies and popular science. As a result, our expectations grew far beyond what it
could really offer us. It promised us virtual worlds
that were interesting and fun to move through. In
reality it gave us Commodore-64 quality graphics,
sloppy positioning, jurassic interaction latencies
VR pioneer Nicole Strenger,  CC BY-SA 3.0
John F. Kennedy
and quite severe headaches. I think that this illustrates the VR-curse. Technology and interaction
design have advanced greatly since 1997, but could
want to interact with a mix of virtual and real con-
that the VR curse is finally ending. Equipment will
apps that match its specifications; instead, design
it be that we are also expecting too much of AR?
tent, as promised during my youth in KIJK maga-
become more readily available, which must also
new ideas and ask the geeks to make equipment
zine1, where an AR gun could shoot the referee in
be good for AR. But I don’t want to wait anymore.
to match it. To paraphrase from JFK’s inaugural
We want instant virtual content that seamlessly
your televised football match. But none of that has
I want good AR, and I want it now! Fortunately for
speech,
overlays onto the world around us, whether in the
even appeared on the AR horizon.
me, I will either be happy when that happens, or
car or when watching football. But so far, the virtu-
proven correct in my pessimism (“I told you so in
”Ask not what AR can do for you; ask
what you can do for AR!”
al ad-signs overlaid onto Eredivisie football games
I may be completely wrong and just impatient.
2014!”). It’s an easy win-win for the pessimistic
work only for one fixed camera position, not for
The recent Oculus-Facebook deal may indicate
me. But in fact, I would like AR to succeed.
want maximum content availability with minimal
1
And for that to happen, we need to engage non-
logical ideas may develop, but what the heck...
distraction. But for now, Facebook profiles are the
ticle. It was published in an issue of KIJK in early
technicians in developing ideas for augmented re-
From what I’ve heard this is the last edition of
pinnacle of detailed content on Google Glass. We
1980’s. Let me know if you find it!
ality. Don’t look at available equipment and make
AR[t] Magazine, and my last chance to rant.
all other cameras recording the same match. We
40
Naturally, this is a utopian view of how technoUnfortunately, I cannot find the original ar-
41
Hello Plant!
Various artists have created installations cen-
justed to foster interaction and better measure
tered around the concept of plant sensing and
a subject's levels of empathy and affections to-
interaction. An early example is the ‘Interactive
wards plants (see [14] for full details).
Plants Growing’ installation, whereby interaction
between people and physical plants impacts the
appearance and growth of digital plants in a virtual environment [7]. An installation that takes
the human out of the loop is ‘Sensobotanics’ by
Thomas Hawranke. In this piece, a plant operates
a first person shooter game, emphasizing the dif-
by Rosen Bogdanov and Peter van der Putten
ference in terms of the time dimension between
both realities [8]. An example that is more of a
mix between the artistic and the practical is Botanicus Interacticus by Disney Research, in which
plants are used as a control device [9].
Nature and technology are often portrayed as en-
But does that mean that studying the bonds be-
A range of academic research projects also aug-
emies. But can technology help us get closer to
tween humans and plants is an off limits, esoteric
ment plants with technology to create interest-
nature? As a first step towards a ‘sonic garden’,
research topic? We don’t think so. Almost every-
ing hybrids. Professor Stephano Mancuso from
we created a device that augments the experi-
one has houseplants at home. This is so pervasive
the University of Florence is an example of an
ence of interacting with an ordinary houseplant
that one may forget to ask why. We no longer
accepted academic plant scientist making a
with audio feedback. It is not meant as some ar-
need houseplants to feed us or keep us dry, but
case for seeing plants as more intelligent crea-
tistic statement, but its purpose is to run a sci-
keep plants to create a pleasant living environ-
tures, and also collaborates with others to com-
entific experiment to determine whether audio
ment, or even to keep us company. If we ignore
bine plants and technology. For example the EU
feedback can deepen our bond with plants.
these bonds, it actually becomes quite difficult
Pleased project [10] focuses on using plants as a
to explain why houseplants are so common. This
network of sophisticated sensing devices [11, 12].
Over the course of evolution we evolved a need
human experience is a real world phenomenon
He also works with Barbara Mazzolai from the
to bond with plants, as plants give us food, shel-
that can be studied.
Italian Institute of Technology in Genua on the
design and prototyping of hybrid, robotic plants
ter and other resources important to our survival.
Augmented plant, equipped with electrodes.
This urge to affiliate with other forms of life was
Plants aren’t just basic, simple organisms either.
called plantoids, for example for soil analysis in
defined as the Biophilia Hypothesis by E.O. Wil-
One doesn’t need to ascribe humanlike high level
future space missions [3, 13].
son [1]. This partially explains why many cultures
intellectual capabilities to plants to recognize
ture soundscapes induce fascination with nature
ascribed magical and human capabilities to vari-
that these are actually complex, learning, and
and feelings of relaxation. Carefully filtered noise
ous aspects of nature including the plant world,
networked interactive systems, with sophisti-
and introduced rituals suggesting communication
cated sensing, signaling and action behaviors [3].
with plants. And this was not restricted to our an-
Take for example the way many plants use their
cient forefathers. ‘The Secret Life of Plants’, the
roots to detect, at any point, numerous chemical
1973 best seller, claims for example that plants
and physical parameters [4]. Plants are able to
So let us now discuss our own project in more de-
droplet is generated. Two different envelopes are
are high level sentient and intelligent beings and
distinguish between different positive and nega-
tail. The device used in this study provides sonic
assigned depending on data coming from the sen-
describes a range of prior experiments that lead
tive experiences [5], while also registering some
feedback from actual interaction (e.g. touch)
sors – one leads to a less saturated droplet gener-
to far reaching claims about plant abilities to per-
sort of memory [6]. Studying these mechanisms,
with a regular houseplant, and the goal of the ex-
ation, while the other leads to a more saturated
ceive and understand human behavior [2]. Not
without making wild claims about humanlike in-
periment was to measure whether this strength-
one, e.g. resembling a small waterfall.1
surprisingly most of the experiments couldn’t be
telligence, are an established practice in scien-
ens the bond between the human and the plant.
reproduced and it was generally seen as a prime
tific plant research.
We took in mind the background research and
1
pilot tests, whereby tasks were created and ad-
plantinteraction.mp3 as a sample example
example of pseudoscience [3].
42
Human Plant Bonding Through
Sonic Augmentation
The sound design is based on the idea that na-
that reminds of rainfall drops with randomized
intensity and pitch were designed in PureData.
Low-frequency modulated envelopes are ramped
every 5 seconds to set the rate at which each
Listen to http://www.liacs.nl/~putten/library/
43
same level of feedback. Plant processes typically
change slowly, are hard to measure reliably and
there is a lot debate on what response mechanisms truly exist. We rather focused on measuring the human experience or perception, which is
a real world phenomenon.
Putting It To The Test
Our prototype was tested for three days in the
Hortus Botanicus in Leiden (see http://www.
botanicus-interacticus.com/),
the
oldest
bo-
tanic garden in the Netherlands, with an overall
sample group of 30 subjects and a single plant –
Ornithogalum Thyrsoides, or Star of Bethlehem.
First the participants filled in a survey to measure
an individual's level of connectedness to nature,
based on a scale used in psychological research
[15]. Then the participants had to carry out a
Ornithogalum, Star-of-Bethlehem
number of interaction tasks such as touching the
leaves, inspecting the plant carefully, cut a leaf
At the sensor we measure electrical conductiv-
and spray water, to enable the participants to
ity in the plant using a Wheatstone bridge de-
‘bond’ with the plant. For half of the participants
vice and two copper taped electrodes. It detects
the interaction during this phase was augmented
two main events; touching of leaves and spraying
with the soundscape sounds through the device,
water on leaves. When touching a leaf, the less
the other half acted as a control group. They
saturated envelopes mentioned above are acti-
then had to carry out a number of tasks aimed
personal and perspective-taking relations in the
of the experiment. However, this is not neces-
vated. However, big changes in capacitance can
at measuring whether one could claim there was
individuals that went through sound-augmented
sarily a negative point. In the future, a larger
increase the amount of droplets: up to 10 drop-
an increased bond, for example setting a price on
interaction with the plant, though some of these
design focusing more on tightly connecting the
let synthesizers each generating droplets every 5
the plant, attaching their name to the plant with
differences may have been due to random bias
sensory experience with the particular context
seconds. If the data begins to peak – in this case,
a choice between push pin or blue tag, add soil
between test (those who received audio feed-
of human-plant interaction can give even more
if water gets on the electrodes, then all droplets
from which the participants could remove some
back) and control (no audio feedback), which we
interesting results. One can think of placing the
are generated with more saturated envelopes, in
cigarette ends, and to push the boundary, read a
measured through the survey at the start. In the
experimental design more inside the actual gar-
order to get a different feedback without losing
poem to for the plant, either silent or aloud. Fi-
self report results the test group scored higher
den, instead of on a table in the entrance. More
the effect of random generation. As opposed to
nally participants themselves would report their
than the control.
plants means more flexibility in the design of ob-
a straight sonification, this randomness creates
level of connectedness to plants, using a visual
a sort of perceptual detachment from the direct
scale.
visual interaction between the human and the
Test setup at Hortus Botanicus, Leiden
jective measures, and more flexibility in terms of
Towards a Sonic Garden
measuring and sensing interaction. Such design
should also allow for more implicit measurements
plant, without losing the feeling of actual feed-
The results show that people with sound feed-
More experiments would be needed to confirm
of bonding by tracking interaction, which should
back from touching and spraying water on plant.
back spent more time on the interaction tasks
some of these results, but in itself the experi-
lead to more reliable results. Other ideas include
than the control group, though one can argue
ment serves as an early example of how such
doing controlled experiments around interaction
A key principle is that we don’t even assume to
whether that was because of the sound itself or
research can be performed and extended. The
with a fake plant – i.e. looking for differences in
measure any real or smart plant responses. In-
the actual interaction. The results on the mea-
context itself, as seen from this study, is impor-
researching connectedness between actual life-
animate objects could actually have provided the
surement tasks demonstrate an increase in inter-
tant – a botanical garden does shape the nature
forms and artificial objects.
44
45
References
The more long term idea of a ‘sonic garden’
fits well with an augmented end vision of
extending man by interfacing it with both
1. Wilson, Edward o. (1984). Biophilia, Cam-
nature and technology. Marshall McLuhan
bridge: Harvard University Press.
2. Tomkins, P. and Bird, C. (1973). The Secret
meant with the quote is that by placing
142–147.
life of Plants, Harper and Row.
physical objects in the realms of our con-
7. Sommerer C. and Mignonneau L. (1993). "In-
sciousness and our nervous systems, we
3. Pollan, M. (2013). The Intelligent Plant. The
translate everything (artificial or natural)
New Yorker, pp 92-105, December 23, 2013.
to a potential information system. The
tools of such augmented physical realities
Markin, V. S. (2009). Electrical memory in
Venus flytrap, Bioelectrochemistry 75 (2):
stated: “In the electric age, we wear all
mankind as our skin” [16]. What McLuhan
6. Volkov, A. G., Carrell, H., Baldwin, A. and
teractive Plant Growing," In Visual Proceedings of the Siggraph ’93 Conference, ACM
Siggraph, 1993, pp. 164-165.
4. Brenner, E. D., Stahlberg, R., Mancuso, S.,
are, of course, often designed as direct
Viv- anco, J., Baluska, F., and Van, V. E.
extensions to our own sensory inputs/out-
(January 01, 2006). Plant neurobiology: an
puts - sensor interfaces and other electrical
integrated view of plant signaling, Trends in
Exkursionen ins Undingliche. Springer Verlag,
Rosen is a MSc Media Technology gradu-
media (screens, projectors, sound and radio
Plant Science, 11, 8, 413-9.
pp 88-93.
ate from Leiden University, Netherlands.
8. Hawranke, T. (2010). Sensobotanics. In
Trogemann, G. (editor), Code und Material
He is interested in building interactive
waves, etc.). In the context of our project
and augmented reality his quote is perhaps
Rosen Bogdanov
5. Goh, C. H., Nam, H. G. and Park, Y. S.
9. Disney Research Botanicus Interacticus,
systems that augment biological processes
better re-phrased to “...we wear all nature's
(2003). Stress memory in plants: A nega-
Retrieved from http://www.botanicus-inter-
in order to investigate the implications
skin”, given that we are not just extending
tive regulation of stomatal response and
acticus.com. Accessed February 22, 2013.
those can have on humans’ perception of
our own senses, but are interfacing to other
transient induction of rd22 gene to light in
living creatures as well.
abscisic acidentrained Arabidopsis plants,
The Plant Journal 36 (2): 240–255.
their environment. For this he enjoys be10.EU Pleased project website, http://pleasedfp7.eu, Accessed February 22, 2013.
ing on the crossroads of science, art and
technology. Recently, Rosen researched
whether sound-augmented human-plant
11. Manzella, V., Gaz, C., Vitaletti, A, Masi, E.,
Peter van der Putten
Peter van der Putten is a part time researcher at
interaction can have an effect on people’s
Santopolo, L, Mancuso, S., Salazar, D and de
relationships to plants. Apart from that,
las Heras. J.J. (2013). Plants as sensing de-
he often engages with different com-
vices: the PLEASED experience. In Proceed-
munities and projects where co-working,
ings of the 11th ACM Conference on Embed-
co-learning and Agile-like processes are
ded Networked Sensor Systems (SenSys '13).
involved.
ACM, New York, NY, USA.
the Leiden Institute of Advanced Computer Science,
Leiden University, the Netherlands, active in the
12.Flinley, K. The Internet of Vegetables: How
14.Bogdanov, R. (2013) Can sonic feedback from
Media Technology program as well as the Data Min-
Cyborg Plants Can Monitor our World. Wired
human-plant interaction increase connected-
ing group. His background is in artificial intelligence
Enterprise. Retrieved from http://www.
ness to plants? MSc Thesis, Media Technology,
and he is particularly interested how intelligence
wired.com/wiredenterprise/2014/01/inter-
Leiden University.
and complex systems in general can evolve through
net-plants/. Accessed February 22, 2013.
learning and interaction, in organisms as well as
machines. Peter has a MSc in Cognitive Artificial
15.Nisbet, E., Zelenski, J., & Murphy, S. (2009),
13.The Roots of plant intelligence,
Intelligence from Utrecht University and a PhD in
TED talk at http://www.ted.com/talks/
data mining from Leiden University, and combines
stefano_mancuso_the_roots_of_plant_
academic research with applying these technolo-
intelligence.html.
gies in business.
Accessed February 22, 2013.
46
The Nature Relatedness Scale, Environment
and Behavior, 41, 5, 715-740.
16.McLuhan, M. (1964) Understanding media: The
extensions of man, New York: McGraw-Hill.
47
Towards hybrid disciplines
in a postdigital world
Reflection on the future of art,
science and tech in arts education
by Isjah Koppejan
The Golden Orb Weaver is a tropical spider which weaves its web with a
unique material – stronger than steel, more elastic than nylon and a better
conductor than copper. Sparked off by a scientific article and her own imagination, artist Jalila Essaïdi discovered new applications for this startling
spider silk. Together with the Forensic Genomics Consortium Netherlands
she developed a piece of bulletproof skin. “Innovation takes place when you
look at something from different angels. I want Bulletproof Skin to show that
more is possible than you realise and to provoke discussion of how far we are
prepared to go.” She calls herself a hybrid artist: “I’m used to thinking beyond my own field”. [1] Essaidi is founder of BioArt Laboratories where artists can experiment with bio-based materials. This initiative, driven by the
dramatically increased progress in biotechnology, aims to make this progress
and its implications ‘the collective responsibility of society’ [2].
In the 21st century, a new form of creativity seems to be surfacing. Driven
not only by the ongoing developments in material technology, but also by
the rise of digital fabrication. In our postdigital world, artists and designers use these advanced means to create artistic expressions never before
thought possible. Recently, the Museum of Arts and Design in New York has
been examining trends in contemporary digital design and fabrication with
MyceliumChair, Studio Eric Klarenbeek
the exhibition and accompanying book Out of Hand: Materializing the Postdigital. Although the book and exhibition have a highly sculptural approach
it shows that digital technologies enable art and design to infiltrate the
boundaries of different disciplines and is changing its relationship to materiality and craftsmanship.
This takes place at the start of the exhibition demonstrated by a self-portrait
by the artist Richard Dupont: a nude male figure with melty, deformed contours, a sculpture that looks more like a refection in a distorting mirror. Dupont
has had his entire body scanned and ran the data through digital modelling
programs. It’s made with computer assistance at almost every stage, from the
design right through to the fabrication process, like digital milling and rapid
prototyping. Another more radical and disruptive example is the work Ob-
48
49
ject Breast Cancer, which has inspired a research
hancements. What’s the impact of Facebook, or
to the postdigital. Digital fabrication tools or new
project at Weill Cornell Medical College. This work
even of the future use of 3D bio printing to cre-
materials are not just the new paintbrush, the
uses 3D software to transform MRIs of tumours into
ate new organs, or the influence of nanotech? The
new musical instrument or the new way of mak-
small sculptures and pieces of jewellery, which
postdigital generation of artists and designers are
ing prototypes. This is why Essaidi has the urge
hint at the profound and often unnerving new ways
investigating, making use of and criticising the
to call herself a hybrid artist. She is redefining
that 3D technologies can explore the body.
implications of these ‘enhancements’.
the notions of creative production and creative
For a long time, art and design work aided by com-
Frustrated by internet restrictions, IDPW1 de-
different disciplines. Due to the nature of digital
puters was usually shown ‘on screen’ (computer,
signer/programmer Shunya Hagiwara made the
fabrication, the use of open source technology
projections, etc). Nowadays, the rise of digital
Whatever Button. The Whatever Button is an ex-
and the culture of sharing, the artistic process
fabrication has created the possibility to materi-
tended free online feature for the Google Chrome
is opening up. From ideation, concepting, mak-
alise the digital into physical, tangible forms. The
web browser, which enables you to click all the
ing and distribution to a process of co-creation,
influences of these developments on Augmented
Facebook Like buttons on a page, from top to bot-
coproduction, crowdsourcing and collaboration.
Reality are stronger than we might think. For ex-
tom. With one click you can automatically click
This changing practice requires new qualities in
ample, scientists have recently been working on
every “Like” button that appears on the screen,
an artist, as the next example demonstrates.
mediated touch, where you can touch things you
regardless of the content: the good, the bad and
cannot grab with an interactive glove (see, e.g.,
the ugly. This work can in part be seen as a criti-
[3,4]. The non-invasiveness of these new devel-
cism of online conditions. While following hacking
opments is interesting for different applications.
etiquette and defying social and technological
In the European MeSch project [5] for example,
conventions, the work is imbued with a richness
The Allard Pierson Museum has tested the use of
that transcends any criticism. In 2012 The What-
AR in its recent exhibition Eternal Egypt Experi-
ever Button designed was awarded the ‘New Face
ence. The introduction and enhancement of the
Award’ at the Japan Media Arts Festival. Shunya
materiality of the experience of museum objects
Hagiwara: “I know we don’t really need this, but
is a challenging one. How can you experience the
we depend so much on the architecture or struc-
physicality of an artefact which is behind glass and
ture arbitrarily set up by companies like Facebook.
can usually not be touched? The MeSCH project
I wanted to do something witty to make fun of it.
approach is grounded on principles of co-design:
My goal is to make people know about it, rather
the participation of designers, developers, scien-
than actually use it. If people know there is some-
tists and stake-holders into the process of creation
one out there who is thinking enough to make a
and evaluation as equal partners, and on a Do-It-
button like this, I think they won’t be tired liking
Yourself philosophy of making and experimenting.
things.” The rise of these ‘critical makers’ have
Richard Dupont, Untitled (#5), 2008, pigmented cast polyurethane. Courtesy Richard Dupont and Artists Rights Society (ARS), New York.
processes, which becomes more like a blend of
50
a practice that requires a split identity; one foot
Critical making
The term postdigital in artistic discourse is often described as the period after the beginning
of the age of digital design. [6] For me, the term
postdigital goes a step further and introduces a
way of thinking that deals with the consequences
planted in the craftwork of design and the other
foot planted in the reflexive work of critique. It
Vivian Hartung, A new laced wheel
requires the understanding of underlying systems,
Fashion designer Iris van Herpen is one of the
technologies, methods and concepts.
early pioneers in 3D prototyping in fashion. She
is well known for her 3D printed shoes for United
Co-creation melts crafts
and technology into one
Nude and for a 3D printed dress that TIME Magazine names one of the 50 Best Inventions of the
year 2011. Van Herpen collaborates with artist
from different disciplines in almost all of her
of the digital and the internet age, being more
concerned with being human than being digital.
‘Out of hand’ approaches the postdigital more or
projects. In an interview with SecondSight, Iris
It’s a sign of our changing relationships with digi-
less as an instrumental view, as a new way of mak-
van Herpen states: “Before I started this project
tal technologies and enforces us to look past the
ing, which creates a new kind of aesthetic. But I
[Bored Dress] I didn’t like computers and tech-
optimist or even utopian views of the digital en-
would like to emphasize a more holistic approach
nology. Now I see the possibilities for the future
51
rooms they developed a concept chair that is made
fashion and other aspects of everyday life and
by mixing water, powdered straw and mycelium.
finding ways to navigate through them and imple-
That mixture is then coated with a thin layer of
ment these findings. This is not an easy task. In
bioplastic, and as the mixture dries and the myce-
this period of economic decline, the arts, higher
lium grows, it creates what’s described as a solid
education and research institutions’ legitimacy is
but light material. “This material has such amazing
constantly under fire and these institutions are
qualities. The chair is merely a metaphor for what
facing serious budget cuts. In the light of two
can be achieved with new materials and produc-
examples (the ArTechLab and the Open design
tion methods. It could be a table, a whole interior,
minor), I would like to show that arts academies
or even a house”, according to Klarenbeek.
realise that this interdisciplinary open practice is
redefining notions of creative processes and pro-
This example of a new type of collaboration be-
duction and that new approaches are needed.
tween arts and science demonstrates the added
Bulletproof skin being pierced, '2.6g 329m/s' by Jalila Essaïdi.  JalilaEssaidi.com
value to combine the tacit with the explicit di-
Examples from a recent exhibition full of experi-
mension of knowledge. Artists and designers are
ment and exploration by the student-researchers
collaborating in new ways that drive innovation
of the ArTechLab at AKI ArtEZ include the tweak-
and are earning their place in the innovation pro-
ing and hacking 3D printers, photo cameras, 3D
cess. One of the projects I’ve worked on, called
scanners or opening up possibilities for new fabrics
CRISP Smart Textile Services, deals with the de-
to create a woven carbon bicycle wheel or to re-
velopment of applications for new ‘smart’ tex-
place plaster. In 2011, ArtEZ started the ArTechLab
tiles. ‘Smart’ stands for textiles that can conduct
as vrije minor (elective) for fine arts and design
heat, lights or currents and as a result become
students, as an innovative interdisciplinary spe-
an interactive product. In this project, an inspi-
cialisation for researching new materials and tech-
rational testbed of workshops is created where
nologies. This research is becoming much more of
are enormous. So much is going on right now,
also driven by the rising notion in recent studies
designers and engineers come together to ex-
a participatory process, as the technologies are
also in the production process. High-end fashion
that fine arts graduates contribute to innovation
periment and play with the materials and look
becoming more complex and co-operation with
doesn’t resolve around the big Parisian houses
throughout their working lives. The study “The art
beyond traditional approaches [9]. The increase
specialists from the field are needed and provided
anymore. It’s about new ways of working, coop-
of innovation” [8] shows that these artists have
of the involvement of artists and designers in
during the process. A team of advisors on which
erating, helping and strengthening each other”.
attitudes and skills that are conducive to innova-
research and development projects, indicates
students can rely on for their professional network
These new processes and methodologies require
tion, that they work in a way that is organised
that a new field is emerging where design, art
and technological, engineering or artistic advice,
different attitudes and skills from the artists and
around portfolio and project work and continually
and science will persistently make use ofeach
coach the student researchers. Another specializa-
designers. As Iris van Herpen puts it: “I am not
have ‘crossover’ and cross-fertilisation of people
other’s interfaces. This new area requires artists
tion at another arts academy, the WdKA in Rotter-
technical myself and the project involves quite a
and ideas across the arts, and within the arts and
and designers to rethink their role and the value
dam, under the name of Open Design Minor deals
lot of software aptitude which really requires the
outside of the arts. Artists and designers have a
of their creativity and imagination. This role re-
with similar implications but specifically focuses on
aid of a specialist. That took some getting used
flexible way of thinking and acting, try to keep it
sembles that of a composer taking the creative
the use of open source technologies and methods.
to: you speak different languages and you have to
simple, follow their passion, do more with less and
lead in the project while speaking different lan-
It deals with questions like: How can you express
develop a common one”.[7]
find possibilities in difficult circumstances. These
guages and conducting everyone’s imagination.
authorship, your identity and position as a designer
Innovation by art
qualities are valued in innovative areas. Think for
in a participatory and open process? What are the
instance of the development of the 3D printed
aesthetics and poetics of Open Design?
For companies, science, artists and designers alike,
passionately: “This material mycelium, the thread-
New approaches in art
education
new materials and technologies offer possibilities
like vegetative part of fungus, is amazing, yet we
These new movements find their way into arts
cal and social developments as a new art discipline
for fascinating innovations. Recently, artist and
know so little about it! Even the scientists are just
and design academies. Art Academies are taking
or separate field. They try to make use of this in-
designers have been playing a more significant
touching the surface.” In cooperation with the
a closer look at the global impact that new tech-
terdisciplinary trend that renders the convergence
role in research and development projects. This is
University of Wageningen and CNC Exotic Mush-
nologies have had on art, science, architecture,
of disciplines from within and outside the arts and
52
chair of mycelium by Studio Eric Klarenbeek. In
one of our conversations, Eric Klarenbeek declared
I emphasize these two examples, as their approach
is similar; they do not approach these technologi-
53
design world. Rather, these specialisations stress
the need of a new way of working and therefore
emphasize skills like hands-on artistic research,
Notes
tion through mediated touch.” CHI’13
1. IDPW is a self-proclaimed ‘100-year-old
Computing Systems. ACM, 2013.
Extended Abstracts on Human Factors in
collaboration, co-creation, craftsmanship and
secret internet society’, which organizes
5. http://mesch-project.eu
crit­ical reflection. Students experience these ap-
events that combine internet with real
6. Alexenberg, M. (2011). The Future of Art
proaches as refreshing and different from the rest
places to do something new and experimen-
in a Postdigital Age: From Hellenistic
of the curriculum, which poses new challenges.
tal. See www.idpw.org
to Hebraic Consciousness. Bristol and
How do you connect these new approaches with
the current curriculum? What kind of expertise do
you ask of teachers? How do you support students
in their search for their own authenticity while
working with others? How do you facilitate the
Chicago: Intellect Books/University of
Chicago Press
References
7. Wiegman, Andrea. “Bored dress.” Second-
1. Chris Gruijters, Chris, and Koert van
8. Oakley, Kate, Brooke Sperry, Andy C.
Sight 1 July 2011: 110-111. Print.
use of future technologies? How can you create a
Mensvoort. Crossover works #2. Amsterdam:
Pratt, and Hasan Bakhshi. The art of inno-
support structure for students’ cooperation with
Federation of Dutch Creative Industries,
vation how fine arts graduates contribute
companies and scientists? The road to the answers
2014. Print.
to innovation. London: NESTA, 2008.
of these questions will help us to figure out how to
2. http://bioartlab.com
strengthen the combination of art, technology and
3. http://hmi.ewi.utwente.nl/MeTA/
research in future art education practice.
4. Huisman, Gijs, and Aduén Darriba Frederiks.
“Towards tactile expressions of emo-
ArTechLab, cross discipline research lab, based at the ArtEZ Enschede
Print.
9. Koppejan, Isjah. “Making opportunities
tangible.” CRISP magazine 1 Apr. 2013:
20-23. Print.
ArTechLab
Isjah Koppejan
ArTechLab is a cross discipline research lab
Isjah Koppejan is the programme man-
based at the Artez Enschede. The lab initi-
ager and lecturer for the Minor course
ates cross discipline projects, in collabora-
ArTechLab at the AKI ArtEZ Academy
tion with other institutes, universities and
for Art and Design in Enschede. She is
companies. As we have a broad team of
responsible for student supervision and
researchers, we’re able to make proposals
for developing the ArTechLab from a
and do research on complex artistic, social
minor course to a master’s programme.
and design issues. Every year we start with
a new team of student researchers. They
Before joining the ArTechLab, Isjah
have a variety of backgrounds. Most of
worked in research and innovation
them are studying at different departments
at the Waag Society and at the HKU
of ArtEz, or at other institutions as Saxion
University of The Arts Utrecht. Isjah
or University of Twente. Our student re-
initiated and realised many innovative
searchers are creative thinkers, are pas-
concepts and projects at the inter-
sionate about what they do and have fresh,
section of culture, ICT, science and
innovative ideas. The researchers present
economics. She specialises in translat-
their research ideas and results at www.
ing social developments and trends into
artez.nl/artechlab. Our team of advisors
strategies, products, services and useful
coaches the researchers. This team con-
ideas.
sists of Eric Klarenbeek, Filip Jonker, Paul
Jansen Klomp, Isjah Koppejan, David Menting, Mark Bakema en Thomas Choduba.
 www.artez.nl/artechlab
54
55
Site Venice Site Biennale:
The Manifest.AR Augmented Reality
Intervention into the 2011 Venice Biennial
Challenging and Exploiting
the Primacy of Site
Artistically, our works often stand in dialogue
Manifest.AR [1] originally formed around an AR
presence of our artworks at the site increasing
intervention into the United States’ most iconic
the potency of their argument.2 In a time when
contemporary art space: the Museum of Modern
many question the relevance of galleries, muse-
Art in New York. In 2010 Sander Veenhof and Mark
ums and biennials – the gated communities of the
Skwarek realized that the institutional walls of
art world – we bring a new form of dialogue into
the white cube were no longer solid, and orga-
their institutions. [5]
with the “official” artworks at a venue, and with
the curator’s theme and concept – with the visual
nized a guerilla exhibit of augmented reality artworks inside the walls of MoMA.1
by Tamiko Thiel
Since time immemorial location has been used
to consecrate objects and people. In the art
world too, access to a location – a gallery, a
Manifest.AR Venice
Biennale Intervention:
Themes and Concerns
museum or other curatorially closed space – is
Introduction
In 2011, using geolocative augmented reality (AR),
the author was the primary organizer of the Manifest.AR cyberartist group intervention into the Venice Art Biennale, together with fellow artists Sander
Veenhof and Mark Skwarek (Manifest.AR Venice
Biennale Manifesto 2013). Using GPS coordinates
we placed virtual artworks – visible in smartphone
displays as overlays on the live camera view of the
surroundings – inside the curatorially closed spaces of the Biennale.
Unlike physical interventions, the artworks cannot
be removed or blocked by authorities. The artworks
exploit the site as their canvas while simultaneously
questioning the value of location, and the power of
the curator as gatekeeper, to canonize works of art.
The Venice Biennale, founded in 1895, is the world’s
oldest art biennial and the city’s main claim to relevance as a contemporary art destination. In the
intervention we wished to question the biennial system, and the art world’s use of that system to define
artistic value, but also address the site as artists:
the reality of Venice’s contemporary concerns and
of life in the city today.
56
tightly controlled to confer value and thus, via
At the 2011 Venice Biennale we wished to reflect
this exclusivity, to canonize the works shown
not on Venice’s past glory, but on its current con-
there as “high art.” What does it mean however
dition: not only wrestling with climate change
to control physical space when in geolocated
and overrun by tourists, but also fighting for rel-
virtual space anyone can place whatever they
evance in the art world. The national pavilions
want? [2]
that dominate the Venice Biennale reflect its origins at the end of the 19th century and the rise
Technically, it is a trivial difference in GPS coor-
of the nation-state with a presumed monolithic
dinates that moves a virtual object from a pub-
ethnic or cultural identity. They stand now in di-
lic space such as Central Park to the curatorially
rect contrast to the globalized, itinerant world of
closed space inside the sacred walls of MoMA.
contemporary artists and their multiple systems
The epiphany of AR however is that although the
of cultural reference. [6]
artworks are virtual, their presence at the site is
real, “actually existing as a thing or occurring in
Curator Bice Curiger’s opening statement ques-
fact; not imagined or supposed” [3], reproducible
tioned this structure as well: “By adopting the
by anyone who views the artwork at that loca-
title ILLUMInations the 54th International Art Ex-
tion. In this “consensual hallucination,” that was
hibition of the Venice Biennale also aspires liter-
the dream of the early cyberpunk authors and
ally to shed light on the institution itself, drawing
virtual reality evangelists [4], augmented reality
attention to dormant and unrecognized opportu-
redefines the barriers between “the real” and
nities, as well as to conventions that need to be
“the virtual.”
challenged... Far removed from culturally conservative constructs of ‘nation,’ art offers the poten­
The artworks engage viewers with the site physi-
tial to explore new forms of ‘community’ and ne-
cally as well. Like bird watchers with binoculars,
gotiate differences and affinities that might serve
AR viewers scan their surroundings with their
as models for the future.” [7] Curiger also posed
smartphones, dodging real world obstacles in
five questions on identity to each of the artists
search of the artwork, situating themselves and
officially included in the Biennale: “Where do you
the act of viewing in their physical experience
feel at home? Does the future speak English or
of that site.
another language? Is the artistic community a na-
57
tion? How many nations do you feel inside your-
54th Biennial of Venice could not justify its reputa-
self? If art was a nation what would be written in
tion without an uninvited Manifest.AR Augmented
its constitution?” 3
Reality intervention. In order to “challenge the
conventions through which contemporary art is
Manifest.AR Artworks in
the Venice Biennale Intervention
As an international artist collective that co-
viewed” we have constructed virtual AR pavilions
Tamiko Thiel’s Shades of Absence is a series of
alesced around challenging conventions of in-
directly amongst the 30-odd buildings of the lucky
three “virtual pavilions” in the Giardini, in Piazza
clusion and participation, we saw this as a per-
few within the Giardini. In accordance with the
San Marco and inside the German National Pa-
sonal invitation to participate. Sander hijacked
“ILLUMInations” theme and Bice Curiger’s 5 ques-
vilion. Anonymized golden silhouettes of artists
Curiger’s curatorial statement and the Venice
tions our uninvited participation will not be bound
whose works have been censored are enclosed
Biennale website to create our Venice Manifesto,
by nation-state borders, by physical boundaries
by terms of censorship. In reply to Bice Curiger’s
in which we proclaimed:
or by conventional art world structures. The AR
questions: “Is the artistic community a nation? If
pavilions at the 54th Biennial reflect on a rapidly
art was a nation what would be written in its con-
As “one of the world’s most important forums for
expanding and developing new realm of Augment-
stitution?” they posit a transnational community
the dissemination and ‘illumination’ about the
ed Reality Art that radically crosses dimensional,
of censored artists. Touching the artworks in the
current developments in international art” the
physical and hierarchical boundaries. [8]
display of a smartphone calls up a website with
Figure 2. Shades of Absence: Public Voids, Tamiko Thiel, 2011.
Augmented Reality, Piazza San Marco, Venice. A memorial for
artists whose works in public spaces have been censored.
cases of censorship. [10]
Sander Veenhof’s Battling Pavilions directly addresses the role of the curator, the exclusivity of
the Giardini and the limited number of national
pavilions allowed within its Sacred Grove. Users
outside the Giardini can subvert Curiger’s authority and create new virtual pavilions for nations
of their choice inside the Giardini. Users inside
the Giardini, in contrast, can help Curiger defend
the Giardini against intruding pavilions by deleting them. In a classic twist, Sander’s interven-
Figure 3. Battling Pavilions, Sander Veenhof, 2011. Augmented
tion also became an official part of the Biennale:
Reality Game. Scoreboard on dropstuff.nl screen during the
dropstuff.nl invited him to show his Battling Pa-
Venice Biennale, displaying scoreboard of unauthorized virtual
vilions on their large screens in three locations
pavilions in the Giardini.
around Venice. [11]
Mark Skwarek’s Island of Hope addresses the
perpetual threat of Venice sinking into the lagoon. Skwarek posits new forces of continental
uplift erupting as fully formed baroque gardens
into the Giardini and in Piazza San Marco. The
Figure 1. Manifest.AR Venice Biennial Intervention website.
islands are full of objects of hope, and tweets
with the hash tag #hope, in order to bring hope
back to Venice. [12]
Questions about control of space are not confined
controlled curatorial space of the Venice Giardini,
John Craig Freeman’s Water wARs: Squatters
to art venues; “public” art is always dependent
but also in the public space of Piazza San Mar-
Pavilion is a virtual squatter’s camp for refugees
Figure 4. The Island of Hope, Mark Skwarek, 2011.
on permissions from authorities, and many a
co, which has itself seen censorship of officially
of water wars, one inside the protecting walls of
Augmented Reality. Seen in the Venice Giardini.
“public” space is actually closely controlled. We
planned artworks. [9]
the Giardini, and another “public” camp in Piazza
therefore placed our artworks not only in the
58
San Marco. In Venice, a city founded by refugees
59
the rest of the world, as worldwide ecological
Naoko Tosa’s Historia addresses Curiger’s ques-
disasters drive people in desperation to violate
tion “Does the future speak English or another
the boundaries of the nation-states in pursuit of
language?” and her view that “art offers the
sheer survival. [13]
poten­tial to explore new forms of ‘community’
and negotiate differences and affinities that
In John Cleater’s Sky Pavilions ships from outer
might serve as models for the future.” Historia
space take over Venice: The mothership hovers over
appropriates iconic images from all nations and
Piazza San Marco emitting a mixture of nonsense
world cultures, modern and ancient, and allows
and guidance to confuse and help tourists, natives,
visitors to arrange them in sequences, assigning
and art seekers. In the Giardini alien “Floaties” lie
them new meanings. It thus playfully examines
in wait, begging to be touched, and when activated
the process by which artists appropriate and re-
Figure 5. Water wARs, Giardini, John Craig Freeman,
by obliging visitors spin upwards, carrying secret
define existing cultural symbols to create their
Figure 7. The Crystal Coffin, Piazza San Marco, Lily & Honglei,
2011. Augmented Reality. Pavilion for undocumented
messages to the mother ship. Sky Pavilions goes
own individual languages. [17]
2011. Augmented Reality. Artwork inspired by the crystal coffin
artists/squatters and water war refugees in front of
beyond the concept of the nation-state, beyond
the Giardini Central Pavilion.
the concerns of mere earthbound humanoids and
reminds us that the last word in the control of space
may not be ours to decide. [14]
Lily and Honglei’s The Crystal Coffin: Virtual
China Pavilion is inspired by China’s petrified
in the Mausoleum of Mao Zedong in Tienanmen Square, seen
here in Piazza San Marco.
To end with a quotation
from Bice Curiger’s curatorial text for the Venice
Biennale:
symbol of eternal Party rule, Mao Zedong’s crystal coffin. In the Giardini it questions the tradi-
“ILLUMInations presents contemporary art char-
tional hierarchy of privilege among national pa-
acterized by gestures that explore notions of
vilions in the Biennale and thematizes the rise of
the collective, yet also speak of fragmentary
China as an important center of contemporary
identity, of temporary alliances, and objects
art. Another pavilion in Piazza San Marco domi-
inscribed with transience. If the communicative
nates the heart of Venice, whose native son Mar-
aspect is crucial to the ideas underlying ILLUMI-
co Polo “discovered” China for the West, with this
nations, it is demonstrated in art that often de-
symbol of Chinese Party power. [15]
clares and seeks closeness to the vibrancy of life.
This is more important now than ever before, in
Figure 6. Sky Pavilions, John Cleater,
2011. Augmented Reality and audio.
Alien Mothership Sky Pavilion floats over
Piazza San Marco.
Will Pappenheimer/Virta-Flaneurazine’s Colony
an age when our sense of reality is profoundly
Illuminati appropriated both the Biennale title
challenged by virtual and simulated worlds.
“ILLUMInations” and the actual visual imagery of
This Biennale is also about believing in art and
many artworks in the Biennale. A secret colony
its potential.” [7]
Figure 8. Colony Illuminati, Will Pappenheimer/Virta‐Flaneurazine, 2011. Augmented Reality. Colony group on Giardini main
concourse.
of virtual bufo toads draws sustenance from high
art; as a form of camouflage their skins appropri-
I could not agree more. Perhaps in ways that Bice
ate imagery from artworks around them in the
Curiger did not anticipate.
Giardini and spread out into the city, seeking the
outlying venues of the Venice Biennale. When
At the latest since the Salon des Refusés in 1863,
touched on the smartphone screen, the toads
questions about the validity of the art canon and
release psychotropic drugs that trigger halluci-
the institutions that define this canon have been
and threatened by constant flooding, Water wARs
nations in the viewer: a swirl of Internet infor-
an important part of the evolution of modern art.
calls attention to the escalating global struggle
mation on the Biennale and waves of Tintoret-
Augmented reality interventions are the continua-
for this basic human need. It questions the ability
toesque ecstasy that Curiger proclaimed to be
tion of this modernist dialogue with 21st century
the true essence of ILLUMInations. [16]
icons floating in the space and assigning a new meaning
of sovereign nations to isolate themselves from
means.
to their message. Seen in front of the Giardini Central
Figure 9. Historia, Naoko Tosa, 2011. Augmented Reality. Users compose messages by appropriating historic
Pavilion.
60
61
Coda: The Future of AR Interventions
References
But how will the law react to increasing trans-
Can institutions use these existing laws to assert
1. Manifest.AR AR Art Manifesto (2011).
gressions in virtual space? By 2013 technologies
“virtual air rights” to “their” GPS coordinates,
Manifest.AR artist group official website.
(2011) Venice Biennial 2011 AR Intervention
such as Google’s Street View and Glass were pro-
thus blocking AR interventions? Intellectual prop-
http://www.manifestar.info, accessed 12
by Cyberartist Group Manifest.AR,
voking wide public discussion of the confluence of
erty lawyer Brian Wassom thinks not: “Property
March 2013.
Reflection on the official curatorial context.
locative, mobile, recording and display technolo-
law is about the right to exclude others from
gies, and what negative effects could come of the
physical space. But an infinite number of people
2. Aceti, L. (2008) The Virtual Places We own:
blurring of boundaries between real and virtual
can each create their own AR layer superimposing
When Communities and Artists occupy
To view the actual artworks in Venice go to
space. Most public unease comes however not
digital data over the same physical space with-
Your Place without Your Consent. Internet
the launch page – only accessible on mobile
from AR display technology, but from recording
out impeding anyone else’s ability to do so, and
Research 9.0: Rethinking Community,
devices: http://manifestar.info/vb11/
(“surveillance”) technology. As Yolande Kolstee
without invading the rights of the real property
Rethinking Place: 15–18.
(both accessed 14 March 2013).
points out, the real debate here is not techno-
owner.” [19]
logical but social, and can probably be negotiated
using existing legislation [18].
8. Manifest.AR Venice Biennale Manifesto
http://www.manifestar.info/venicebien-
3. Oxford English Dictionary. Definition of
nial2011/
9. Magill, R. J. Jr. (2007, 16 April) For Gregor
“real.” http://oxforddictionaries.com/us/
Schneider’s cube, a long pilgrimage.
definition/american_english/real, accessed
New York Times.
12 March 2013.
http://www.nytimes.com/2007/04/16/
arts/16iht-cube.1.5303319.html,
4. Gibson, W. (1984) Neuromancer.
accessed 14 March 2013.
Ace Books, New York.
10.Thiel, T. (2011, May) Shades of Absence. Mani5. Fidel, A. (2010) Art Gets Un- masked in
Notes
fest.AR Venice Biennale 2011 Intervention.
the Palm of Your Hand. New York Times
http://manifestarblog.wordpress.com/thiel_
http://www.nytimes.com/2010/12/02/
venice-2011/, accessed 12 March 2013.
arts/02iht-rartsmart.html, accessed 30 April
2012.
1. “We AR in MoMA” [20] was part of the Conflux Festival of Psychogeog-
11. Veenhof, S. (2011) Battling Pavilions. Manifest. AR Venice Biennale 2011 Intervention.
raphy [21]. Cyberpunk author Bruce Sterling blogged the intervention
6. Madra, Y. (2006) From Imperialism to Trans-
on WIRED [22], MoMA tweeted a somewhat nonplussed acknowledg-
national Capitalism: The Venice Biennial
sander-veenhof-venice-biennial-2011/,
ment [23], and in a New York Times interview MoMA’s director of digital
as a ‘Transitional Conjuncture.’ Rethinking
see also:
media welcomed our engagement [5].
Marxism 18(4).
http://www.sndrv.nl/battle/, accessed 14
http://www.academia.edu/2072820/
March 2013.
2. The author’s contribution to “We AR in MoMA” was a matrix of scre-
From_Imperialism_to_Transnational_Capital-
aming faces titled “ARt Critic Face Matrix,” a self-referential artwork
ism_The_Venice_Biennial_as_a_Transitional_
that critiqued its own validity as an artwork, reflecting on the role of
Conjuncture_, accessed 14 March 2013.
MoMA NY to define what did or did not constituted art. [24]
12.Skwarek, M. (2011) Parade to Hope. Manifest. AR Venice Biennale 2011 Intervention.
http://manifestarblog.wordpress.com/
7. Curiger, B. (2011) Introduction by Bice
3. Although Curiger refers frequently to the “five questions,” they are not
http://manifestarblog.wordpress.com/­
Curiger. ILLUMInazioni – ILLUMInations.
to be found on the official Venice Biennale website. See however Flash
Venice Biennale website.
Art [25]
http://www.labiennale.org/en/art/
skwarek-venice-2011/, accessed 12 March
2013.
13.Freeman, J.C. (2011) Water wARs: Squatters
archive/54th-exhibition/curiger/,
Pavilion. Manifest.AR Venice Biennale 2011
accessed 12 March 2013
Intervention website. http://manifestarblog.
wordpress.com/freeman-venice-2011/,
accessed 14 March 2013
62
63
14.Cleater, J. (2011) Sky Pavilions. Manifest.AR
21.Conflux Festival (2010) We AR in MoMA.
Venice Biennale 2011 Intervention website.
https://web.archive.org/
http://manifestarblog.wordpress.com/
web/20100928175908/http://www.
cleater-venice-2011/, accessed 14 March
confluxfestival.org/projects/conflux-
2013.
festival-2010/we-ar-in-moma/,
accessed 12 May 2014.
15.Lily & Honglei (2011) The Crystal Coffin:
22.Sterling, B. (2010, 6 october) Augmented
Biennale 2011 Intervention website.
Real- ity: AR uninvited at MoMA NYC.
http://manifestarblog.wordpress.com/lily-
Beyond the Beyond. WIRED.
honglei-venice-2011/, accessed 14 March
http://www.wired.com/beyond_the_
2013.
beyond/2010/10/augmented-reality-aruninvited-at-moma-nyc,
16.Pappenheimer W., Virta-Flaneurazine (2011)
accessed 30 April 2012.
Colony Illuminati. Manifest.AR Venice
Biennale 2011 Intervention.
http://manifestarblog.wordpress.com/
23.MoMa (Museum of Modern Art) NY Twitter
site (2010, 8 october). Nice, looks like we’re
pappenheimer-venice-2011/, accessed 12
havin an “uninvited” AR exhibition tomor-
March 2013.
row! Part of @confluxfestival.
http://twitter.com/MuseumModernArt/
17. Tosa, N. (2011) Historia. Manifest.AR Venice
Biennale 2011 Intervention.
statuses/26786135774,
accessed 12 March 2013.
Tamiko Thiel
Virtual China Pavilion. Manifest.AR Venice
http://manifestarblog.wordpress.com/naokotosa-venice-biennial-2011/, accessed 30 April
2012.
18.Kolstee, Y. (2013, May). Who owns the space
24.Thiel, T. (2010) We AR in MoMA exhibit.
Tamiko Thiel website.
Tamiko Thiel is an internationally known
IBM Innovation Award for Art and Technol-
http://www.tamikothiel.com/We-AR-in-
visual artist exploring the interplay of place,
ogy, FACT Liverpool and Zero1 Biennial. She
MoMA/, accessed 10 May 2014.
space, the body and cultural memory. She is a
is also augmented reality artistic advisor for
founding member of Manifest.AR, participat-
the Caribbean Cultural Center and African
ing in 2010 in the pathbreaking augmented
Diaspora Institute’s augmented reality project
2, AR[t], issue 3, 40-43.
25.Flash Art (2011) Bice Curiger speaks about
the Venice Biennale.
reality intervention at MoMA NY, and being
'Mi Querido Barrio' in Spanish Harlem, NY, for
Free Speech. Augmented Legality blog.
http://www.flashartonline.
the main curator and organizer of their 2011
which she helped bring in a Rockefeller Foun-
http://www.wassom.com/from-the-archives-
com/interno.php?pagina=news_
AR intervention at the Venice Biennale.
dation Cultural Innovation Award.
augmented-reality-as-free-speech.html,
det&id=953&det=ok&title=Bice-Curiger-
accessed 2 April 2014.
speaks-about-the-Venice-Biennale,
Her works are featured in the reference
Her guest professorships include Carnegie-
accessed 12 March 2013.
books Digital Art (Whitney curator Christiane
Mellon University, UC/San Diego, Bauhaus-
19. Wassom B (2014) Augmented Reality as
Paul - Thames and Hudson World of Art), The
University Weimar, the Berlin University of
MoMA oct 9th 2010 AUGMENTED REALITY
World of Digital Art (DAM director Wolf Lieser)
the Arts and in 2014 at Nanyang Technological
art invasion! Sander Veenhof website.
and “Not Here Not There” AR special issue of
University School of Art, Design and Media
http://www.sndrv.nl/moma/, accessed 12
Leonardo Electronic Almanac.
(ADM), Singapore.
March 2013.
20.Veenhof, S. (2010, 9 october) DIY day
Her grants and fellowships include the Mac-
Tamiko Thiel
Dowell Colony, WIRED Magazine, Japan Foun-
64
dation, MIT, Berlin Capital City Cultural Fund
 [email protected]
(Hauptstadtkulturfonds), Goethe-Institut,
 www.tamikothiel.com
65
Hitting imaginary walls,
pulling virtual strings
What augmented reality can learn from urban dance
by Hanna Schraffenberger
66
A few weeks ago my colleagues convinced me
your arms should not hit the walls.” To be hon-
to join their weekly Hip Hop fitness exercise at
est, this tip didn’t help me at first. Rather,
the University Sports center. Moving my limbs
I was distracted – those invisible walls remind-
in the rhythm of well-known radio hits turned
ed me of my research into augmented reality
out to be more difficult than I had anticipated.
(AR) and the presence of virtual objects in real
After all, I had been running to similar music on
space. These walls we had to avoid were solely
a regular basis [1]! A particularly difficult move
a product of our imagination. Nonetheless, our
required us to turn 360 degrees while at the
movements acknowledged their presence. The
same time imitating a windmill with our arms.
walls were, in a most basic and fundamental
In order to help us get the movement right, our
way, becoming part of and augmenting our sur-
instructor gave us a simple but effective hint:
roundings... Could we call this a form of imagi-
“Imagine two walls, one in front of you and one
nation-based AR? Could it be that dance and AR
behind you. You can only move between them,
had more in common than I thought?
67
No technology required!
Only minutes later this suspicion got confirmed.
same time mesmerized the audience with move-
By now, our hands were connected to our feet
ments that made us doubt whether his hands
with imaginary strings. In order to move our
were constrained by the same kind of bones
First of all, dance teaches us that there are
not adhere to our physical laws; they commonly
we had. Among the vid-
alternative means to display virtual objects in
float in space, right before the dancer. At the
eos that were shown, one
space besides AR technology. AR most common-
same time, the way a dancer moves them about
dancer had left a lasting
ly uses smartphone screens, heavy headsets or
in space implies that they do, however, have a
impression: Albert Hwang,
other kinds of visual displays that overlay the
certain mass – it just does not cause them to
a master in making three
real world with virtual elements. In illusion-
fall down. And of course, unlike real objects,
dimensional boxes appear
based dance, imaginary objects are revealed
these imaginary objects are essentially invisible
in real space – solely by
to the audience through a dancer’s body move-
and certainly do not occlude what’s placed be-
running his hands through
ment. The dancer can, for instance, run his or
hind them. More than that, they often appear
thin air. A quick look at
her hands over the shape of an imaginary object
out of nothing just to disappear in thin air a few
his YouTube channel [3]
in order to make it appear as if the object is ac-
seconds later. Fascinatingly, it does not bother
decided the matter; I had
tually present [5]. Illusion-based dance reminds
us that these imaginary objects are not really
to find out how dancers
us that AR is not restricted to digital mediums
present, don’t look like real objects and do not
created the illusion that
and that we do not have to resort to computer
behave like anything we know from the physical
imaginary objects existed
technology in order to make virtual objects ap-
world – the objects are believable and convinc-
in space, I wanted to know
pear in real space. Maarten H. Lamers discussed
ing nonetheless (cf. [11])!
“I wanted to know how much
illusion-based dance styles
and Augmented Reality had
in common and I definitely
had to master some of those
movements myself.”
from real objects. First of all, dance-objects do
how much illusion-based
the Pepper’s Ghost as an instance of pre-digital
feet, we had to pull the strings. To my surprise,
dance styles and augmented reality had in com-
AR [6] in the third issue of AR[t]. In this regard,
when our teacher illustrated the movement, it
mon and I definitely had to master some of those
dance-illusions can serve as yet another compel-
appeared as if those strings did, indeed exist.
movements myself.
ling example of AR that remains in the physical
some part of me was fooled into believing that
What you see isn’t what
you get
domain.
Although I knew that they were merely imaginary, and even though I could not see the strings,
these ‘dance-objects’ could not differ more
Dance AR?
they were actually there. Given the teacher’s
I expect multimodal AR to become one of the more
Realism, really?
interesting topics in the future. However, I do not
think that a multimodal or richer sensory experi-
movement, her hands and feet simply had to be
Compared to learning the basics of liquid danc-
connected by a thin, invisible rope! There was
ing, my theoretical considerations were rather
AR should be more like reality and virtual objects
based dance styles, Diego et al. [2] make an in-
no digital technology required, I was not wear-
simple. AR and illusion-based dance styles have
should both look and behave like real, physical
teresting observation: when dancers let imaginary
ing a headset, nor was I staring at a screen: a
one central aspect in common: both create the
objects! At least, this is the impression I get
boxes appear in space through their movement,
relatively simple movement was sufficient in
impression that virtual objects actually exist in
from existing AR research. Scientists and devel-
the viewer can interpret this in two different ways.
order to convey the presence of virtual objects
our real, physical environment. If we understand
opers strive for photorealism, they struggle with
Either there is no box in space and the dancer is
(or, to be precise, virtual strings) in real space.
augmented reality as a concept of combining and
occlusion and investigate how virtual objects
moving in a very complicated way or there is a box
It might not have looked like it, but watching
relating the virtual and the real [4] rather than
can cause reflections and cast shadows just like
in space that guides the movement of the dancer’s
these invisible ropes certainly felt a lot like AR!
a collection of technologies, it is not far fetched
real objects do (see, e.g., [7, 8, 9]). Likewise,
hand. While watching, our eyes tell us that there
to think of these dance-illusions as a time and
virtual objects should behave and interact with
is no box but our body (or our embodied cognition)
Over the next days, aching muscles reminded
movement based form of augmented reality.
the world like real objects [10]. A virtual ball is
tells us that there is. Diego et al. propose that it
me to investigate this phenomenon further.
What is more, the traditional, technology-fo-
supposed to drop and bounce on the floor, just
is “this moment of embodied/cognitive dissonance
Luckily, I already knew where to start. In 2013,
cused field of AR can learn quite a few things
like a real ball would. There is certainly noth-
[that] makes the movement compelling” [2, p. 173].
I attended a presentation about illusion-based
from urban dance!
ing wrong with that. However, illusion-based
I believe that AR can benefit from a similar dis-
dance shows us that another approach is pos-
sonance: looking at a breakfast cereal box through
dance by Diego Maranan at the Creativity and
ence is always better. In their paper on illusion-
Cognition conference in Sydney [2]. During his
So how does urban dance approach the virtual
sible. Dance shines when it comes to expressing
our phone’s screen, we see the virtual dinosaur
talk, Maranan not only illustrated technologi-
and how do their methods inform the general
simple geometrical shapes and structures, such
eating our cereal, but we cannot touch it. Our eyes
cal metaphors used in the urban dance styles
field of AR?
as rectangular boxes or walls. In some respect,
tell us “it is there” while our body tells us that it
‘liquid’, ‘digitz’ and ‘finger tutting’, but at the
68
69
isn’t. I do not claim that all AR benefits from such
Likewise, I am sure you cannot build any virtual
a dissonance. But I am convinced that getting con-
AR walls without imagining them beforehand.
tradicting information from our different senses
can actually add to, rather than subtract from, the
In the future, AR will surely overcome many
overall experience.
technical challenges. However, the future of
augmented reality is not only about what is or
The power of movement
5. Hwang, A. How to Dance Liq-
Real-time Estimation of Light Source Environ-
Wwxwc_2vDKQ?t=7m50s
ment for Photorealistic Augmented Reality.
In ICPR (2) (pp. 911-914).
6. Lamers, Maarten H. (2013, May). Pre-digital
Augmented Reality. AR[t] 3, 24-25.
10.Kim, S., Kim, Y., & Lee, S. H. (2011, July).
On Visual Artifacts of Physics Simulation in
will be possible technically. It is also about what
we can imagine and how our imagination works.
9. Kanbara, M., & Yokoya, N. (2004, August).
uid: Rails [video], http://youtu.be/
7. Agusanto, K., Li, L., Chuangui, Z., & Sing, N.
Augmented Reality Environment. In Inter-
One of AR’s unique powers is that it can be dif-
W. (2003, October). Photorealistic render-
national Symposium on Ubiquitous Virtual
Ultimately, AR can learn from illusion-based
ferent from our real, unaugmented reality. But
ing for augmented reality using environment
Reality (ISUVR), 2011 (pp. 25-28). IEEE.
dance that movement is a powerful means to
how can virtual objects differ from real objects
illumination. In Proceedings of the 2nd IEEE/
express the presence and properties of virtual
without losing their believability? How can aug-
ACM International Symposium on Mixed and
content. By moving virtual objects through
mented reality differ from reality? Studying re-
Augmented Reality (pp. 208-216). IEEE.
space, AR can communicate that which it could
lated arts such as dance, mime or magic helps us
hardly convey otherwise. If a virtual leaf moves
find answers and think outside of our imaginary,
through space in a certain way, its movement
invisible and virtual boxes.
11. Schraffenberger, H., & Van der Heide, E.
(2013). From coexistence to interaction:
influences between the virtual and the real
8. Gibson, S., Chalmers, A., Simon, G., Viguer-
in augmented reality. In Proceedings of the
as-Gomez, J. F., Berger, M. O., Stricker, D.,
19th International Symposium of Electronic
shows us that there is wind. If a virtual ball rolls
& Kresse, W. (2003). Photorealistic augment-
Art, ISEA2013, Sydney.
over a real floor, it tells us something about
ed reality. In Second IEEE and ACM Interna-
its weight and resistance. Furthermore, using
References
Reality-ISMAR'03.
movement, we are able to create the impression of yet other – invisible – objects being pres-
tional Symposium on Mixed and Augmented
1. Schraffenberger, H. (2012, November).
ent in space. How would you display an invis-
Chasing virtual spooks, losing real weight.
ible wall with AR technology? Dance gives the
Augmented Running and a side trip into the
answer: by having something bump against it, by
history of Audio Augmented Reality. AR[t] 2,
movement! And there are more possibilities: if
48-51.
a virtual object looks heavy but moves through
space weightlessly, we might be able to discern
2. Maranan, D. S., Schiphorst, T., Bartram,
a change in gravity. By rewinding their move-
L., & Hwang, A. (2013, June). Expressing
ments, good dancers are almost able to fool me
technological metaphors in dance using
into believing that time goes backwards. Maybe
structural illusion from embodied motion. In
AR technology can evoke a feeling of time mov-
Proceedings of the 9th ACM Conference on
ing differently by rewinding the movement of
Creativity & Cognition (pp. 165-174). ACM.
objects or by varying their speed. I hope future
AR will explore what can be expressed by simply
moving virtual objects through real space.
Future AR is not reality,
it is our imagination
3. Hwang, A., YouTube channel: http://www.
youtube.com/user/phedhex
4. Schraffenberger, H., & van der Heide, E.
(2013). Towards Novel Relationships between
the Virtual and the Real in Augmented Reality.
In Proceedings ArtsIT 2013 – Lecture Notes of
Let us return to the imaginary walls that were
the Institute of Computer Sciences, Social In-
occupying the university’s dance studio some
formatics and Telecommunications Engineering
weeks ago. I am not sure whether these walls
(pp. 73-80). Springer Berlin Heidelberg.
can be called AR. But I am sure that a dancer
will not be able to create the illusion of a virtual
wall in space without imagining the wall first.
70
71
There is more
than meets
the eye in
Augmented
Reality Game
Environments
by Stephan Lukosch,
Heide Lukosch
& Alexander Verbraeck
1. The Vision: Actors within augmented reality
games
Orson Scott Card impressively shows the effect
of high fidelity in simulation games in his novel
Ender’s Game [1]. The main actor Ender and his
team believe they are playing a training simulation game for fighting a war on an alien race. In
the end, it becomes clear that Ender was in fact
Realistic agents and environment of the CharliePapa game
commanding the real fleet through the game, attacking and finally extinguishing the alien race. In
his saga Otherland [2–5], Tad Williams describes a
which almost every object is networked and the
Science Fiction authors Orson Scott Card, Tad
They will mature into holistic embodied experi-
future world with a widespread availability of full-
use of augmented reality is normal. Humans in-
Williams and Vernor Vinge forecast our vision on
ences, which are seen as a prerequisite for social
immersion virtual reality [6] installations. These
teract within augmented reality [8, 9] by wearing
simulation games and augmented reality in the
cognition, with interaction as a crucial element
installations allow people to access an online
smart clothes and contact lenses that can overlay
future. Several years from now, there will be
[10]. Similar to Ender’s game, augmented reality
world, called simply ‘the Net’. Within the Net a
the physical environment with computer graph-
more than meets the eye in augmented reality
environments will provide a degree of fidelity, or
group of people aims to achieve immortality. In
ics. In Rainbows End [7], augmented reality is
game environments. Instead of only overlaying
realistic representation, which will make it diffi-
his novel Rainbows End [7] Vernor Vinge describes
used for various purposes, e.g., large-scale com-
the physical environment with computer graphics
cult to distinguish the real world from the virtual
how the main character Robert Gu is slowly re-
mercial gaming areas, supporting maintenance
and thereby focusing on human vision, augment-
computer-generated world. The use and the in-
covering from Alzheimer’s disease due to medical
workers with blueprints of machines or buildings,
ed reality game environments will address all hu-
teraction within augmented reality environments
advances in the future. While recovering, former
communication with virtual avatars and diagnos-
man senses, i.e. sound, smell, taste and touch, as
will become as natural as in Vernor Vinge’s novel
technophobe Robert adapts to a changed world in
tic purposes in medical settings.
envisioned in the saga Otherland by Tad Williams.
Rainbows End.
72
73
First steps toward the combined visions and ideas
complex problems from different perspectives [16]
com), the Netherlands Forensic Institute (NFI) in-
et al. [26] show that such virtual avatars are pos-
of Orson Scott Card, Tad Williams and Vernor
and provide changing scenarios. That makes them
vestigated the use of new technologies to train
sible, but are currently fixed to one location due
Vinge have already been taken. There has been
powerful tools for players to gain new insights in
forensic investigators and also help forensic in-
to the necessary technology such as 3D beamers
quite some research on introducing smell into
a given situation, and to build up or adapt their
vestigators in the field. One part of the project
or depth cameras.
movie theaters and television [11] and even more
mental models. The variety of uses for simulation
explored the potential of augmented reality for
research on haptic feedback [12]. One of the
games related to complex problems is enormous,
crime scene investigation. The project particu-
A project targeting the work of reconnaissance
most difficult aspects to reproduce, however, is
varying from teaching about complexity to un-
larly focused on the real-time alignment of physi-
teams called CharliePapa, a joined effort of Delft
a high-fidelity interaction with other (real or vir-
derstanding system behavior or testing relations
cal and virtual environments and interaction
University of Technology, the Dutch Police, a
tual) humans. In 2000, Olson and Olson [13] ana-
and rules between system elements [16, 17]. The
with the environment using augmented reality
private security firm, a training institution and
lysed the extent to which groupware technology
military, for example, has a long tradition of us-
to support collaborative spatial analysis on loca-
a developer of simulation game scenarios, ex-
allows geographically distributed teams to work
ing simulations for strategy and combat training,
tion [25]. By means of augmented reality, remote
plored the effect of highly realistic virtual sce-
together as if they were co-located. They came
because of the opportunity to clearly illustrate
experts were able to connect to investigators on
narios on team awareness and interaction [27].
to the conclusion that distance matters and that
consequences of actions in a safe environment,
the crime scene and guide and help in the in-
Three different scenarios of a highly realistic ur-
the analysed technology is not mature enough to
without risk of injury or other damage [18, 19].
vestigation by providing their
enable virtual co-location yet. Olson and Olson
Ender and his team experience this exact feeling
expertise, but also by adding
state that even future technology will struggle to
of safety, meaning to act without causing serious
virtual objects to the perceived
enable virtual co-location, as providing aware-
consequences when “playing” their game.
reality of the investigator on
ence as well as spatial referencing will remain a
In Ender’s game [1], the simulation game has a
tablishing virtual co-location.
challenge [13]. Gaver [14], on the other hand, un-
high level of realism. Chalmers and Debattista
An evaluation of this approach
derlines the importance of supporting awareness
[20] show that for knowledge transfer, simula-
has shown that the visual feed-
information to help actors shifting from working
tion games should be designed very much like the
back in augmented reality pro-
alone to working together.
‘real’ world. However, abstraction and simplifica-
vides mutual understanding of
tion can also lead to excellent training outcomes
the analysis state. One expert
Considering current groupware technology, this
[21]. Which level of realism or fidelity is necessary
stated: “There is not much
forecast is still mainly correct. Complex problem
to make a simulation game effective has not yet
arguing about something you
solving often requires a team of experts to physi-
been answered [22, 23]. So far, mainly the physical
can both see”. The informa-
cally meet and interact with each other, since
fidelity of simulation games, such as visual, spa-
tion exchange was very visu-
identifying the problem and creating a shared
tial, auditorial and kinesthetic design, has been
ally oriented, e.g.: “behind the
understanding is a prerequisite for efficiently
considered [24]. For a simulation game in an aug-
table”, “don’t cross the ribbon
solving a problem [15]. Typical scenarios are e.g.:
mented reality environment, as played in Ender’s
on the far right”, “if you look
solving complex construction problems, training
game, more dimensions have to be considered.
slightly to the left”. There was
the usage of complex machinery, analysing com-
These can be the psychological fidelity aspects
ambivalent feedback on the
plex situations in emergency services or diagnos-
such as stress or joy [22], functional fidelity such
recording and remote observa-
ing complex medical situations. Unfortunately, it
as learning goals or representation of tasks, and
tion. Some were afraid of being
is not always possible to bring a team together
social fidelity with regard to the ability of players
monitored in their work while
to handle a complex situation. This is due to ex-
to interact with each other or virtual avatars.
others welcomed the possibility
the scene (see Figure 1), es-
ness among co-workers and enabling co-refer-
to capture best practices. The
perts’ availability, critical timing issues or accessibility of a location. While in the novel Rainbows
End [7], such situations are supported with high
fidelity augmented reality technology, current
technology is not yet there.
Figure 1. A staged crime scene augmented with restricted area ribbons
2. The present: approaches
to augmented reality
and simulation games
evaluation further showed that
the presence of the remote expert as well as the awareness of
the remote expert’s activities
at the crime scene needs to be
When considering ‘the Net’ in the saga Otherland
Recent research has shown, however, that vir-
improved, e.g. by using virtual
[2-5], simulation games have the potential to pro-
tual co-location is in fact possible. Within the
avatars as described by Vernor
vide a rich environment and the ability to approach
project CSI The Hague (http://www.csithehague.
Vinge in Rainbows End [7]. Beck
74
Figure 2. A scene from the CharliePapa simulation game
75
ban environment (see Figure 2) were developed
taken. However, this highly realistic set-up was
social fidelity was too low as well. By add-
ficult to design a holistic embodied experience,
and tested with the target group, reconnaissance
not sufficient in really developing an immersive
ing a time limit and competition element,
based on a high level of physical, functional, psy-
teams. Tasks and stress level as well as time pres-
activity like those experienced by the heroes of
the psychological fidelity level was the only
chological and social fidelity with current com-
sure were represented as realistically as possible.
the science fiction novels mentioned earlier. Re-
factor that was high enough, as reported by
mon simulation game technologies.
The reconnaissance teams focused on protecting
alistic interaction with the virtual environment is
the test group, to simulate a realistic ex-
a VIP and detecting deviant behaviour of other
of utmost importance, but recent technology of
perience.
subjects at the site. The assignment of the most
“pure” simulation game environments and agents
mature scenario was to walk through the environ-
does not have the ability to create a high-fidelity
From these experiments we can conclude
ment with three players in a row, each of them
experience with a high level of physical fidelity.
that Olson and Olson’s observation that it
confronted with objects, persons and actions that
Human ability to focus on an object is present in
remains very difficult for current technol-
For Ender, it is a shock to learn that the simulation
were partly the same and partly different from
reality but is hard to resemble in a game, as one
ogy to enable virtual co-location [28], also
he commanded was not a game, but that it was
those of the other players, and to pass over in-
only has a wide-angle view. Furthermore, the in-
holds for a high-fidelity virtual environment
reality, and that he killed a whole race with his
formation gathered from the virtual environment
teraction with people and objects in the game is
within a simulation game. This result can be
actions. Augmented reality game environments
in order to make a decision on the actions to be
still too slow and cumbersome, so functional and
enhanced with the finding that it is very dif-
for training purposes and virtual co-location in
3. The future: augmented
reality simulation games
professional working environments should not
be as shocking as his insight, and will not be as
massive as the “game” Ender was made to play.
On the other hand, combining augmented reality with simulation games and making use of the
advantages of both technologies can provide environments of high fidelity and immersion, leading to high effectiveness of professional teams.
A feeling of “being there” can be achieved, supported by a very realistic training experience.
Augmented reality is already more than a visual
experience; it enables graphical real-time simulation, which can support fast and intuitive understanding of a situation [8, 9]. Still, there are a lot
of current and future issues in simulation games
and augmented reality environments that need to
be addressed in order to make the combined vision of Orson Scott Card, Tad Williams and Vernor
Vinge possible.
With advances in hardware and software capabilities, camera-equipped mobile devices continually gain more interest in augmented reality
games [29, 30]. Recent studies show that such
mobile devices (MD) can foster virtual co-location
of distributed players [28, 29] and enlarge the
game experience by blending virtual and physical
game spaces [31]. Players are no longer bound to
an experience provided only by a computer interface or a physical environment, but can enter
Free hands user interface in augmented reality
76
an enriched physical world, with virtual objects
and agents adding real-time information. The en-
77
tertainment gaming industry is already pav-
References
ing the way. Ingress [32] is an augmented
reality massively multiplayer online game
1. Card, O.S. (1985). Ender’s Game. Tor Books.
created by Niantic Labs, in which players
2. Williams, T. (1996). Otherland - City of
belong to one of two factions (Enlightened
or Resistance) for which they have to conquer territory. Although the physical fidelity of the game with regard to augmented
reality is rather low, several hundred thousand players are engaged within the game.
The future will bring even more immersive
augmented reality games that make use of
Golden Shadow. Legend Books.
3. Williams, T. (1999). Otherland - Mountain of
Black Glass. Legend Books.
4. Williams, T. (1998). Otherland - River of Blue
Fire. Legend Books.
5. Williams, T. (2001). Otherland - Sea of Silver
Light. Legend Books.
6. Milgram, P. and Kishino, F. (1994). A tax-
Journal of Information Technology & Decision Making (IJITDM). 11, 6 (2012), 1–39.
16.Geertje Bekebrede (2010). Experiencing
complexity: a gaming approach for understanding infrastructure systems. Delft
University of Technology.
17. Klabbers, J.H.G. (2006). The magic circle :
26.Beck, S. et al. (2013). Immersive Group-toGroup Telepresence. IEEE Transactions on
Visualization and Computer Graphics. 19, 4
(2013), 616–625.
27.Lukosch, Heide et al. (2014). Building a
Virtual World for Team Work Improvement.
Frontiers in Gaming Simulation. Springer,
principles of gaming & simulation. Sense Pub-
London/Dordrecht/Heidelberg/New York.
lishers, Rotterdam, The Netherlands. 141–166.
60–68.
18.Bonk, C.J. and Dennen, V.P. (2005). Mas-
28.Broll, W. et al. (2006). Meeting Technology
novel head-mounted devices (HMDs), e.g.
onomy of mixed reality visual displays. IEICE
sive Multiplayer Online Gaming: A Research
Challenges of Pervasive Augmented Reality
Google Glass [33], Oculus Rift [34] or Meta
Transactions on Information Systems. E77-D,
Framework for Military Training and Educa-
Games. Proceedings of 5th ACM SIGCOMM
Spaceglasses [35], and combine the visions
12 (1994).
tion. Technical Report #TECH-RPT-2005-1.
Workshop on Network and system support
of Orson Scott Card, Tad Williams and Vernor Vinge.
7. Vinge, Vernor (2006). Rainbows End. Tor
Books.
8. Azuma, R. et al. (2001). Recent advances in
In order to realize this vision for both entertainment and more serious purposes such
as problem solving and training, augmented
reality games first have to become more
than meets the eye and address all human
senses, i.e. sound, smell, taste and touch,
Indiana University, Bloomington.
19. M. Macedonia (2002). Games Soldiers Play.
IEEE Spectrum. 39, 3 (2002), 32–37.
for games, Netgames’06 (2006), Article 28.
29.Flintham, M. et al. (2003). Where on-line
meets on the streets: experiences with
augmented reality. Computer Graphics and
20.Chalmers, A. and Debattista, K. (2009).
mobile mixed reality games. Proceedings of
Applications, IEEE. 21, 6 (Dec. 2001), 34 –47.
Level of Realism for Serious Games.
the SIGCHI conference on Human factors in
9. Azuma, R.T. (1997). A Survey of Augmented
Reality. Presence: Teleoperators and Virtual
Environments 6 (1997), 355–385.
10.Hanne De Jaegher and Ezequiel Di Paolo
Proceedings of the Conference in Games
and Virtual Worlds for Serious Applications
(2009), 225–232.
21.Toups, Z.O. et al. (2011). The team coordina-
computing systems (2003), 569–576.
30.Rohs, M. (2007). Marker-based embodied
interaction for handheld augmented reality
games. Journal of Virtual Reality and Broadcasting. 4, 5 (2007), 1860–2037.
as envisioned by Tad Williams. Secondly, us-
(2007). Participatory Sense-Making An Enac-
tion game: Zero-fidelity simulation abstract-
ers will have to be aware of each other’s
tive Approach to Social Cognition. Phenom-
ed from fire emergency response practice.
activities by using an augmented reality
enology and the Cognitive Sciences. 6, 4
ACM Transactions on Computer-Human
games: bringing computer entertainment
Interaction. 18, 4 (Dec. 2011), 23:1–23:37.
back to the real world. Computers in Enter-
environment for spatial remote collaboration, as described by Vernor Vinge. The fi-
(2007), 485–507.
11. Kim, H. et al. (2011). An X-Y Addressable
22.Alexander, A.L. et al. (2005). From Gaming
nal and most difficult challenge seems to
Matrix Odor-Releasing System Using an On-
to Training: A Review of Studies on Fidelity,
be to create a realistic interaction between
Off Switchable Device. Angewandte Chemie.
Immersion, Presence, and Buy-in and Their
real objects and virtual objects [36]. This
123, 30 (2011), 6903–6907.
Effects on Transfer in PC-Based Simulations
will enhance the fidelity of the game play
12.Samur, E. (2012). State of the Art. Per-
similar to Ender’s experiences as described
formance Metrics for Haptic Interfaces.
by Orson Scott Card. When these three chal-
Springer London. 9–26.
lenges are addressed, a realistic multi-mod-
13.Olson, G.M. and Olson, J.S. (2000). Distance
al, multi-user game experience can be cre-
matters. Human-Computer Interaction. 15, 2
ated that is in many ways indistinguishable
(2000), 139–178.
and Games. Aptima, Inc., Woburn, MA.
23.Harteveld, C. (2011). Triadic Game Design: Balancing Reality, Meaning and Play.
Springer, Berlin.
24.Feinstein, A.H. and Cannon, H.M. (2001).
Fidelity, Verifiability, and Validity of Simula-
31.Magerkurth, C. et al. (2005). Pervasive
tainment (CIE). 3, 3 (2005), Article 4A.
32.https://www.ingress.com/, last accessed
07/03/14
33.http://www.google.com/glass, last accessed
07/03/14
34.http://www.oculusvr.com/, last accessed
07/03/14
35.https://www.spaceglasses.com/, last accessed 07/03/14
36.Schraffenberger, H., & Van der Heide, E.
tion: Constructs for Evaluation. Technical
(2013). From coexistence to interaction:
a whole new generation of applications that
Collaboration. Proceedings of the Second
Report #2001-006. Wayne State University
influences between the virtual and the real
will benefit from distributed people inter-
Conference on European Conference on
Marketing.
in augmented reality. In Proceedings of the
acting naturally with each other and with
Computer-Supported Cooperative Work
their synthetic environment.
(1991), 293–308.
from reality in most senses. This will enable
78
14.Gaver, William W. (1991). Sound Support for
25.Poelman, R. et al. (2012). As if Being There:
Mediated Reality for Crime Scene Investiga-
15.Piirainen, K. et al. (2012). The Joint Struggle
tion. CSCW ’12: Proceedings of the 2012
of Complex Engineering: A Study of the Chal-
ACM conference on Computer Supported
lenges of Collaborative Design. International
Cooperative Work (2012), 1267–1276.
19th International Symposium of Electronic
Art, ISEA2013, Sydney.
79
The Coronation Hall in the historic town hall of Aachen, Germany
Corona:
Audio AR for historic sites
by Florian Heller
80
Many historic sites, as important they may have
Hall were the coronation feasts of the 15th and
been, often lack visual cues to events that took
16th century. However, the only visual remainder
place there. Sites become ruins or are changed
is a set of coats of arms engraved in the
intentionally through structural modification.
pavement. The Route Charlemagne1 (www.route-
The city hall of Aachen, Germany is an example
charlemagne.eu) project was initiated to make
of such a site. Built in the 9th century, it was
several historic buildings in Aachen accessible to
the seat of Charlemagne’s and several other
the public. The main focus was on the city hall,
emperors’ governments, and it is still in use
since it was only possible to be visited as part of
as city hall today. Among the most important
a guided tour. To allow a self-paced exploration,
ceremonies that took place in its Coronation
we created a series of interactive exhibits and
81
the Aixplorer audio guide, which automatically
gorging on meat and drinking from the fountain
The overall result of the rendering is not per-
detects the room you are in. For the Coronation
of wine. The nobility had to follow protocol. The
fectly realistic, but plausible and results in an en-
Hall, we wanted to create a landmark exhibit
ceremony of the coronation of Charles V from
tertaining experience. In fact, a realistic simula-
that reminds visitors of the important ceremonies
the 16th century is very well documented, which
tion of the room’s acoustics would lead to a poor
that happened there. As the hall is regularly used
gave us the opportunity to bring some of the at-
understanding of the dialogs as the hall’s natural
for public activities and is under monumental
tendees back to life.
reverb makes it impossible to communicate over
a distance larger than eight meters.
protection, fixed installations were not an option.
Since the visual impression of the room had to
In the Corona audio space, virtual characters are
stay untouched, we decided to create an audio-
placed at their historically handed down positions.
only exhibit.
In groups of two they discuss different aspects of
Individual audio spaces
the ceremony. For example, the newly crowned
The Corona audio space
Corona is an audio augmented reality experience
King discusses matters of the Black Death with
As with traditional audio guides, the audio space
the Archbishop, servants describe the order of the
is individual to the listener. Usually, this is used to
dishes, and two persons standing by the window
provide the content in different languages or for
observe the festivities on the market place.
different audiences, e.g., a simplified version for
children. With our continuous audio space, howev-
that overlays the physical space with a virtual audio space, generating an impression of being right
in the middle of a coronation feast. You would
probably imagine such a banquet as a lavish cele­
The technology behind the
scenes
er, we can create a more playful approach. Moving
sound sources could lure people to areas that they
have not yet explored or create an audio scavenger hunt. The continuous audio space also has the
bration, but that was only true for the populace
To create the illusion that the audio sources are
advantage that features which need manual inter-
located at fixed positions in the physical space,
vention or additional implementation effort with
we need information about the user’s position
traditional audio guides, such as synchronized
and head orientation. The Coronation Hall is 45
playback, can easily be achieved. Depending on
by 20 meters large with four stone pillars in the
the path that a user takes, the different sources
only actively listening to one of these. This means
middle of the room, which makes location track-
start and stop playing at different points in time.
that key information has to be presented repeat-
ing difficult if it is supposed to be invisible. Fur-
Thus, to experience the audio space as a group,
edly, but using short audio fragments bears the
thermore, optical tracking using video cameras
you just have to walk side by side, without having
risk that the visitor notices the loop. In our con-
has to meet strict privacy regulations. We opted
to make sure that you press play at the same time.
tent, key information is repeated several times in
different words, allowing a late drop-in.
for a Ubisense wireless radio location tracking
system that performs with an accuracy of 10-
A visitor explained his impression as follows:
50 cm and delivers updates 4 times per second.
“Corona is an emotional experience, like a
The orientation is measured by a digital compass
film. This is much more interesting and thus
chip, with an update rate of 10 Hz, mounted to
memorable than a normal audio guide”. The act
the headphones. This information is then fed into
of discovering hidden information was clearly
the OpenAL spatial audio rendering engine avail-
favored over learning plain facts as with tradi-
The human brain is quite good at making sense out
able on iOS. Since the results of this engine do
tional audio guides.
of sensory information that does not fit exactly.
not provide a good separation of sources that are
directly in front or behind you (a common problem with spatial audio rendering called front-
82
Simplifying the
implementation
For example, if a virtual sound should emerge from
Challenging the authors
back confusion), we added a low-pass filter to the
Corona use concept
Spatial layout of auditory sources in
the Coronation Hall
a specific physical artifact, but the perceived location of the virtual sound source and the location of
the artifact slightly differ, the virtual source seems
signal coming from a source behind the listener.
The challenge in writing the dialogs of the audio
to snap to that physical location (the so called ven-
To avoid an auditory overload resulting in poor
space was mainly the fact that we did not know
triloquist effect [1]). A visitor experienced a simi-
localization, we pause sources that are more than
when the visitor would actively be listening to
lar effect when the network connection used to
11 meters away.
the source. In contrast to classic audio guides,
transmit the location data broke down. Since the
several sources might be playing but the visitor is
rendering engine could only use the head orienta-
83
Acknowledgements
tion, the distance to the sources was not updated
device orientation does not dramatically affect
anymore. She interpreted this a bit different and
the perceived presence in the virtual environ-
told us: “That was amazing! After some time, the
ment. The localization accuracy does not neces-
voices started walking with me!”
sarily need to be as high as in our implementa-
Foundation and the state of Northrine West­
tion. A series of outdoor installations, a sound
phalia through its EU-ERDF program ``Ziel 2’’.
Based on these effects, and the observation that
garden in a municipal park [3] for example, have
many users of our system do not turn their head
used GPS to get location information and this was
to orient themselves in the audio space, but in-
not perceived as a problem.
This work was financed by the German B-IT
References
stead turn their entire body, we investigated
1. Alais, D., & Burr, D. (2004). The ventriloquist
whether the implementation of a system like Co-
effect results from near-optimal bimodal
rona could be simplified. The hardware require-
Outlook
ments for this exhibit make it a complex instal-
integration. Current biology, 14(3), 257-262.
2. Heller, F., Krämer, A., & Borchers, J. (2014,
lation; every headphone needs to be equipped
Audio augmented reality applications are engag-
with a compass, which has to be interfaced with
ing experiences that go beyond the plain presen-
for mobile audio augmented reality applica-
the smartphone. Similarly, the location informa-
tation of historic facts. In the context of a mu-
tions. In Proceedings of the 32nd annual ACM
tion has to be communicated to the audio guides
seum it can be used to create an atmosphere the
conference on Human factors in computing
as well. Since current smartphones are equipped
user dives into and that stimulates the visitor’s
with location and orientation sensors, the re-
fantasy. Current smartphones provide enough
3. Vazquez-Alvarez, Y., Oakley, I., & Brewster,
quired hardware is basically already available.
processing power to handle the spatial audio ren-
S. A. (2012). Auditory display design for ex-
Using these built-in sensors does not allow the
dering and if realism is not the top priority, their
ploration in mobile audio-augmented reality.
same degree of realism, since they only measure
built-in sensors provide all required information.
Personal and Ubiquitous Computing, 16(8),
the orientation of the device - not the head, but
So there aren’t any excuses not to have an audio
987-999.
our current experiments [2] indicate that using
AR app for your museum.
April). Simplifying orientation measurement
systems (pp. 615-624). ACM.
Corona in use
Florian Heller
Florian Heller is a PhD Student in the
Media Computing Group at RWTH
Aachen University in Germany. From
2009 to 2012 he was a fellow of the B-IT
Research School. He worked on the implementation of Corona as his Diploma
Thesis and continued his research in the
field of physical interaction with audio.
The physical component can either be
movement like in Corona, or interaction
with tangible interfaces such as an augmented DJ turntable or with wearable
interfaces.
 http://hci.rwth-aachen.de/heller
 http://hci.rwth-aachen.de/corona
84
85
Developing augmented reality
applications:
a future perspective
by Robert Prevel
this, GG is very promising in terms of hard-
our environment must also expand. We will
ware development for AR. Most existing head
need to be able to enter an entirely unknown
mounted displays (HMDs) capable of delivering
environment and learn the boundaries and
an AR experience are far too heavy and bulky
the kinds of objects therein. Software that is
for extended use. In this regard, GG gives us a
capable of mapping the environment in real-
glimpse of what we can expect from a light-
time is already being developed, an example
weight, compact HMD in the future.
of this is Google’s ‘Project Tango’. Knowl-
In addition to issues of comfort, the HMDs
edge of our working environment is key to
of the future will need to solve a number of
providing a believable interactive AR experi-
existing problems. Firstly, whilst the technol-
ence. Lastly, we will want multiple users to
ogy behind current HMDs varies, when used
share experiences in the same environment,
for AR applications they all provide a dimin-
so that the actions of one user are observed
Augmented reality (AR) applications, once
the public domain is to be encouraged, then
ished view of the environment; a reduced
by others. This will require an advanced level
seen as a future technology, have in recent
our future hardware and software solutions
field of view or resolution for example. Sec-
of data sharing and communication between
years become far more prevalent in the
must be affordable and readily available.
ondly, there are safety concerns with display-
devices.
public domain. This is in large part due to
The quality of an AR experience depends on
ing too much or ill-timed information directly
the availability of affordable hardware that
both the hardware and software used to de-
in front of the user, particularly if that user
We have given many demonstrations of our
can be used to facilitate many basic AR ap-
liver that experience. For example, a heads
is performing an attention demanding task at
work at the AR lab, and people’s experiences
plications, the most common example being
up display provides a far more immersive AR
the time, such as driving (though it could be
are usually similar. Some come to us with no
the smartphone. These lightweight compact
experience than a hand held display. Unfor-
argued that this is less of a safety issue and
prior experience of AR, others with some ex-
devices typically consist of a camera, on
tunately, unlike smartphones, AR headsets
more a demonstration of the user’s lack of
periences through a smartphone or tablet.
board processing, a screen, and other use-
are not readily available. In addition, even
common sense). Lastly, the public accept-
There is something special about putting on
ful functions such as wireless capability. As
the cheapest self-assembled headsets are
ability of using HMDs in public places must
a HMD; whether it’s the highly immersive 3D
smartphones and in particular smartphone
more costly than an average smartphone.
improve. We predict that once these issues
AR experience, or the novelty of wearing fu-
applications have become more widespread
As the hardware used in AR applications
have been addressed, it will be the norm to
turistic looking headgear. We expect that this
in society, there has been a notable increase
develops, so too will the software allowing
wear lightweight compact HMDs in public. In-
novelty will wear off as people become more
in the number of AR applications and devel-
the best experience from the available hard-
deed, in the future our children will laugh at
used to wearing HMDs in everyday life. Our
opment companies.
ware. AR application development is a cycle
the fact that we used to use our hands for
augmented reality applications will be pro-
where new applications require improved
such simple things as holding devices.
viding on the spot instructions for cooking,
furniture assembly, and car maintenance, as
This growth of augmented reality in the pub-
hardware, which allows for new software
lic domain is very positive. However, most
ideas that in turn require new hardware. We
In order to leave the desktop behind and ex-
well as providing us with directions to the
existing commercial AR applications are con-
predict a number of ways in which both hard-
plore the wider augmented world, a number
restaurant. In the future, AR HMDs will have
fined to small working environments, and
ware and software for AR applications will
of software challenges must be met. The
become just as common as smartphones are
provide a very basic AR experience; they
develop in order to improve on our current
key to successful tracking in AR is to identify
now. We expect that the development of ad-
allow you to play virtual ‘desktop’ games,
AR experiences.
stationary landmarks, be they the corners
vanced, interactive AR applications will be
of a fiducial marker or the natural features
driven by the gaming industry initially. But
or overlay a simple virtual object or image
on top of a real world object. Many of these
In the future, hardware for AR will be far
present in the environment; this becomes far
the potential for AR is limitless, not forget-
‘desktop’ applications have very little knowl-
lighter and more comfortable, allowing for
more difficult when working in dynamic en-
ting that our experience is not limited to vi-
edge of the working environment, and use
extended use. Devices capable of providing
vironments. If we are truly to explore larger
sion and audio alone. Certainly the most ex-
what we would already consider ‘old technol-
a heads up display have already started to
environments, we will undoubtedly observe
citing applications for AR are in the medical
ogy’ for tracking; namely a fiducial marker or
infiltrate the public domain, the most publi-
moving objects. Calculating which landmarks
domain. Imagine your surgeon standing over
some equivalent. Future AR applications will
cised being Google Glass (GG). At present GG
belong to stationary objects in a dynamic en-
you, with their AR HMD displaying where to
most certainly require more detailed knowl-
is rather expensive and not a suitable display
vironment is a must. Changes in environmen-
make the incision for your neural interface.
edge of much larger working environments
for most existing AR applications (unless you’re
tal lighting will also become more frequent,
‘Yes sir, and if you put on your bedside HMD
and more robust tracking solutions. If the
happy playing your desktop games in a small
requiring more robust tracking algorithms. In
you’ll be able to see what your bionic arm
continued expansion of AR applications into
box in the top right of your vision). Despite
addition to tracking, the process of mapping
will look like’.
86
87
often completely unrelated to art, and the work of
the influential plastic artist Piet Mondrian, to present the digital technology not as an object, but as
a medium for art.
by Dirk Schart, Diego Montoya & Melo Montoya
The idea behind HyperMondrian
Piet Mondrian, Composition with Lines (1917)
The Montoya brothers chose a game engine in
order to animate and present three-dimensional
State of the Art –
A future perspective
data and image tracking as a way to connect the
Compositions in 3D
physical works and the digital imagery. Using
this technique, they developed an app in col-
The first one, Composition with Large Red Plane,
laboration with the Augmented Reality company
Yellow, Black, Gray and Blue, serves to extend
The new advances in technology enable explora-
RE'FLEKT, that shows new features on top of the
Mondrian's theory. The elements found in the
tion of all kinds of possibilities. Works found in
existing paintings when seen through the lens of
original become three dimensional and mobile,
galleries don’t have to be static. Paintings and
an Android or iOS device.
with time intervals and range of movements depending on mathematical relations. To respect
sculptures can move and come to life through
these advances. The digital artistic works are no
HMondrian stands for Hyper-Mondrian in refer-
the original harmony of all the elements, the
longer contemplative, but rather participative:
ence to the hyper-plasticism, a proposed exten-
augmented version of the painting reconstructs
actions by the observers nearby can be digitally
sion of the neoplasticism. Through the use of the
the original work when seen from the correct
measured and the work can react accordingly.
new technologies such as 3D tracking, gesture rec-
perspective, even if the pieces are in permanent
ognition and generative algorithms,neoplasticism
movement.
The younger generations – especially the digital
acquires new properties; exploring new dimen-
natives – like to discover through interaction, and
sions and allowing the spectator to participate,
The second painting is Victory Boogie Woogie. Re-
Plane, Yellow, Black, Gray and Blue (1921)
they know their way around technology. On the
and thus acquiring the prefix "hyper".
grettably, the painter died before finishing it. Be-
Piet Mondrian contributed greatly to the De Stijl
to interact with the digital world in a more natural
The first step was deciding what the main ele-
can't hold itself together, so it crumbles down in
movement, also known as neoplasticism, and
way. Also, the capabilities of algorithms running in
ments of the app should be and what it would
pieces and reforms as the user moves closer to or
found beauty in the abstraction of simple, har-
powerful micro-processors allow for real-time art
be ableto do, taking advantage of the capabili-
further away from the augmented painting in a
monious forms. But what if he would have been
production on devices that fit in our pockets.
ties of the 3D visualization and image tracking.
juxtaposition of time and space.
Piet Mondrian, Composition with Large Red
cause of that, the augmented version of this work
other hand, the older generations have found ways
Mondrian was a prominent person, and the idea
free from the constraints of traditional two-dimensional painting, being able to use the form
With all this in mind, Diego Montoya, an Augment-
was to explore his art, his life and his creative
For Mondrian, the working environment influ-
in all directions, including time? Would he have
ed Reality developer, and his brother and multi-
environment. After experimenting with various
enced the creative process greatly and therefore
rather stayed in the flatness of the canvas, or
media artist Melo Montoya intervened in some of
paintings and reviewing the story behind them,
he arranged it accordingly. When seen through
would he have pursued his search for beauty in
Piet Mondrian's works. One of their goals was to
three of his most symbolic works were chosen to
the mobile device, the augmented version of the
all dimensions? What if he could have given the
reach a wider range of target groups and fasci-
do this: Composition with Large Red Plane (1921),
third work Composition with Lines reveals a stu-
spectator a role, giving him a part to play in the
nate them for art. They also wanted to establish
Victory Boogie Woogie (1942-1944) and Composi-
dio inspired by those were Mondrian used to work,
harmony of his works?
a bridge between the latest digital developments,
tion with Lines (1917).
showing decorations that recreate his paintings.
88
89
The presentation
Diego Montoya
This intervention of Mondrian’s work was pre-
Having obtained degrees in electronics
sented in the Monitor Digital (MOD) - one of Latin
and information systems, he works on developing aug-
America’s most important digital art festivals -
mented reality and natural user interface applications
in Guadalajara Mexico in 2013. There were some
and lives in Munich. He is an enthusiast of interactiv-
iPads available to the public to let them use the
ity, generative design and the theories of the universe
app and visitors were able to scan a QR code to
and an admirer of the modern and post-modern art
download the app on site.
movements. Diego enjoys creating digital art works
using state-of-the-art technologies to blur the line
HMondrian was well received by the public. The
between the art and the digital, the observer and the
people visiting the festival were very diverse
object, the real and the virtual.
and they had all kinds of reactions, ranging from
young children exploring the 3D work as if it was
a real object, to the old lady who screamed “Oh!
Piet Mondrian, Victory Boogie-Woogie (1942–44)
I broke the painting!” and who then dropped the
iPad after seeing the virtual Victory Boogie Woogie crumble down.
The implementation
Melo Montoya
Armed with the experiences they gathered while
Melo studied Multimedia at CAAV. He is a pro-
developing and presenting this work, the au-
grammer, open format DJ and loves the sun.. He
Presenting 3D imagery in a mobile device re-
thors are currently working on other hyperplastic
has presented works along with collaborators in
quires choosing the right tools to do it, and in
works that involve the tracking of sculptures and
the main Digital and Contemporary Art museums
the case of HMondrian, the 3D scenes were devel-
using the spectator as a living canvas.
in Guadalajara Mexico such as MOD, MAZ and
oped using Unity3D, a game engine primarily used
MURA The internet loves him and is the source of
to develop mobile games. In order to locate and
his collection of quirky content. He now lives in
position the elements added to the paintings an
How does it work
“La perla tapatía” but he wishes to return to the
Caribbean after doing a Master’s in Europe.
image tracking framework, Vuforia, was chosen
from a handful of options because of the charac-
Scan the QR code and download the app “HMon-
teristics of the project. This framework is espe-
drian” from the App Store or Google Play Store.
cially good at tracking simple geometric figures,
Once you open it there are two modes available
like those found in Mondrian’s work. The model-
– the Augmented Reality and the Virtual Reality
ing of the 3D objects as well as the animations
mode. Choose the Augmented Reality mode to
were done in Maya.
enjoy the extended versions of Mondrian’s works.
 www.melomontoya.com
Dirk Schart
Point the camera of your device towards one of
For this work, it was important that the tracking
the paintings featured in this article. You can also
Dirk Schart is a communication and PR expert,
was stable and fast to swiftly grab attention and
download them from the app itself. Just tap on
blogger and author with a strong passion for digital
involve the user. Illumination of the room where
the info button and scroll down
communication and interactive media. He holds a
the paintings were displayed had to be carefully
to find the download button. The
Master degree in communication science and was
set up. Another factor that could affect the im-
Virtual Reality mode just serves
in 2013 awarded for his communication concept
age tracking was the size of the images, but the
to show the 3D-functionality
"connect & climb". Dirk is Corporate Communica-
artists verified that the sizes of the original works
in case you are not around the
tions Manager and Spokesperson at Munich-based
were adequate and gave good results.
paintings or a copy of them.
Augmented Reality company RE'FLEKT.
90
91
The so-called
Augmented Reality
by Pawel Pokutycki
Augmented Reality is probably one of the most
ery technological innovation is about some kind
pretentious, exaggerated and yet philosophi-
of augmentation of our reality, no matter if it’s
cally challenging terms in the field of computer
an automobile, a vacuum cleaner or a light bulb.
technology. Next to concepts of Virtual Real-
Technology enriches our life (arguably of course,
ity, popularized by a spiritually inclined music
as it’s not always for better). Take, for instance,
composer and computer scientist Jaron Lanier,
religion. Isn’t it all about the augmentation of life
or Mediated Reality, formulated by a wearable
(and death)? Or science in general? Medicine? Lit-
computing inventor and “the world’s first cy-
erature? These disciplines offer a better under-
borg” Steve Mann, Augmented Reality stands for
standing of the world we live in and a potentially
a peculiar understanding of ‘reality’ defined not
better, augmented life as a result of understand-
by the acknowledged writings of Aristotle, Plato
ing it and doing something with that knowledge.
or Wittgenstein, but by the pioneers of digital
For example, we would never think of a pacemak-
revolution, or rather, geeks of our times. Just
er as an Augmented Reality application, yet it un-
google AR to see it’s not about Leibniz’s “pos-
doubtedly changes our reality for the better if we
sible worlds” or Kant’s “transcendental ideal-
suffer from an inadequate heart rate. So does the
ism”, but “computer-generated sensory input”
Bible or the Quran, if we are religious. Augmenta-
and “pattern recognition”. Computer-oriented
tion is a predominant desire in human life, either
terminology is dominating the discourse on Aug-
in terms of spiritual enlightenment, or purely
mented Reality and therefore is limiting its ideo-
pragmatic and rational development enhancing
logical significance, going otherwise far beyond
the quality of survival. The so-called Augmented
technology itself. But shouldn’t we - in times
Reality, as defined by its “pioneers”, computer
of Google Glass – not only discuss what kind of
scientists and hardware and software develop-
image processing software or head-mounted
ers today, is therefore only another step in the
displays are used in AR technology, but also, if
evolution of augmentations of all kind, claiming
not primarily, what does it mean to society and
rights to a phenomenon, which is much broader
culture to be in Augmented Reality, to live and
and universal, known already long time before
think in it permanently in the future? And if so,
the digital revolution. how to reflect on it from the perspective of humanities? Koert van Mensvoort, in his essay “Real Nature is
not Green” [1], draws a parallel between nature
‘Augmented Reality technology’ might be con-
and culture, discussing processes in which nature
sidered a pleonasm. In other words, almost ev-
becomes culture, and culture becomes nature.
92
93
In the case of Augmented Reality, we deal with a
to Augmented Reality”, in order to finally upgrade
similar problem of classification and interpreta-
the meaning of this term in the public debate.
tion. This leads to questions such as: what kind
Otherwise, we may end up doing things irrelevant
of reality are we augmenting? What do we mean
to what really matters in the long term; only in-
by reality, how do we define it? Is augmentation
venting AR gadgets that become culturally obso-
making reality more real or less real? Is it culture
lete just a day later and therefore falling in the
or nature? trap of “future thinking”, which is neither about
future, nor about thinking. These are, of course, difficult questions to answer. I claim, however, that in the discourse on
Augmented Reality there is a tendency to over-
References
look the broader picture and to ignore the historical, ideological and philosophical foundations of
 (p. 93-94) Maarten Broekhuizen
“I believe the way we draw the boundary between nature and culture will change. The domain of origin, of
‘birth’ previously belonged to nature, while culture encompassed the domain of the ‘made’. Thanks to developments of science and technology, this distinction is
blurring. (...) Culture is that which we control. Nature
is all those things that have an autonomous quality and
fall outside the scope of human power. In this new classification, greenhouse tomatoes belong to the cultural
category, whereas computer viruses and the traffic-jams
on our roads can be considered as natural phenomena.”
Koert van Mensvoort
94
1. Van Mensvoort K. (2006). Real nature is not
this trending topic. It is perhaps time for some-
green. Article on NextNature.com, Retrieved
body like Slavoj Žižek (or another popular con-
from http://www.nextnature.net/2006/11/
temporary thinker) to make “The Pervert’s Guide
real-nature-isnt-green/
Pawel Pokutycki
Pawel Pokutycki is an interaction designer, thinker and lecturer at the Royal
Academy of Art (KABK) in The Hague,
where in 2005 he initiated early activities of the RFID Lab, in collaboration
with Yolande Kolstee and Wim van Eck
later becoming the AR+RFID Lab.
His recent artistic and educational
projects focus on exploring relationships
between new media theory and political, social or cultural studies by concept
development and prototyping for interactive media. He believes in a methodology of design based on his own, peculiar
interpretation of the Black Box Theory,
presented at a TEDx event in 2012.
 http://tedxtalks.ted.com/video/
Exploring-the-Black-Box-Pawel-P
 Simone Engelen
95
The locations and ranges of the sounds of one of the Wormhole Dordrecht sound worlds.
Image courtesy Studio Edwin van der Heide. Map data: Google, Aerodata International Surveys, Tele Atlas.
Radioscape, Wormhole Dordrecht and Beyond
dioscape: a version using a mobile handheld de-
perception, or the perceptual creation, of space.
vice containing a small computer that could play
Radioscape has a true immersive quality that I had
and mix audio files and would be equipped with
also expected as a result of the Wormhole Dor-
On the Sensation of Presence
a GPS sensor to make it possible to specify and
drecht software. Any movement of the Radioscape
control which sounds would play (and how loud) in
receiver, small or big, leads to a very direct change
by Edwin van der Heide
relation to the position of the device. In that year
in the received signals. Different movements and
Apple opened up iPhone software development
positions of the receiver result in equally different
to third party developers and I realized that the
changes in the balance between, and the spatial
In the AR[t] 2 magazine I wrote about my work
through the environment you develop a perceptual
iPhone could be used as the mobile device I was
representation of the received signals. Further-
Radioscape and proposed that, although it was
understanding (and sensation of presence) of the
looking for. I proposed the above idea as a gen-
more, the repetition of a movement leads to the
not meant to be an augmented reality artwork, it
signals in the space around your body. Radioscape
eral format and 'platform' for the upcoming exhibi-
repetition of that sonic result. There are many
makes a lot of sense to approach it from an aug-
is special in that the transmitted compositions mix
tion at the CBK Dordrecht. For the exhibition, ten
sonic combinations and changes we can explore
mented reality perspective [1]. Radioscape cre-
in space. Each different position (of the receiver)
alumni were asked to each make a location based
without walking and by just moving the receiver
ates a parallel reality intersecting the streets of
in space corresponds with a different unique mix
sound world for the city center of Dordrecht. The
in space. By doing so we navigate through a very
an urban environment. It uses 15 radio transmit-
of the signals.
exhibition received the title Wormhole Dordrecht
detailed and diverse space around us; a space that
because of the possibility to switch (and 'travel')
we experience as truly present. Although we can't
between the 10 different parallel sound words.
fully understand it, it appears to be a coherent and
ters distributed over the chosen city area, each
transmitting its own specific composition. Togeth-
In 2008, Joost Rekveld and I were asked to orga-
er these compositions form the meta composition
nize an exhibition with alumni of the ArtScience
of the work. The audience gets a custom handheld
Interfaculty in the Hague for the CBK (Center for
While the exhibition led to ten interesting and very
gets a tangible character because of the proprio-
receiver to explore and navigate through the work
Visual Art) in Dordrecht. For a number of years,
different artworks, it brought me unexpected new
ceptive and visual perception resulting from mov-
(and the city). By moving the receiver and walking
I had been considering a software version of Ra-
insights; insights related to Radioscape and the
ing the receiver.
96
consistent space. It forms an intangible space that
97
Another quality present in Radioscape is the in-
interesting. For example, the quality of the works
teraction between the radio waves and the physi-
was a result from the musical or narrative quali-
cal environment. Because of the relatively long
ties used within the work.
wavelength (170 meters) buildings become possible resonators for the transmitted signals. Hold-
We are currently experimenting with additional
ing the receiver close to the facade of a build-
ways of creating the sensation of sounds being
ing often leads to unexpected amplification of
present in the space around us and relating or
certain signals, resulting in a direct interaction
interacting with the physical environment around
between the physical environment and the elec-
us. Mobile devices have developed greatly and
tromagnetic waves. The normally unperceivable
often incorporate gyro sensors besides the ac-
electromagnetic waves interact directly with the
celerometers, compass sensor and improved GPS
perceivable space around us. Lastly, Radioscape
sensing. We have experimented with head track-
uses what I call a stereo panoramic receiver. It is
ing and positioning sounds in the space around
a receiver with a stereo antenna similar to the
us with real-time binaural convolution. The bin-
principle of a stereo microphone. The position
aural convolution allows us to simulate the spa-
of the transmitter in relation to the receiver an-
tial position of a sound in any direction including
tenna determines the position of the transmitted
height. While striving towards perfection in the
sound within the stereo image on the receiver's
simulation, it turned out that that the simulation
headphone.
can, and never will be, perfect simply because
the earth's magnetic field is often distorted and
Until Wormhole Dordrecht was realized, I was
influenced by factors within the local environ-
only half aware that Radioscape had such a self-
ment. It became clear that creating a convinc-
explaining spatial quality, such a sensation of
ing spatial environment is not about the (perfect)
presence and why. It was because of Wormhole
simulation of reality but about creating interest-
Dordrecht that I realized that the direct changes
ing relationships between the sounds, the space
within the 'empty' space around our body and the
around our body and our physical environment.
interaction with our surrounding physical space
This is exactly what programmed relationships in
were fundamental for the near tangible sensation
software enable us to do. We could have already
of the electro magnetic space around us. The use
learned this from Radioscape. Radioscape has a
of GPS positioning in Wormhole Dordrecht leads
natural complexity that has its own logic; a logic
to changes that are taking place while walking.
that is so convincing that we easily mistake it for
Standing still and moving the iPhone around
being present or real although we could just as
the body has no effect on which sound files are
well argue that it's all virtual. Interrelating the
played or on the balance between the sound
virtual and the real makes the virtual only more
files. The changes that occur while walking can
real (present) without having to copy or simulate
be technically perfect but don't necessarily lead
reality.
to a perception of presence in space. At the time
the GPS sensing had a precision of approximately
10 meters, but even if the calculated GPS coor-
References
dinates would have had a precision of about a
meter, there still would not be any sense of the
1.
van der Heide, E. (2012, November).
sounds being present in space. It is important to
Radioscape – in the context of Augmented
note, however, that the absence of the sensation
Reality. AR[t], issue 2, 18-23.
of spatial presence did not make the works less
Radioscape as part of the 'Urban Explorers Festival'. Image courtesy Studio Edwin van der Heide.
98
99