Chapter 8
Immersive Art in Augmented Reality
Todd Margolis
8.1 Introduction
This chapter addresses the emerging trend favoring socially immersive artworks
via mobile Augmented Reality over sensorially immersive AR that was based on
historical Virtual Reality ideals. It will first define the various types of immersion
being discussed and make distinctions in how the same term has taken on a
new connotation with modern information technology. Then we’ll explore socially
immersive artwork by looking at both the empowering head-mounted technologies
as well as some early examples using Google Glass. This will lead to a discussion on
collaborative locative media by investigating enabling toolkits and tracing its roots
back from Land Art of the 60s through early community-created gaming platforms
to massively multiplayer online games and ultimately to the current phenomenon of
collaborative “Field Art” taking place in the AR game Ingress. Then we’ll conclude
with a better understanding on how artists are exploring mobile Augmented Reality
within an internet native culture focused on connecting people with each other
through information sharing.
8.2 Definitions/Distinctions
8.2.1 Sensorial Immersion
The goal of immersion within Virtual Reality has been to surround participants
with visual (and sometimes auditory, tactile or other) stimuli to evoke a sense
T. Margolis (!)
California Institute of Telecommunications and Information Technology (Calit2),
University of California, San Diego, CA, USA
e-mail: [email protected]; http://www.toddmargolis.net
V. Geroimenko (ed.), Augmented Reality Art: From an Emerging Technology
149
to a Novel Creative Medium, Springer Series on Cultural Computing,
DOI 10.1007/978-3-319-06203-7__8, © Springer International Publishing Switzerland 2014
[email protected]
150
T. Margolis
of displacement whereby the physical world is supplanted by an alternate reality.
Sensorial immersion can be applied to Augmented Reality in a manner that
bases this reality on the nearby physical world, but adds virtual objects into that
world. Technically speaking, Tobias Höllerer (2004) defines an AR system as “one
that combines real and computer generated information in a real environment,
interactively and in real-time and aligns virtual objects with physical ones” (Höllerer
and Feiner 2004). This form of mobile AR has been typified in science-fiction film
and literature with examples such as Princess Leia’s holographic avatar, William
Gibson’s AR tableau of River Phoenix in Spook Country and the user interfaces in
Minority Report and Iron Man, to name a few. AR interfaces would know your
physical position and head orientation so that an onboard microcomputer could
determine how to correctly “insert” objects into your visual perception in such
a manner as to convince you that they are as real as the physical objects they
are next to.
8.2.2 Social Immersion
Margolis et al. (2012) expanded upon Jonathan Steuer’s (1992) definition of Virtual
Reality (Steuer 1992) to describe an emerging form of Social Immersion used in
mobile AR as “the social richness of a mediated environment as defined by its
interactive features, that is, the way in which an environment layers networked
information via real and virtual means” (Margolis et al. 2012). This form of media
is not simply virtual object overlays, but of stories that connect these objects to the
people and places that imbue their meaning. This new view of immersion enables
us to describe participatory practices within contemporary culture that use mobile
social media networking applications which give us an unprecedented ability to
share information with social groups and the world instantaneously. For example,
Wikitude streams information about nearby events, tweets, Wikipedia articles
and user reviews into the live camera view on smart phones. Also, transmedia
storytelling creates a sense of social immersion by unfolding elements of a fictional
story across different platforms to create an interactive narrative over time and space.
8.3 Socially Immersive Art
8.3.1 Head Mounted AR (Wearable Smartphones)
Head Mounted Displays (HMDs) have been used as a means of displaying Virtual
Reality content for decades. These systems present 3D computer graphics by placing
small screens very close to the eyes. There are several devices that have come to
[email protected]
8 Immersive Art in Augmented Reality
151
market over the last few years such as Epson’s Moverio, NEC’s Tele Scouter and
the Oculus Rift which can be described using the sensorial form of immersion for
AR since they overlay graphics onto the real world through see-through optics or
by rendering CG over live video to opaque displays. It’s important to understand
these devices and the affordances they provide in order to explore what differentiates
them from new platforms that are now forthcoming which are better categorized as
providing social immersion for AR.
While technically the Oculus Rift is a VR HMD, given the fact that there are
other vendors (Ovrvision 2014) offering camera modification attachments to enable
real-time video overlays within the Rift display, it is worthwhile to compare and
evaluate the Rift as a viable head mounted AR device. The latest version of the
Rift dubbed Crystal Cove incorporates markers on the outside of the Rift along
with an external camera that enables head tracking so that as users move their
heads in the real world the virtual environment responds accordingly. This is a key
feature for designing an AR system since the computer needs to know exactly where
to position virtual objects on top of the live video stream as users move around.
The AR-Rift (AR-Rift 2014) is also particularly interesting as it enables users to
create virtual desktops within AR to create entirely new ways to work with standard
applications.
As one of the most commercially successful HMD manufacturers, Oculus Rift
raised ten times more than the original $250,000 Kickstarter campaign goal. It is
therefore important to understand how they are focusing their business interests
on the gaming sector as evidenced by hiring several of the most influential game
developers such as Epic’s John Carmack and EA’s David DeMartini as key company
strategists. This will likely make the device much more powerful and user-friendly
than if it were intended primarily for scientific or military use like so many other VR
headsets. It’s then worth considering how game consoles have become increasingly
social as they have become inherently networked. Both the PS4 and Xbox One have
added built-in DVR systems to enable gameplay recording and playback as well
as enabling sharing those videos to social media sites like Facebook. Perhaps even
more importantly, Sony has added broadcast capabilities to the PS4 so that users can
stream their live gameplay (with live picture-in-picture camera view of the player)
to sites like twitch.tv and ustream.tv with the push of a button on the controller.
Some of these home broadcasts reach tens of thousands of viewers and in the first 6
weeks since launch, PS4 users have already broadcast 1.7 million streams generating
more than 55 million minutes while being watched by 22 million sessions (Twitch:
PS4 streaming 2014). These broadcasts are accompanied by chat windows to enable
discussion both within the virtual community as well as with the original player.
It’s then very likely that these highly established game consoles will influence the
Rift and so we can consider how these emerging video sharing platforms may be
integrated with a HMD like the Rift.
Google Glass on the other hand is a new device that might be best described
within AR as a heads-up display (HUD) rather than a HMD like the Rift. HUDs were
[email protected]
152
T. Margolis
originally designed starting around WW II for military use to overlay navigational
information onto aircraft windshields or helmets, but are now commonplace within
most gaming user interfaces to display information such as health and inventory
as well as chat boxes and mini-maps to collaborate with teammates. Google Glass
works in a similar way by showing various types of information in the upper-right
section of your field of view. It can provide you with directions, look up facts,
translate your voice or physical signage and of course help connect you with others.
One of Glass’s most powerful features is the ability to live-stream video from your
own point of view anywhere and anytime. It’s similar to a Google Hangout, but
it’s completely untethered. These features quite distinctly differentiate Glass from
other forms of AR by creating what we can describe as Social Immersion rather than
Sensorial Immersion like the Rift.
8.3.2 Socially Immersive Artworks
Google Glass art is like the new reality TV – Glass art through livestreaming ties
back to artist and Internet pioneer Josh Harris and his various ‘90s projects like
“Quiet: We Live in Public” which surrounded 100 secluded artists with webcams
and broadcast their every move onto the Internet. Now Glass ups the ante by
enabling completely untethered high quality directorial power to broadcast your
own life or the lives of those you think the public may covet.
Helpouts (Helpouts by Google) were released in November 2013 in order to
create a public marketplace for purported experts to provide real-time video chats
for a fee. They are in essence an extension of Google Hangouts which have been
continuously evolving to provide better quality multi-user video chat rooms with
included screen sharing and embedded text chats. While the marketplace is only a
couple months old, there are already thousands of advertised services listed there
grouped into the following categories: Art & Music, Computers & Electronics,
Cooking, Education & Careers, Fashion & Beauty, Fitness & Nutrition, Health and
finally Home & Garden. The specific services being offered tend to range from premade lessons (e.g. over 50 people offering Helpouts to Learn to Play Guitar) to more
personal problem-solving sessions (e.g. over 70 people offering Helpouts to solve
peoples’ Computer Issues). These live video sessions are particularly well suited for
use with Glass in that many of them require the instructor to be hands free in order
to help execute the Helpout.
New York artist Molly Crabapple made a drawing of porn star and circus
performer Stoya while wearing a pair of hacked Glasses in a project entitled
Glass Gaze (Artist gazes at porn performer 2013). The Glass enabled Crabapple
to live-stream what she saw and thereby “to take the most classical, thousandyear-old way of looking: an artist looking at a model. It’s the most undistracted,
direct, unmediated thing. And then run it through this super-mediated, captured,
commodified way of seeing that Glass represents.” Crabapple reports that wearing
[email protected]
8 Immersive Art in Augmented Reality
153
Fig. 8.1 First person perspective via Google Glass from an artist Katz interviewed as she created
a new drawing
Google Glass while drawing made it tempting to constantly look at the image in the
corner of her vision of what was being recorded. Crabapple describes the uncanny
drawing process by stating “Glass is all about distraction, and it’s the ultimate in
unphysical – it’s light on a piece of clear plastic in front of your eye.”
New York city curator Samantha Katz created Gallery Glass in September 2013
as a YouTube channel (Gallery Glass – YouTube) to highlight Brooklyn artists by
filming 30 artists over 30 days while wearing Google Glass (Bushwick Artist Uses
‘Google Glass’ 2013). Although the videos posted to YouTube are still edited and
generally feel like they could have been filmed with a handheld camera, as shown
in Fig. 8.1, Katz argues that Glass can give viewers a first person perspective that
presents “what it’s like to walk in a studio.” At times Katz hands her Google Glasses
over to the artists as they draw or paint, but unfortunately the videos have been edited
so that the audio from the interviews are overlaid onto the visuals of the painting and
drawing. While this is a standard editing technique for cinematic film, in the case
here for Gallery Glass it tends to disrupt the attempt at experiencing the creation
of an artwork from the artists’ perspective. Katz admits the project was a “beta”
test and is interested in continuing to explore the potential use of Glass for learning
about art.
It could be argued that David Datuna’s Viewpoint of Billions is perhaps one of the
only works of art to date that is truly unique to Google Glass (David Datuna 2014).
Datuna worked with mobile app developers BrickSimple to create an interactive
installation that at first glance represents an American flag – see Fig. 8.2. However,
as one approaches the artwork, you’ll see the surface is covered with found eyeglass
lenses. Beneath the lenses is a collage of photographs and newspaper clippings as
well as hidden cameras and microcomputers. The cameras can record visitors as they
[email protected]
154
T. Margolis
Fig. 8.2 Viewpoint of billions by New York artist David Datuna at Art Basel Miami (Datuna
Glassfeed)
experience the artwork and then live stream that video back to those same visitors
wearing the Google Glasses. This creates a sort of feedback mechanism in which the
artwork appears to be watching you back as you watch it. While other video artists
like Steina and Woody Vasulka pioneered using video feedback, Datuna’s use of
Glass to transmit a live video of one’s self via a camera hidden directly within the
artwork perhaps takes this concept to a new level.
8.4 Collaborative Locative Media
8.4.1 Handheld AR Platforms
In order to understand how mobile AR is evolving into a socially immersive
medium, we will first present two exemplars of popular mobile Augmented Reality
applications for smart phones and tablets. Both of these began with a traditional
attempt at fulfilling the original AR goal of providing a platform for developers to
create sensorially immersive AR experiences. However, they both continue to evolve
to enable authors of mobile AR experiences to produce content in an increasingly
more accessible manner as well as provide more and more hooks into social media
platforms for quickly sharing content and experiences with friends, colleagues and
collaborators.
[email protected]
8 Immersive Art in Augmented Reality
155
Layar is a good example of a popular consumer mobile AR browser that allows
developers to geotag a location with images, text, sound or other media (Layar).
When a user is exploring the physical world through their phone or tablet, they
are able to view nearby AR content overlaid on top of a real-time video feed via
the device’s camera. Layar enables developers to encourage end-users to easily
share their AR experiences with friends on various social media platforms such
as Facebook and Twitter.
Over time, Layar has changed its business model as to how to best support mobile
AR technologies. They have added computer vision based scanning to their toolkit
of sensing technologies. So instead of finding content solely based on your physical
location, you can now point your phone or tablet’s camera at an image target.
Historically, AR marker targets tended to be black and white geometric patterns
that resembled 2D bar charts. Nowadays, Layar and many other computer vision
manufacturers use markerless targets that can be defined by any natural image
as long as it adheres to a few basic technical requirements such as non-repeating
patterns, no motion blur, avoiding reflections and transparencies and has generally
flat illumination. This new sensing platform enabled Layar and others to support AR
experiences within interior spaces where GPS satellites would not work reliably.
However, from a demographics perspective, this new feature provides for countless
new advertising and marketing opportunities as now companies can integrate mobile
AR with their brand experience much easier.
Qualcomm’s AR Software Development Kit, Vuforia, enables Android and
iPhone developers to incorporate sophisticated natural image based tracking into
their mobile AR applications (Qualcomm Vuforia). Vuforia also requires developers
to specify an image target that users can then point their mobile phone or tablet
towards to view a real-time augment. However, Vuforia also adds extended tracking
to enable users to have continuous visual experiences even when the tracked target
is outside of the camera view. While this might seem to be a trivial technicality, it
provides for a much more seamless and immersive AR experience since now you
aren’t required to keep your mobile device pointed at a particular physical object,
but instead one can simply use their phone or tablet to explore the entire surrounding
environment as well.
It’s important to understand that Layar and Vuforia have taken different
approaches to mobile AR development and publishing. As an AR browser, Layar
requires that end users install a single official Layar application onto their smart
device. Once the user has installed this app, they have access to all of the AR
“channels” that developers have published using the Layar platform. Users can
either scan an image target or search through keywords and categories to find
results of what AR content might exist near them. Vuforia on the other hand is
meant to be packaged as a custom application for each AR experience. This means
that end users must download and install a different app for every application built
using Vuforia. However, it also provides developers with much more control over
the application and enables apps to utilize all available Android or iPhone services
and libraries to build a highly rich and engaging mobile experience.
[email protected]
156
T. Margolis
Fig. 8.3 Spiral Jetty by Robert Smithson (1970). 1,500 ft in Utah, US
8.4.2 Handheld AR Media
These new socially based immersive media platforms provide affordances for new
forms of locative artworks that were not possible before. Here we will discuss how
geo-locative artworks and experiences have been emerging and where there may be
precedents of artists and gamers working collaboratively within large scale environments (both physical and virtual) that perhaps have laid the cultural landscape for
artists today to explore these new socially immersive AR collaborations.
In the late 1960s Joseph Beuys developed a theory of Social Sculpture that
describes a society in which “Everyone is an artist”. He illustrates the idea that art is
meant to be participatory and can hold the power to transform society. Like Richard
Wagner, Beuys strongly argued for a Gesamtkunstwerk in which society as a whole
was to be regarded as one great work of art. This helped to lay the foundation for
Land Art in the late ‘60s where the landscape and the work of art were inextricably
linked as seen in Fig. 8.3 documenting Spiral Jetty. Much like contemporary AR
artwork, land art was meant as a protest against the traditional museum and gallery
art worlds in order to bring power back to the community.
Perhaps inspired by the well-known artists who have been transforming the
landscape of our planet for the last 50 years, other lesser known artists have been
transforming the landscapes of virtual environments for the last decade. Perhaps
most famous of these online virtual worlds is Second Life (SL) which has had
over 20 million subscribers. Many SL users pride themselves on the creative design
of space and objects in SL and collaborate in groups on large-scale installations.
Some of these installations mimic spaces from the real world such as in Becoming
[email protected]
8 Immersive Art in Augmented Reality
157
Dragon (Cárdenas et al. 2009) while others create psychedelic art installations like
StormEye (Story and Enfield 2009) out of giant moving shapes coated in video
and sound. Artists build these virtual installations and then share them with other
citizens of Second Life to experience.
However, this idea of locative art within a virtual world is not new to online
platforms. Gamers have been designing their own levels for decades. Lode Runner
from 1983 was one of the earliest games to come with a level editor that enabled
level designers to create their own custom environments for playing within. The
website worldofleveldesign.com is built exclusively for this community to aid
sharing articles and tutorials for environment artists and level designers to help with
all aspects of working on level designs and 2d/3d environments. Also, LEVELDESIGN.org has a reference database of tagged image screenshots to help share
ideas on level design.
Now, there are countless massively multiplayer online (MMO) games where
large groups of users collaborate together in teams and factions to move through
the game story. In addition to solving the challenges presented by the original game
designers, players often choose to create their own virtual art installations within
the game world. LittleBigPlanet is an example launched in 2008 on the Playstation
3 that focused primarily on user-generated content as evidenced by their tagline
“Play, Create, Share” and over 1 million user-created levels to play.
Ingress is a transmedia game by Google played out through live events, the
web and mobile applications. The players’ goal is to capture “portals” which are
co-located with objects in the real world such as public art sculptures, landmarks
and libraries. Once captured, three or more portals can be linked together to create
fields that give users points. There are two factions players must choose between,
the enlightened and the resistance, which battle each other to control these portals
and fields. As of January 2014, over one million people are walking through cities
around the planet capturing portals and linking fields.
There are no pre-defined rules on what the shape of these portal fields should
look like and most of them are abstract shapes designed to maximize the amount
of underlying land controlled with no consideration given to aesthetics. However,
having reviewed the history of locative art in Social Sculpture, computer gaming
and science fiction, we might find it inevitable that players have begun to create
specific designs within these augmented reality platforms so that they link portals
together in a strategic manner as to form a representational image across actual
cities. This so called Field Art may connect dozens of points spanning entire cities.
Given that every portal and field is in constant danger of being attacked, it is easy
for one to appreciate the transient complexity of these undertakings and the need
for large groups of people to help create them. We might perhaps expect to find that
several Field Art installations utilize collaborations between factions to create even
more complex designs. In the screenshots shown below in Fig. 8.4, you can identify
the two factions contributions by looking at the two colors on the maps.
[email protected]
158
T. Margolis
Fig. 8.4 Field art created with Ingress. (a) Flower in Germany. (b) Christmas tree in London is
over two miles long (Monnington 2013). (c) Butterfly in Heidelberg, Germany (RNK Field Art).
(d) Woodpecker in Germany (Stadler 2013)
In addition to aesthetic designs created within Ingress, players have used the
game platform for political and social commentary. After the Boston marathon
bombing, MIT Ingress players from both opposing factions agreed to a temporary
ceasefire on the MIT campus as well as erected a dual owned virtual memorial for
the slain MIT police officer Sean Collier.
[email protected]
8 Immersive Art in Augmented Reality
159
8.5 Conclusion
We are beginning to experience mobile Augmented Reality in a new context within
a culture that has come to expect (and rely upon) persistent connections between
people and information. Naturally then we architect systems to better connect with
each other through the Internet. These new platforms are enabling us to create new
worlds that bring us together through our common interests. So whether it is learning
how to paint nudes with 100 strangers via Google Glass Helpouts or touring CERN
via Glass or creating virtual landscape art installations on top of and in the middle
of New York City, these new socially immersive media are changing the way we
can create, share and play.
Acknowledgements Eternal thanks to my beautiful wife Tracy Cornish for her love and support
as well as her critical discourse in exploring Social Immersion with me. Many thanks to Alexander
Nano Horn for his help reviewing and guiding this article towards clarity as well as providing
wonderful examples to help contextualize some of the history of level design in gaming.
References
Cárdenas M, Head C, Margolis T, Greco K. Becoming dragon: a mixed reality durational
performance in Second Life. Proc SPIE. 2009;7238:723807-723807-13.
David Datuna for Glass. http://datuna.com/. Accessed 8 May 2014.
Höllerer TH, Feiner SK. Mobile augmented reality. In: Telegeoinformatics: location-based
computing and services. United Kingdom: Taylor & Francis Books Ltd; 2004. pp. 1–39.
Margolis T, Cornish T, Berry R, DeFanti TA. Immersive realities: articulating the shift from VR to
mobile AR through artistic practice. In: Proceedings of SPIE, The engineering reality of virtual
reality, 82890F. SPIE Digital Library, San Francisco, California; 2012.
Monnington L. tl;dr: we made a purdy tree!; 2013.
Ovrvision. http://ovrvision.com/. Accessed 8 May 2014.
Qualcomm Vuforia. https://www.vuforia.com (2010). Accessed 29 Jan 2014.
RNK Field Art. https://plus.google.com/communities/105136616454643645459 (2013). Accessed
29 Jan 2014.
Stadler M. Field art Greifswald – operation woodpecker; 2013.
Steptoe W. AR-Rift. http://willsteptoe.com/post/66968953089/ar-rift-part-1 (2013). Accessed 29
Jan 2014.
Steuer J. Defining virtual reality: dimensions determining telepresence. J Commun. 1992;42:
73–93.
Story D, Enfield D. StormEye 2009. http://slstormeye.blogspot.com/ (29 January 2014).
[email protected]