Incorporating Subliminal Perception in Synthetic
Environments
David Pizzi1, Ilkka Kosunen2, Cristina Viganó3, Anna Maria Polli2, Imtiaj Ahmed2, Daniele
Zanella4, Marc Cavazza1, Sid Kouider5, Jonathan Freeman6, Luciano Gamberini3, Giulio Jacucci2
1
Teesside
University,
Middlesbrough,
England
{d.pizzi;
m.o.cavazza}
@tees.ac.uk
5
3
4
6
University of
University of
Electrolux
Ecole
Goldsmiths,
Helsinki, Finland
Padova, Italy
Italia S.P.A.,
Normale University of
{ilkka.kosunen; luciano.gamberini Pordenone, Supérieure,
London,
giulio.jaccucci}
@gmail.com
Italy
Paris,
England
@helsinki.fi
daniele.zanella
France
j.freeman
@electrolux.it sid.kouider @gold.ac.uk
@ens.fr
2
ABSTRACT
processing through psychophysiology has been done in
laboratory environment on specific tasks [1]. The use of
subliminal stimuli has been shown to prevent overloading
of the user when a huge amount of data needs to be
explored [9]. We propose to investigate a more complex
scenario to explore typology and delivery of subliminal
cues in the context of realistic information and 3D content
of appliances. Furthermore having found the appropriate
mechanisms to communicate subliminally with users the
interest extends beyond the “local” impact on the
immediate action and perception of the user to the potential
“global” impact of the intelligent use of such mechanisms
to the overall product experience and unfolding narrative.
Advanced interactive visualization such as in virtual
environments and ubiquitous interaction paradigms pose new
challenges and opportunities in considering real-time
responses to subliminal cues. In this paper, we propose a
synthetic reality platform that, combined with
psychophysiological recordings, enables us to study in realtime the effects of various subliminal cues. We endeavor to
integrate various aspects known to be relevant to implicit
perception. The context is of consumer experience and
choice of an artifact where the generation of subliminal
perception through an intelligent 3D interface controls the
spatio-temporal aspects of the information displayed and of
the emergent narrative. One novel contribution of this work
is the programmable nature of the interface that exploits
known perceptive phenomena (e.g. masking, crowding and
change blindness) to generate subliminal perception.
To address these problems we have developed a synthetic
reality platform that gives us an exact control of type,
timing and placement of subliminal cues during an
embodied human-computer interaction with a 3D scene
rendered on a large display.
Author Keywords 3D Environment, Subliminal Cues,
Interactive Narrative.
In the following, we present the interactive 3D environment
we created to study implicit responses and subliminal cues
in the context of interaction with appliance models. We
discuss the technical set up, content, application and
interactivity that are necessary to develop a research
platform. The approach is to allow immersiveness and
intuitive interaction for an optimal engagement with the
content. To this end the size of the display is seen as an
important feature. For example, we have experimented with
a 65 inch display. When interacting with large displays
intuitive interaction includes multi-touch and gesture
interfaces [6]. We chose to develop explicit interaction and
commands of subjects through a depth camera. Issues in
developing gestures in this context include developing
effortless interaction (e.g. raising arms can be tiring) and
transparent feedback in the 3D environment [3].
ACM Classification Keywords H.5.1 [Multimedia
Information Systems]: Artificial, Augmented, Virtual
Realities; H.5.2 [User interfaces]: Evaluation/methodology.
General Terms Design, Experimentation, Human Factors,
Measurement.
INTRODUCTION
Previous work [5] suggests that unconscious processes are
actually involved in the (consumer) choice of products that
manifest a certain level of complexity, which we assume to
be correlated to a mapping between decision criteria and
artifact perceptive features. Products such as appliances can
be displayed in 3D environments relying on both system
and user agency to customize an emergent product
experience and narrative. Traditionally, investigating
subliminal cues, implicit responses and other unconscious
Copyright is held by the author/owner(s).
UbiComp’12, September 5-8, 2012, Pittsburgh, USA.
ACM 978-1-4503-1224-0/12/09.
1139
Figure 1. Overview of Presentation Strategy in the CEEDs Engine. According to the current user and world states, the real-time
planner computes and sends sequences of actions to be executed by the immersive visualization engine. User state formalizes both
implicit (e.g. EDA, EMG and EEG signals) and explicit (e.g. visual focus, pointing gestures, etc.) information.
Interactivity of Virtual Appliances
dynamic use of implicit subliminal visualization as part of
the data presentation strategy, and to incorporate this in the
real-time, interactive user experience.
Virtual Environments support many new applications in
data visualization as well as the possibility to enhance user
experience to facilitate access or integration of such data. In
addition, recent progress in Intelligent User Interfaces
facilitates the staging of virtual experiments on perception
and interaction by explicitly encoding such hypotheses in
the knowledge layer that underpins the intelligent interface.
In this paper, we present on-going work aimed at extending
virtual environments to support implicit perception.
Previous work [2] [10] suggested that some perceptive
phenomena traditionally considered part of implicit
perception, such as change blindness, could be exploited in
the design of virtual reality systems. Rather than directly
encoding specific perceptive phenomena in the design of
the Virtual Reality system, our objective is to support the
The application that supports our experiments explores
consumer visualization of domestic appliances. The
rationale is that consumer choice has been demonstrated to
involve implicit phenomena [5], that these artifacts are well
suited for 3D interactive visualization (as well as their
enhancement by additional textual or graphic information)
and that the mechanisms for perceiving the features of these
products are largely unknown. We also hope that the
realism of the task can facilitate the definition of ground
truth mechanisms for evaluation of perception or decisionmaking.
1140
Virtual Content of Appliances
Candidate strategies include crowding, change blindness
and masking. A subsequent one consists in devising display
strategies for these phenomena, which are defined taking
into consideration the real-time detection of user position
and field of view (region of interest), and the various
spatial, visual and temporal capabilities of the VR display.
Electrolux Group’s media library is the database where, at
present, it is possible to find all the product images, videos,
logos and other content from across all Electrolux group’s
brands including Electrolux, AEG and Zanussi. Among the
contents, it is possible to find trailers, guidelines, banners,
stickers, leaflets, pictures, short movies, animations, radio
podcasts, TV commercials and CAD models.
STAGING
APPLIANCES:
INTERACTION
VISUALIZATION
AND
Promotional material appliances are often displayed in
contexts such as an integrated kitchen; therefore, we used
the same strategy as an introduction to recreate ecological
conditions for exploration and choice. We implemented an
environment where subjects first encounter appliance
categories in a kitchen environment. In this phase users may
select one appliance type integrated in the kitchen operating
a pointer. This initiates user exploration, navigation and
manipulation of the artifacts, which will create an
appropriate context for the introduction of subliminal cues.
User interaction with the appliances is accomplished using
a depth camera (i.e. Microsoft® Kinect™), which allows
gesture recognition. As an example of scene exploration,
after highlighting an appliance with the pointer by a
pushing gesture, the kitchen disappears and the user is
shown all the available models of the chosen appliance type
(e.g. a refrigerator, an oven and a dishwasher). After
selection of one of them, the new appliance replaces the
previous one and the kitchen is displayed again. This virtual
environment allows focusing on categories of appliances
starting from a realistic exploration. During subsequent
stages, the system also supports mechanisms for multiple
visualizations,
duplication
of
appliance
parts,
magnification, and enhancement by textual and symbolic
elements and, of course, subliminal information display.
CAD models are prepared during the product development
process and traditionally used as the basis for
manufacturing and testing the product. With recent
developments in realism reached by 3D graphics,
Electrolux started to use these models to create valuable
content to support activities such as training, marketing,
sales and research. For example, CAD models can be used
by software houses, which integrate the model as it is in
their kitchen configuration software, or agencies, which
take care of the communication campaigns, and to create
short video clips, pictures for catalogues, applications for
mobile devices, etc.
SYSTEM OVERVIEW
Our system supports this industrial scenario by showing
how this content can be used not only in passive
advertisement videos but in interactive applications. The
system (Figure 1) presents itself as a virtual environment
supporting user tracking and interaction via traditional (i.e.
physical) means. The user navigates freely and explores the
environment featuring various appliances that can react to
user interaction (both explicit, interface-like, and implicit,
acquired through physiological sensors). At the heart of the
system, the narrative engine aims at inducing specific user
experiences by orchestrating the display and behavior of the
various appliances so as to induce a given set of perceptive
events. We are extending the traditional concept of
interactive narrative to encompass the temporal presentation
of a set of scenes featuring the appliances. Each scene and
each transition between scenes, are aiming at the
presentation of a given configuration of perceptive features.
Within a given scene, the distribution between features
presented explicitly and those presented implicitly form
part of the experimental hypotheses to be assessed. The
underlying mechanisms for the narrative engine are adapted
from state-of-the-art work in interactive narrative. The
display of specific features is under the control of
visualization operators. Specific perceptive phenomena can
be organized through the temporal/causal arrangement of
sequence of visualization operators, such sequences being
generated in real-time using heuristic search planning. The
narrative engine constantly plans for perceptive impressions
in the background and these are staged in the virtual
environment. User actions and user input through
physiological sensors are constantly fed back to the planner
so as to adjust the subsequent display strategy.
We have adapted original CAD models (simplified and with
a reduced polygonal resolution), meaning that interaction
with a refrigerator, which constitutes our main example in
the next sections, is possible through the same mechanism.
Appliance behavior is defined within the model itself in
terms of interaction events and how mobile parts should
respond to interaction events, using standard features from
the Unity game engine that we use for visualization.
Individual appliances can also be interactively rendered
zooming, rotating and opening doors and drawers. The
depth camera interaction provides different opportunities to
define interaction following affordance models of intuitive
gestures or the pointing, highlighting and selecting
paradigm explained above. Furthermore it is possible to
activate the Physics engine on some mobile parts where
user interaction initiates physical motion providing a more
realistic exploration.
ELICITING IMPLICIT PERCEPTION
The overall narrative principle consists in displaying
information according to certain semantic dimensions of
user experience, which can be mapped onto properties of
the appliance (Figure 2). Some of these dimensions are
One important step in the design of the system consists in
identifying strategies for implicit information display.
1141
Figure 2. Example of Exploratory User Experiences. Several presentation strategies are deployed according to the different
possible scenario progressions. At start, the user faces several objects within a virtual 3D environment. The nearest appliance
attracts the user attention by subtle (e.g. oscillations), more salient (e.g. automatic opening) or even implicit (e.g. suggestions)
sequences of actions inside or outside her field of view. The overall theme of the user experience follows three main dimensions
(usability, practicality and aesthetics) and different strategies are presented according to the dominant direction chosen (e.g.
lighting effects for design or ease of access for usability). Finally, subliminal information can also be presented at the user visual
periphery to suggest new exploratory directions (e.g. when the user state remains stationary for a prolonged period of time).
interrupting the task at hand. It is to be used parsimoniously
to leverage on its potential.
usability (which characterizes the physical interaction and
the appliance’s operation), practicality (relations to usage)
and aesthetics (design preferences and social norms). It is
important to note that these dimensions are not entirely
subordinated to the appliance description; some of them,
including practicality and usage, relate to user profiles and
how they would interpret generic properties of the
appliance from their personal perspective.
One typical example would be the display of subliminal
information assisting the user in the operation of the
appliance upon detection of difficulties in interacting with
movable internal components. Let us assume that the user
in an exploratory phase of explicit interaction with the
various internal compartments of a refrigerator: upon
detection of difficulties (which can take place through a
combination of physiological signals and task performance
analysis), the system may decide to provide subliminal
assistance, inspired by an approach described by De Vaul et
al. [5], contextualized to i) the type of parts on which the
user attention has been concentrated and ii) the current
position and orientation of the user.
The overall context is one of mixed-initiative: the user is
free to explore the appliance visually and through
interaction, while the appliance itself is endowed with
reactive behavior that adapts itself to the interaction history
to assist the user exploration. The system assists user
exploration based on its analysis of user experience rather
than an intrinsic “advertising” strategy dictating which type
of information or appliance feature to promote.
In a similar fashion, subliminal information can be used to
manage transitions between dimensions to be explored, or
preserve consistency of user experience by maintaining the
user’s interest on a given dimension. One particular case of
In this context, subliminal information should be allocated a
specific role: rather than simply behaving as an additional
information channel, it is to be used as a mechanism to
drive the user exploration at critical stages without
1142
Figure 3. Example of Subliminal Presentation Strategy using Masking Techniques (adapted from [8]). We assume a rendering
frame rate for the graphics engine of 60 fps, without irregularity during the masking technique. The first sequence (frames 1-31)
consists in displaying a mask (here a snowflake symbol). Priming between frames 31-34 makes use of a low-intensity, small size
picture of a tomato. This is followed by a shorter display of the same mask (three frames 34-37), then a more persistent (42 frames)
display of the target image (a clearer and larger image of a tomato). The use of such a target has been inspired by some of the
original Electrolux presentations of the refrigerator’s features. The impact of such a target will be assessed by considering
subsequent variations in the user’s topics or areas of interest.
devising an appropriate sampling rate to analyze user
interaction and plan system behavior.
transition consists in inviting the user to explore certain
aspects after detecting a phase of user inactivity. The nature
of subliminal informational content should also be part of
this research. We shall explore the use of text, symbols or
icons, as well as images directly congruent to the
experience themes at hand. One of the issues to be
considered is the type of response that can be elicited as a
function of the subliminal display strategy and the contents
of the subliminal information presented (e.g. semantic,
affective, etc.). Techniques for subliminal display include
masking (images) [8], crowding (symbol, text) [7], and
flashing (symbols). Display can take place preferentially in
peripheral areas of the user’s Field of View (FoV) for
crowding techniques, or inside the FoV itself (in the case of
masking techniques).
While the user can autonomously engage in an exploration
of the appliance’s features by freely manipulating doors,
compartments and drawers, the role of the system is to
control and offer a satisfying experience by i) measuring
and analyzing user activity and interest, and ii) by planning
and integrating which visual strategy to follow over the
next interaction period. User experience analysis consists of
a knowledge-based interpretation of the user’s interactions
with the system, which are available through the system’s
I/O devices. Integration is performed at different levels of
analysis:
The lowest level consists of assessing user activity
in relation to the appliance. This consists in monitoring
user’s position and orientation in relation to the appliance,
in particular when it is significant to determine areas of
interest: this is based on recognition of body gestures and
visual focus through head/eye tracking. The other element
of activity analysis is to record the actual operations on the
device, i.e. the interaction events that physically manipulate
the mobile or reactive parts of the appliance (doors,
compartments and any other real-world object commands if
available)
EXPLORING CONSUMMER APPLIANCE
Unlike traditional Virtual Reality systems, this interactive
scenario implements a “mixed-initiative” exploration of
kitchen appliances by articulating user actions with system
visual presentations. The concept of mixed-initiative was
originally developed in the field of human-computer
dialogue to authorize anytime user interaction rather than
only respond to prompts. In a VR setting, whose purpose is
information visualization, it consists in endowing virtual
objects with some form of autonomous behavior through
which they will present themselves in order to attract or
focus the user’s attention. In practice, it is not sufficient just
to include smart objects, and there is a need for the overall
system behavior itself to be under the control of a specific
module which will analyze user experience as a whole
(continuously and in real-time), to determine appropriate
system response. The form of support to an interactive
experience is similar to the one implemented in interactive
narrative systems, in which user interaction is interpreted to
produce the next stages of a visual narrative. We have thus
used similar technologies for system behavior based on AI
Planning techniques; one specific variant consists in
A higher level of analysis consists in interpreting
physical interaction in terms of user experience and
activity. The underlying principle is to map the region of
interest of the appliance to semantic dimensions relevant to
user exploration. For instance, external, non-functional
regions of the appliance devoid of affordances will be
associated to aesthetic dimensions, while internal, mobile
and functional parts may be associated to usability.
In the continuous process of integrating the user activity
with an appropriate overall experience the system response
has to be deliberative and interleaves phases of user
interaction with phases of information presentation (the
1143
REFERENCES
sampling rate for these phases is not fixed during any given
session, and can be adjusted depending on spontaneous user
activity). Information presentation takes the form of visual
narrative sequences generated using AI Planning
techniques. The next presentation operator is selected from
the perspective of narrative evolution. For instance the
planner determines whether it is preferable to introduce a
new perspective on the appliance’s properties rather than
remain in the same current state of exploration, deepening
the current presentation strategy (acting here as a form of
level of narrative details).
1. Bernat, E., Bunce, S., and Shevrin, H. Event-related
brain potentials differentiate positive and negative mood
adjectives during both supraliminal and subliminal
visual processing. In International Journal of
Psychophysiology, 42, 1, (2001), 11-34.
2. Beeharee, A. K., West, A. J., and Hubbold. R. Visual
attention based information culling for Distributed
Virtual Environments. In Proceedings of the ACM
symposium on Virtual Reality Software and Technology
(VRST '03). ACM, New York, NY, USA, (2003), 213222.
These presentation strategies consist of sequences of
variable length which can be of different nature depending
on the instant characterization of user experience (e.g.
exploration, discovery, assessment, etc.). In that sense, both
explicit and implicit dramatization elements are present in
our scenario. While the former is being used for
information reinforcement, the latter is rather more
appropriate when we need to change the phase of
interaction (Figure 3). For instance, when the analysis
reveals the user is facing an important choice (i.e. different
of what is actually happening), or where the level of
interaction is particularly weak, our hypothesis is that
subliminal information display can be accessed without
disrupting the user experience.
3. Bowman, D. and Hodges, L. User Interface Constraints
for Immersive Virtual Environment applications. In
Graphics, Visualization, and Usability Center Technical
Report, (1995).
4. De Vaul, R., Pentland, A., and Corey, A. The Memory
Glasses: Subliminal vs Overt Memory Support with
Imperfect Information. In ISWC 2003. (2003), 146-153
5. Dijksterhuis, A., Bos, M.W., Nordgren, L.F. and van
Baaren, R.B. On making the right choice: The
deliberation-without-attention effect. In Science, 311,
(2006), 1005-1007.
6. Jacucci, G., Morrison, A., Richard, G.T., Kleimola, J.,
Peltonen, P., Parisi, L., and Laitinen, T. Worlds of
information: designing for engagement at a public multitouch display. In ACM CHI '10: Proceedings of the 28th
international conference on Human factors in
computing systems, (2010), 2267-2276.
CONCLUSION
We have described work in progress on an intelligent
interface designed to elicit and study various forms of
implicit perception. To the best of our knowledge, this is a
new kind of application which integrates the various
contexts in which implicit processes are relevant, both in
perception, information acquisition, and decision making.
The intelligent part of the interface, which is based on AI
planning techniques, makes it possible to experiment with
various phenomena associated with subliminal perception,
provided these can be expressed as a sequence of
visualization operators. This should make it possible to
explore more complex phenomena than transient display,
including change blindness, flashing, crowding and
masking. Meanwhile in our preliminary experiments we
have shown that our system is capable of generating
contextual responses on both EDA and fEMG, potentially
closing the loop between explicit interaction, implicit
presentation of information and implicit user response.
7. Kouider, S., Berthet, V., and Faivre N. Preference is
biased by crowded facial expressions. In Psychological
Science, 22, (2011), 184-189.
8. Kouider, S. and Dehaene, S. Levels of processing during
non-conscious perception: a critical review of visual
masking. In Phil. Trans. R. Soc. B. 362, (2007), 857875.
9. Riener, A. Information injection below conscious
awareness: Potential of sensory channels. In 3rd
International Conference on Automotive User Interfaces
and Interactive Vehicular Applications Automotive
UI'11, (2011), 6.
10. Steinicke, F., Bruder, G., Hinrichs, K., and Willemsen,
P. Change blindness phenomena for stereoscopic
projection systems. In IEEE Virtual Reality 2010,
(2010), 187-194.
ACKNOWLEDGEMENTS
This work has been funded (in part) by the European
Commission under grant agreement CEEDs (FP7-ICT258749).
1144