Virtual Lemonade: Let’s Teleport Your Lemonade!
Nimesha Ranasinghe1 , Pravar Jain1 , Shienny Karwita1 , Ellen Yi-Luen Do1,2
Keio-NUS CUTE Center, Interactive & Digital Media Institute, National University of Singapore1
School of Industrial Design & School of Interactive Computing, Georgia Institute of Technology2
{nimesha, pravar, shienny, ellendo}@nus.edu.sg, [email protected]
ABSTRACT
This paper presents a novel methodology to digitally share
the flavor experience of a glass of lemonade (or other similar
beverages) remotely. The proposed method utilizes a sensor
to capture valuable information (primarily, the color and the
corresponding pH value) of the lemonade and a customized
tumbler to virtually simulate these properties using plain water.
Thus, the system consists of three main components: 1) the
lemonade sensor, 2) the communication protocol, and 3) a customized tumbler, acting as the lemonade simulator. Initially,
the sensor captures the color and the pH value of the lemonade
and encodes this information based on an established communication protocol for wireless transmission. On receiving the
information from the sensor, the lemonade simulator overlays
the color of the drink on plain water using an RGB Light
Emitting Diode (LED) and simulates sour taste sensations on
the user’s tongue via electrical stimulation. An experimental
study was conducted to evaluate this novel approach of digitally teleporting a glass of lemonade: 1) to assess the pre-taste
perceptions based on the user’s visual perceptions of the colors
(real vs. virtual lemonade) and 2) to assess the taste sensations
(real vs. virtual lemonade). By simulating the experience of
drinking a glass of lemonade through the digital reconstruction of the beverage’s main visual and taste factors, the results
from these experiments will be able to justify the feasibility of
teleporting a glass of lemonade using this novel methodology.
ACM Classification Keywords
H.5.1. Multimedia Information Systems: Artificial, augmented, and virtual realities
Author Keywords
Teleport; virtual lemonade; virtual reality; food teleportation;
beverage communication; multimodal interaction
INTRODUCTION
Social sharing of beverages among friends and loved ones
is a common practice among humans. However, thus far, it
has not been possible to digitally share beverages remotely,
for example, over the Internet. Three of the main challenges
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from [email protected].
TEI 2017, March 20–23, 2017, Yokohama, Japan.
Copyright © 2017 ACM ISBN 978-1-4503-4676-4/17/03 ...$15.00.
http://dx.doi.org/10.1145/3024969.3024977
Sensing
Communicating
encoding
decoding
RGB Sensor (Color)
pH sensor (Sourness)
XML protocol
Actuating
Lemonade Simulator
(color and electrical
stimulation)
Figure 1. The system architecture of the lemonade sharing platform
highlighting the three main modules.
encountered in this action are: 1) being a chemical sense, the
sense of taste is one of the least understood and most subjective
human senses, 2) the need for complex sensing technologies,
similar to an electronic tongue, to detect the chemical composition of the beverage, and 3) the impracticalities of transferring
or storing chemicals in an interactive system [19].
As a solution, this paper introduces a new method and a digital
platform for remotely sharing a glass of lemonade by sensing,
communicating, and simulating its flavor information. The
proposed method is based on the following two concepts: 1)
usually, on seeing a glass of lemonade we create a pre-taste in
our mind based on the beverage’s color, fizziness, temperature,
and other such attributes [15], 2) on drinking it, we perceive
the multisensory experience of flavor, which includes the factors mentioned above, as well as taste sensations (primarily
sourness in the case of lemonade), 3) additionally, it is noticed that the visual cues such as the color of lemonade may
influence a cross-modal effect on the perceptions of different
sensory channels (e.g. brighter yellow colors may provoke a
strong sour taste sensation) [17]. Based on these phenomena,
we hypothesize that, by capturing and simulating the color and
sourness of a glass of lemonade, we can digitally teleport its
flavor.
Therefore, in this proposed platform, as illustrated in Figure 1,
we digitally sense, transfer, and simulate two of the most
important attributes of a glass of lemonade: the color and the
degree of sourness. To sense these two attributes, an RGB
color sensor and a pH sensor are used. For the purpose of
simulation, we have developed a customized tumbler that
overlays different colors on plain water using an RGB LED
and delivers controlled electrical pulses on the user’s tongue
to digitally simulate varying levels of sourness. Utilizing these
stimulation methods enables us to further study the human
perception of sourness with different colored beverages and to
optimize the digital simulation of these factors.
A user experiment was conducted to assess and compare users’
experiences of real and virtual lemonades regarding the perception of sourness. It was designed specifically to compare
two aspects: 1) preconception of sourness based on the color
and 2) perception of sour taste sensations between the real and
virtual lemonades. The key highlights of the results suggest
that there is no statistically significant difference between the
real and virtual lemonades with regards to the perception of
sourness when both the pre-taste and actual taste feedbacks
are combined. However, in general, it was noted that a significantly higher perception of the sour taste was associated with
the real lemonade when compared to the virtual lemonade.
In the rest of this paper, we first present a review of related
literature on the crossmodal perception of taste sensations,
the electrical stimulation of sour taste and the digital communication of flavors. Based on these existing works, we then
introduce a new digital method to teleport a glass of lemonade
remotely. Following this, we describe the execution and results of the user experiment that investigating how participants
experience real and simulated lemonades using the proposed
system. Finally, before concluding the paper, we discuss the
experimental results and propose several future works based
on the limitations of this approach.
RELATED WORK
Digital interactions related to food and beverages have been
identified as an extremely challenging area in the HumanComputer Interaction (HCI) domain. Due to physical and
perceptual factors, it is even more difficult to realize digital
interactions related to food and beverages when the interaction takes place remotely from the physical food or beverage
source. This section reviews existing pieces of literature that
relate to the three most important aspects of this challenge:
pre-perception of flavors based on the color of a beverage,
simulating taste sensations, and communication of flavors.
Pre-perception of Flavors Based on Colors
Pre-perception of taste sensations (pre-taste) is mainly recognized as a cross-modal interaction between colors and taste
sensations. A very useful research work conducted by Narumi
et al. [10] experimentally studied this phenomenon in interactive systems. They describe an experiment with a pseudogustatory display based on the virtual color of a real drink.
Here, a wireless LED module attached to the bottom of a transparent plastic cup is used to superimpose the virtual color of a
customized drink. Studies with this methodology examined
the cross-sensory effects of visual feedback and flavor interpretation of real drinks, and results from this suggested that
different colors may induce a perception of different flavors,
even with the same drink.
Similarly, there are a number of research works that have
reported the influence of food and beverage colors on flavor
perception [15, 16]. These studies are mainly concerned with
exploring associations of food and beverage colors with the
perception of flavors (such as lemon, banana, and the like)
or basic taste sensations (such as sour, sweet, salty, bitter,
and umami) and their intensity levels. These studies have
assessed colors that people commonly associate with specific
taste sensations. For example, it has been found that yellow
and green are commonly linked to sour sensations while red
to sweet sensations. They have also noticed slight differences
in perceptions based on participant’s culture and geographical
region.
Also, several studies have highlighted that individuals usually
have a preconception that brightly colored meals and beverages will be more palatable, and that the color of the food
and drink generally contributes to their decision of overall acceptability [17, 4]. Similarly, individuals usually get confused
when they taste colorless solutions with different flavors simply due to the lack of color information. These observations
and phenomenon underline the importance of crossmodality
in flavor perception and the influence that color can have on
this aspect of eating and drinking.
Flavor Simulation
Recently, there has been an active growth in attention towards
researching the simulation of taste sensations in interactive
systems. Ranasinghe et al. presents Digital Taste Interface
to simulate primary taste sensations based on electrical and
thermal stimulation of the human tongue in [11, 14]. Based on
their experimental results, sour, salty, bitter, and sweet (minor)
tastes are simulated through electrical and thermal stimulation
on the human tongue.
FunRasa is an interactive drinking platform which enhances
drinking experience by combining techniques of overlaying
different colors on the beverage and stimulating the tongue
using electrical pulses through a 3D printed straw. They also
report similar preliminary results where the participants perceived enhanced sourness, saltiness, and bitterness [12]. Similarly, Taste+ has several utensils such as spoon and bottle
to augment the flavors of food and beverages [13]. Again,
the perception of sour, bitter, and salty sensations were reported during these studies with varying levels of intensities
and accuracies.
Alternatively, Nakamura et al. demonstrate the use of electricity for augmented gustation in [8]. They apply an electric
current through isotonic drinks (which contains electrolytes)
and food (juicy vegetables, fruits, and cheese) to change their
taste perceptions. In this study, the main concern is the change
of voltage levels to augment the sensations of food items. However, applying an electric current through isotonic drinks could
elicit chemical reactions (through the process of electrolysis),
which could be harmful in certain situations. They have also
explored and measured the latency between presentation and
perception using visual and electric taste stimuli in [9].
Apart from the research works described above, several interactive systems explore other important sensory aspects during
the consumption of a beverage such as sounds, temperature,
and vibrations. Straw User Interface, proposed by Hashimoto
et al. [5], allows users to experience virtual drinking through
an interactive system. The drinking experiences are generated
by referencing prerecorded data of actual sounds, pressures,
and vibrations produced by drinking from a normal straw.
“Affecting Tumbler” [18] also focuses on altering the flavor
perceptions of a beverage by applying thermal sensations on
the skin around the nose to simulate skin temperature changes.
User studies with this system suggested that the flavor richness
and aftertaste strength were significantly improved by heating
up the skin around the nasal region.
Communication of Flavors
Scientists and designers, as well as sci-fi novelists, have long
been anticipating the remote communication of flavors to create novel interactions. In Taste/IP [11], the authors presented a
novel methodology to introduce the concept of taste messaging.
They have developed a mobile application to formulate a taste
message while defining a new extensible markup language
(XML) format called the TasteXML (TXML) to communicate
the taste message. However, this work does not have a complete teleportation capability as they were transferring only
the basic taste sensations such as sourness and saltiness.
Another important aspect to consider when teleporting a flavor
sensation is the sensing of the flavors. Several sophisticated
electronic tongue systems have been developed to analyze and
sense the chemical composition of various beverages such as
wine and tea [7, 6]. Most of these taste sensors are developed with multichannel electrodes using lipid membranes as
a transducer of taste substances. Similar to the human gustatory system, these sensors identify taste sensations based
on recognizing response patterns of electric signals that transform information about the available taste substances in a
beverage [20].
Teleporting Lemonade
The method presented in this paper to digitally teleport a
lemonade is primarily inspired by the research works as mentioned above. While these works address various challenges
to simulate or augment taste sensations, they have not been
adapted to the unique requirements of teleporting a beverage.
We have combined some of the approaches mentioned above
into a single piece of technology to develop an interactive
platform that simulates lemonade drinking experiences electronically and remotely.
There is deep history and meaning rooted in the preparation
and serving of beverages during social gatherings. This often
catalyzes conversations and brings people together to celebrate
occasions. With the advent of telepresence and virtual reality
technologies, the prospect of remotely interacting with friends
and family in a social context becomes ever more feasible.
However, the ability to establish a feeling of presence in such
scenarios can be restricted by a lack of co-experiencing other
interactions, such as sharing beverages. Moreover, recent
research works have shown that sharing a taste experience
with another person, even without verbally communicating,
amplifies their taste experiences [1]. Thus, the main motivation
behind our research is to develop a technology that enables
the remote digital sharing of beverages, in-order to enrich our
everyday digital interactions. We are also motivated to study
human multisensory perception when consuming beverages
and hypothesize that the experience of consuming a glass of
lemonade can be effectively simulated by overlaying only the
visual (color) and taste (sourness) sensory information on plain
water.
In this paper, lemonade is selected to demonstrate the concept
of digitally sharing and simulating beverages due to the fol-
lowing reasons: 1) It is a common beverage in most cultures,
2) Sourness (acidity) - the primary taste sensation associated
with lemonades can be readily measured using pH sensors,
and effectively simulated using the electrical stimulation on
the tongue as demonstrated by [14, 13, 8], and 3) Lemonade is
available in numerous colors such as yellow, green, and cloudy
white, which can be detected using an RGB color sensor and
simulated using an RGB LED.
The applications of this work extend not only towards virtual
flavor technologies, but also in entertainment and well-being.
Users may use such devices during social gatherings for multisensory interactions, for example, by toasting drinks users
could share the flavors of their drinks with others. In the future, we envision a cloud repository for people to share digital
signatures of their beverages. This concept may also impact
personal well-being by encouraging people to drink virtually
flavored water rather than artificial soft drinks.
METHOD
This section describes the implementation and technical details of the proposed platform. As explained, this platform
incorporates three modules to teleport a glass of lemonade:
the sensor, actuator, and communication protocol. The main
modules of this platform are illustrated in Figure 1.
The Lemonade Sensor
As displayed in Figure 2, the lemonade sensor has two main
components. It utilizes an RGB color sensor (ISL29125
from SparkFun1 ) and an analog pH sensor (SEN0161 from
DFRobot2 ) to capture the color and sourness of the beverage
respectively. As established in previous related work, it is
important to simulate beverage color accurately as it directly
affects the user’s flavor perceptions. Moreover, at present,
there are hundreds of different artificial lemonade drinks in
the market, available in various colors such as red and blue
apart from the traditional yellow, cloudy, or green variants.
Additionally, a Bluetooth Low Energy (BLE) enabled Arduino
board is used in the control module to handle the Bluetooth
communication.
The pH sensor captures the level of sourness of the lemonade
(measuring the degree of acidity) and correlates this value with
five groups: barely detectable, mild, medium, strong, and very
strong. We have experimentally developed this range by diluting 15ml of squeezed lemon juice with different amounts of
distilled water - barely detectable: 50ml, mild: 30ml, medium:
15ml, strong: 5ml, and very strong: 0ml. To develop this range,
we asked participants to rank the solutions on the above scale
while recording the pH values of the solutions as reported in
Table 1.
As shown in Figure 3, based on the current design, the user
has to submerge the sensor and press the push-button to capture and send the information to the simulator. The sensor is
designed in such a way that it can be easily mounted on a glass
of lemonade during use, and the control module is attached
separately to improve the portability.
1 www.sparkfun.com
2 www.dfrobot.com
Connections
to control module
pH sensor
Push button
Sensor
handle
Group
pH range
barely-detectable
mild
medium
strong
very-strong
2.84 and above
2.52 - 2.84
2.38 - 2.52
2.25 - 2.38
2.17 and below
Magnitude of
current
60µA
80µA
120µA
160µA
180µA
Table 1. pH values are mapped with different magnitudes of current.
sor captures the color and pH information, the control module
formulates the XML message and transfers it to the simulator. An example message formulated from a yellow colored
lemonade with strong sour taste is displayed below:
<lemonade>
<color r=239, g=239, b=31 />
<sourness>strong</sourness>
</lemonade>
The Lemonade Simulator
RGB sensor
Container
Figure 2. Different components of the lemonade sensor. Once submerged, the pH sensor captures the pH value while RGB sensor captures
the color.
Control module with
Bluetooth communication
Submerged
lemonade
sensor
A glass of lemonade
Figure 3. Once the lemonade sensor is submerged, it captures the information of the beverage. The control module sends this information to
the simulator via Bluetooth.
As shown in Figure 4, a customized tumbler has been developed to simulate the lemonade experience on plain water. The
tumbler has an RGB LED with a light diffusing mechanism
at the bottom to provide different color stimuli and a ring
of silver electrodes attached to the mouthpiece to simulate
sourness by electrically stimulating the user’s tongue. Users
simply touch the silver electrodes using their tongues while
drinking to simulate the sourness. To simulate the sour taste,
controlled electrical pulses (frequency: 800Hz, magnitude of
current: 60µA - 180µA) were used [12, 13]. The mapping of
pH values to magnitudes of current is shown in Table 1. Due to
limitations of the human taste system when perceiving small
step changes in taste intensity, and the subjective nature of
taste, one-to-one mapping between pH values and magnitudes
of current was not considered.
When using this tumbler, the users only consume plain water
but the experience of the lemonade flavor is simulated using
the crossmodal effects as mentioned above. In addition to the
RGB LED at the bottom of the tumbler, the control module
has a constant current source to output controlled electrical
pulses and a BLE enabled Arduino board handles the Bluetooth communication and control signals. We also used white
colored diffuse-papers in every tumbler as the plain water is
too clear to diffuse the light properly.
EXPERIMENTAL EVALUATION
We have conducted a user experiment to evaluate the effectiveness of this approach by comparing different aspects of
the real vs. virtual lemonades. Three common lemonade
colors - green, yellow, and cloudy were chosen to compare
the pre-taste and actual taste aspects between real and virtual
lemonades. Our main hypothesis is that the participants will
have a similar degree of flavor experience in real as well as
virtual lemonades because of the crossmodal effects.
Communication Protocol
Participants
Communication between the sensor and simulator is established via Bluetooth and an XML based protocol is utilized to
encode the data as explained in [11]. Once the lemonade sen-
Thirteen (13) participants (5 male and 8 female), aged between
20-30 (AVG = 22.76, M = 22, SD = 2.01), were recruited for
this experiment. Participants were randomly recruited from
Mouth piece with
two silver electrodes
Superimposing
virtual colors
L1
L2
L3
Figure 5. The real lemonades prepared in three commonly available
colors (green, cloudy, and yellow).
RGB LEDs
Control module
Figure 4. Different components of the lemonade simulator. It stimulates
the user’s tongue using electrical stimulation and superimposes virtual
colors onto the water to simulate lemonades
the university staff and students and had no prior experience
with the concept or the prototypes before being tested. All
participants were in good health (without known problems
like cold, fever, or a runny nose), non-smokers, and reported
no taste or smell problems. Furthermore, all of them were
familiar with using technology such as computers and mobile
phones.
In general, certain medical conditions and medications, as well
as consuming spicy and hot food, may temporarily interfere
with the sensitivity of the sense of taste. Therefore, they were
instructed not to consume spicy, alcoholic, too hot or too cold
- food or beverages at least one hour before the experiments as
these may alter their perceptions. The experiments were conducted in the morning between 10 AM - 12 PM and afternoon
between 2 PM - 6.30PM to avoid breakfast and lunch periods.
Furthermore, as some loss of taste sensation also occurs due to
the normal aging process, we selected a particularly focused
age group spanning 20 to 30 years [2].
Apparatus
The experiments were conducted in two quiet private meeting
rooms (with approximately similar ambient light and temperature settings) inside the laboratory. One room was used to
conduct the experiments with the real lemonade (as displayed
in Figure 5) and the other with the virtual lemonade (as shown
in Figure 6). To prepare the three colored lemonade solutions,
we used the Florida’s Natural Lemonade3 (pH = 2.34) with
tasteless food coloring agents. In this way, we preserved the
same taste for all the three lemonades but applied different
colors for each. Furthermore, all lemonades, real and virtual
were presented in vessels that were identical in physical form.
3 https://www.floridasnatural.com/our-juices/lemonade.php
V1
V2
V3
Figure 6. The three commonly available colors of lemonades (green,
cloudy, and yellow) are simulated by overlaying respective colors using
LEDs.
To simulate the virtual sour taste, the lemonade simulator
was configured with the settings related to the strong level
(frequency: 800Hz, magnitude of current: 160µA).
Experimental Method
As mentioned, this experiment was focused on studying the
effectiveness of simulating a virtual lemonade remotely by
capturing and sharing the color and pH levels of a real lemonade in three different colors: yellow, cloudy, and green. In this
study, we mainly focused on studying two aspects of the experience: pre-taste of sourness based on the colors (appearance)
and perception of actual taste sensations.
Before the formal experiment, for each participant, we conducted a short training session (using an empty tumbler with
no real or artificial stimuli) to get them familiar with the apparatus. Then, during the experiment, each participant was
randomly presented with 12 different stimuli in two sessions six stimuli based on real lemonade (two stimuli per lemonade:
color and taste) and similarly six other based on virtual lemonades. The real and virtual lemonades were arranged in two
separate meeting rooms (to counterbalance) to simulate a remote sharing scenario. The experiment was conducted in two
similarly structured separate sessions (one session per room)
Results and Discussion
As presented in Figures 7, 8, and 9, we analyzed the results
of perceiving the sourness based on three aspects: pre-taste
(primarily based on the color), taste of sourness, and as a
combined measurement (perception of sourness based on color
and taste).
Pre-taste of sourness based on the color
Paired-Samples T-Tests were conducted between the real and
virtual lemonade scores for the three colored beverage groups
regarding the pre-perception of sourness. The results suggest
that there is no significant difference in the scores for the pairs
of green L1 - V1 [t(12)=0.519, p > 0.5, SD=0.26, SE=0.07] and
yellow L3 - V3 [t(12)=-1.723, p > 0.1, SD=0.28, SE=0.078] (as
in Figures 5 and 6). This finding suggests that the participants
had a similar preconception of sourness based on green and
yellow beverages. However, there is a significant difference
in the scores for cloudy L2 - V2 [t(12)=-0.208, p < 0.05,
SD=0.408, SE=0.113], which suggests that the participants
perceived a higher intensity of sourness in the virtual lemonade
than the real lemonade based on their colors. This may be
due to the white color of the LED which is much brighter
than the cloudiness in the real lemonade; the participants
might associate higher intensity of sourness because of the
intensity of the color as explained in [17]. The normalized
and averaged results based on the pre-taste of sourness are
displayed in Figure 7.
Perception of sour taste
Similarly, Paired-Samples T-Tests were conducted between
real and virtual lemonade scores for the three colored beverage
groups to compare the participant’s perception of sour taste.
In this phase, visual information of the drink was intentionally
overlapped with its taste information to mimic the normal
setting, wherein the preconception of taste based on visual
information, along with the color and taste of the beverage at
the time of consumption, contribute to the overall flavor experience. However, when the participants taste the sourness of real
and virtual lemonades, as the paired-samples t-tests suggest,
Average pre-perception of sourness
(normalized)
During each session, participants were randomly assigned to
one room (to counterbalance). They were then asked to judge
and rank the lemonades on the level of sourness based on its
color (pre-taste) and next, based on the sourness (after drinking). Three colored lemonades were randomly presented (to
counterbalance) one after another with minimum five-minute
intervals between stimuli during which they were asked to
rinse their mouth with deionized water. Furthermore, similar
tumblers were used for all the stimuli to prevent bias and for
counterbalancing. At the end of the first session, participants
were rested for a minimum of five minutes before the next
session. They were asked to rank the intensity of sourness on
a six-point scale of no-taste, barely-detectable, mild, medium,
strong, and very-strong.
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Green
Real Lemonade
Cloudy
Yellow
Virtual Lemonade
Figure 7. Normalized average intensity scores reported for the preperception of sourness of the three colored lemonades. Error bars represent 95% confidence interval (N = 13).
Average intensity of sourness
(normalized)
for real lemonade and virtual lemonades. The fact that the
beverages in the tumblers were lemonades was not disclosed
to the participants along with the purpose of this experiment.
Before each experiment, mouth-pieces and the electrodes were
rinsed using tap water, then sterilized using 70% isopropyl
alcohol swabs, followed by deionized water [3].
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Green
Cloudy
Real Lemonade
Virtual Lemonade
Yellow
Figure 8. Normalized average intensity scores reported for the sour taste
of the three colored lemonades. Error bars represent 95% confidence
interval (N = 13).
there is a significant difference in the scores for both pairs of
green L1 - V1 [t(12)=2.25, p < 0.05, SD=0.27, SE=0.076] and
cloudy L2 - V2 [t(12)=3.593, p < 0.05, SD=0.308, SE=0.085].
As reported in Figure 8, in both of these occasions the intensity of the real sour taste was much stronger than the virtual
sourness produced by the simulator. We suspect two possible
reasons: 1) The perception of the sourness and its intensity
through electrical stimulation on the tongue may be weakened
because of the plain water or 2) Participants may perceive
mixed sensations, for example, higher sweetness or saltiness
due to the color change and crossmodal effects as also noticed
in [21]. One suggestion to increase the perception - the intensity of virtual sour taste - is to utilize a slightly stronger
electrical pulse, for instance - apply the electrical stimuli as in
very-strong category when the sourness of the real beverage
falls only in the strong category (as in Table 1). Nevertheless,
there is no significant difference in the scores for yellow L3
- V3 [t(12)=0.762, p > 0.05, SD=0.272, SE=0.075], which
confirms that the sour taste perception of yellow colored real
and virtual lemonades are similar.
Overall perception (combined measurement)
Finally, we compare the overall perception of real and virtual
flavor experience (combined measurements of pre-perception
Average intensity of sourness
(combined - normalized)
real and virtual lemonades [t(77)=0.809, p > 0.05, SD=0.349,
SE=0.039]. These results also confirm that, although there
are several aspects we need to study further (for example, the
results of varying perceptions of sour taste in cloudy and green
color lemonades), in general teleportation of lemonades may
be achieved by transferring only the color information and pH
values (which matched to the degree of sourness).
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
LIMITATIONS AND FUTURE WORK
Green
Cloudy
Yellow
Real Lemonade
Virtual Lemonade
Figure 9. Normalized average intensity scores reported for combined
pre-perception and sour taste for the three colored lemonades. Error
bars represent 95% confidence interval (N = 13).
Average intensity of sourness
of all three lemonades
(combined - normalized)
0.8
0.7
0.6
0.5
Designing technologies to share flavor experiences remotely is
challenging, mainly due to the lack of fundamental knowledge
on the multisensory perception of flavor experiences. However,
in this work, we utilized the unique opportunity of crossmodal
perception to simulate the flavor experience of lemonades.
One of the main limitations of this work is that at its present
stage, we are only focused on the primary taste attribute of
lemonades, i.e. the perception of sour sensations. Hence,
future work would concentrate on other sensations including
the sweetness, saltiness as well as bitterness. More importantly,
we intend to study how crossmodal effects might change the
perception of these sensations and their intensities.
Future studies should also focus on more variables such as the
change in color intensities and the properties of electrical stimulation to study their effect on the overall flavor perception
in humans. Moreover, other modalities such as smell, temperature and other properties of beverages such as fizziness
should be included to simulate more varieties and realistic
virtual experiences.
0.4
0.3
0.2
0.1
0
Real Lemonade
Virtual Lemonade
Figure 10. Normalized average intensity scores reported for overall all
the real and virtual lemonades. Error bars represent 95% confidence
interval (N = 13).
of the sourness and the sour taste). As mentioned earlier, we
hypothesize that by simulating only the color and basic taste
information (i.e. sourness) participants will have a similar
degree of flavor experience in virtual lemonades due to the
crossmodal effects. Paired-Samples T-Tests were conducted
between real and virtual lemonades of the three colored beverage groups to compare the participant’s perception of lemonade flavors. Interestingly, the paired-samples t-tests suggest
that there are no significant difference in the scores for green
L1 - V1 [t(25)=1.958, p > 0.05, SD=0.275, SE=0.054], cloudy
L2 - V2 [t(25)=0.324, p > 0.05, SD=0.454, SE=0.089], and
yellow L3 - V3 [t(25)=-0.679, p > 0.05, SD=0.288, SE=0.056]
lemonades when compared in terms of their overall perception
of sourness. Furthermore, as can be seen in Figure 9, the mean
values are approximately equivalent when compared with the
overall flavor perception of the three lemonades.
To further confirm this result, another Paired-Samples T-Test
was conducted between the overall perceptions of all real and
virtual lemonades (combining both factors: pre-perception and
taste of the sourness, as well as the three colored lemonades
- green, cloudy, and yellow). As depicted in Figure 10, the
results show that the overall perception of sourness between
real and virtual lemonades are not statistically significant, thus
in general participants perceive a similar flavor experience in
During the user experiments, most participants found the experience of flavor sharing pleasant and surprising. However, on
several occasions elicited taste sensations and their intensities
were found to be subjective mainly in terms of their intensities. One possible solution to overcome this limitation is to
calibrate the stimuli (intensity of color and properties of the
electrical stimuli) of the lemonade simulation according to
users’ perceptions. This phenomenon is common in natural
(or chemical based) flavor perception as well [22].
CONCLUSION
In conclusion, we presented a novel method to teleport the
flavor experience of a glass of lemonade. The platform consists of three stages: sensing (which captures the RGB color
and the pH value of the beverage), communication (the XML
protocol to encode the data), and simulating (a custom built
tumbler to overlay the color and stimulate the tongue using
electrical pulses to simulate the sourness and control its intensity). Furthermore, we conducted experiments to determine
the effectiveness of this approach and evaluate the crossmodal
effects on flavor perception, which employs the overlaying of
colors and stimulating the tongue to simulate varying degrees
of sourness. The results of this experiment recommend that,
in general, the proposed platform could teleport and simulate lemonades with an approximately similar perception of
sourness when compared to real lemonades.
However, being an exploratory study, this work merits further investigations in multiple directions including studying
of other aspects of flavor experiences to share remotely. More
work is required to be done before introducing a digital platform to share any beverage remotely both in terms of technology as well as usability of this platform. Once introduced, their
interactions could revolutionize the domain of digital interactions. Further experimentations with this work will help us in
developing social food and beverage sharing platforms with
multisensory experiences in the future. We believe that this
research on digitally sharing of lemonades will enable crossdisciplinary, novel, and innovative application possibilities in
the future.
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its International
Research Centres in Singapore Funding Initiative.
REFERENCES
1. Erica J Boothby, Margaret S Clark, and John A Bargh.
2014. Shared experiences are amplified. Psychological
science 25, 12 (2014), 2209–2216.
2. JM Boyce and GR Shone. 2006. Effects of ageing on
smell and taste. Postgraduate medical journal 82, 966
(2006), 239–241.
3. R.J. Dobrin. 1984. Liquid cleaner-disinfectant
composition for use in wiping down dental operatories.
(Aug. 7 1984). US Patent 4,464,293.
4. Cynthia N DuBose, Armand V Cardello, and Owen
Maller. 1980. Effects of Colorants and Flavorants on
Identification, Perceived Flavor Intensity, and Hedonic
Quality of Fruit-Flavored Beverages and Cake. Journal of
Food Science 45, 5 (1980), 1393–1399.
5. Yuki Hashimoto, Naohisa Nagaya, Minoru Kojima,
Satoru Miyajima, Junichiro Ohtaki, Akio Yamamoto,
Tomoyasu Mitani, and Masahiko Inami. 2006. Straw-like
user interface: virtual experience of the sensation of
drinking using a straw. In Proceedings of the
international conference on Advances in computer
entertainment technology. ACM, 50.
6. A Legin, A Rudnitskaya, L Lvova, Yu Vlasov, C
Di Natale, and A D’amico. 2003. Evaluation of Italian
wine by the electronic tongue: recognition, quantitative
analysis and correlation with human sensory perception.
Analytica Chimica Acta 484, 1 (2003), 33–44.
7. Andrey Legin, Alisa Rudnitskaya, Yuri Vlasov, Corrado
Di Natale, Fabrizio Davide, and Arnaldo D’Amico. 1997.
Tasting of beverages using an electronic tongue. Sensors
and Actuators B: Chemical 44, 1 (1997), 291–296.
8. Hiromi Nakamura and Homei Miyashita. 2011.
Augmented gustation using electricity. In Proceedings of
the 2nd Augmented Human International Conference.
ACM, 34.
9. Hiromi Nakamura and Homei Miyashita. 2012.
Development and evaluation of interactive system for
synchronizing electric taste and visual content. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems. ACM, 517–520.
10. Takuji Narumi, Munehiko Sato, Tomohiro Tanikawa, and
Michitaka Hirose. 2010. Evaluating cross-sensory
perception of superimposing virtual color onto real drink:
toward realization of pseudo-gustatory displays. In
Proceedings of the 1st Augmented Human International
Conference. ACM, 18.
11. Nimesha Ranasinghe, Adrian David Cheok, and Ryohei
Nakatsu. 2012. Taste/IP: the sensation of taste for digital
communication. In Proceedings of the 14th ACM
international conference on Multimodal interaction.
ACM, 409–416.
12. Nimesha Ranasinghe, Kuan-Yi Lee, and Ellen Yi-Luen
Do. 2014. FunRasa: an interactive drinking platform. In
Proceedings of the 8th International Conference on
Tangible, Embedded and Embodied Interaction. ACM,
133–136.
13. Nimesha Ranasinghe, Kuan-Yi Lee, Gajan Suthokumar,
and Ellen Yi-Luen Do. 2016. Virtual ingredients for food
and beverages to create immersive taste experiences.
Multimedia Tools and Applications 74, 24 (2016), 1–19.
14. Nimesha Ranasinghe, Ryohei Nakatsu, Hideaki Nii, and
Ponnampalam Gopalakrishnakone. 2012. Tongue
Mounted Interface for Digitally Actuating the Sense of
Taste. In Wearable Computers (ISWC), 2012 16th
International Symposium on. IEEE, 80–87.
15. Charles Spence, Carmel A Levitan, Maya U Shankar, and
Massimiliano Zampini. 2010. Does food color influence
taste and flavor perception in humans? Chemosensory
Perception 3, 1 (2010), 68–84.
16. Charles Spence, Xiaoang Wan, Andy Woods, Carlos
Velasco, Jialin Deng, Jozef Youssef, and Ophelia Deroy.
2015. On tasty colours and colourful tastes? Assessing,
explaining, and utilizing crossmodal correspondences
between colours and basic tastes. Flavour 4, 1 (2015), 1.
17. Jennifer A Stillman. 1993. Color Influences Flavor
Identification in Fruit-flavored Beverages. Journal of
Food Science 58, 4 (1993), 810–812.
18. Chie Suzuki, Takuji Narumi, Tomohiro Tanikawa, and
Michitaka Hirose. 2014. Affecting tumbler: affecting our
flavor perception with thermal feedback. In Proceedings
of the 11th Conference on Advances in Computer
Entertainment Technology. ACM, 19.
19. Kiyoshi Toko. 2000a. Biomimetic sensor technology.
Cambridge University Press.
20. Kiyoshi Toko. 2000b. Taste sensor. Sensors and
Actuators B: Chemical 64, 1 (2000), 205–215.
21. Xiaoang Wan, Andy T Woods, Jasper JF van den Bosch,
Kirsten J McKenzie, Carlos Velasco, and Charles Spence.
2014. Cross-cultural differences in crossmodal
correspondences between basic tastes and visual features.
Frontiers in psychology 5 (2014), 1365.
22. Massimiliano Zampini, Emma Wantling, Nicola Phillips,
and Charles Spence. 2008. Multisensory flavor
perception: Assessing the influence of fruit acids and
color cues on the perception of fruit-flavored beverages.
Food quality and preference 19, 3 (2008), 335–343.