- WP8 workshop, Trento, 17/18 Nov.
Design of a Pilot Experiment for
the Evaluation of Automated
Persuasion systems
Marco Guerini (ITC-Irst)
Shlomo Hareli (University of Haifa)
Shy Tzafrir (University of Haifa)
- WP8 workshop, Trento, 17/18 Nov.
General Motivations
Persuasive interfaces need to be
evaluated: they have to be effective with
real users.
Evaluation studies up to now have focused
mainly on usability.
Evaluation of effectiveness is not
straightforward, specific methodologies
have to be used / defined.
2
- WP8 workshop, Trento, 17/18 Nov.
Approach
Use of experiments/measures
mainly from Social Psy.
coming
Specific Question: Can display of emotion
enhance the persuasive force of a
message in HCI?
3
- WP8 workshop, Trento, 17/18 Nov.
Specific Motivations
2 specific interests:
Automated
persuasive
communication:
evaluation study of the Promoter Prototype,
developed at Irst.
Emotions in persuasive comm.: study of the
Fitting Principle (Univ. of Haifa) in persuasive
Computer-Human Communication
4
- WP8 workshop, Trento, 17/18 Nov.
Promoter prototype
 Content Ordering Meta-Strategies (COMS). Given basic
persuasive strategies, how to compose them to create a
complex message.
 Example of basic persuasive strategy on positive
consequences: “action a has the positive outcome o”.
 COMS1 a message made with positive pieces of information. As
a pilot experiment we will start from this one.
 COMS2 based on framing effect theory (positive consequences
of the required action following negative consequences of a
conflicting action)
 etc…
5
- WP8 workshop, Trento, 17/18 Nov.
The Fitting Principle
The theory we use for emotion displaying
is the Fitting principle (University of
Haifa):
 A message that is accompanied by an
emotional expression that fits the valance of the
message will be more persuasive than the
same message emitted with no emotion or an
emotional expression that contradicts the
message.
6
- WP8 workshop, Trento, 17/18 Nov.
Advantages
 an evaluation study of the specific persuasion
system (Promoter)
 a contribution to the design of a general
model for evaluation of automated persuasion
systems
7
- WP8 workshop, Trento, 17/18 Nov.
Conditions of the Experiment
The experiment will study one persuasive
meta-strategy (i.e. COMS1) on 3 different
conditions (between subjects, each person
gets only one condition). 3 conditions are:
Fitting emotion
Unfitting emotion
Neutral, no emotion, just message
8
- WP8 workshop, Trento, 17/18 Nov.
The message
You should come and visit the renewed
exhibition on the archaeology of the Land of
Israel at the Hecht Museum. It has pieces
from the Chalcolithic, through the Hellenistic,
Canaanite up to the Byzantine Period. This is
a good opportunity to enlarge your knowledge
about Israel History, to learn nice anecdotes
to tell your friends and it is really easy to
reach, since it is inside your own university!
9
- WP8 workshop, Trento, 17/18 Nov.
Fitting Clip
10
- WP8 workshop, Trento, 17/18 Nov.
Unfitting Clip
11
- WP8 workshop, Trento, 17/18 Nov.
Neutral Clip
12
- WP8 workshop, Trento, 17/18 Nov.
Expected Results
The Fitting message is more persuasive
than the other messages. The unfitting
message is less persuasive relative to the
other messages.
13
- WP8 workshop, Trento, 17/18 Nov.
Conclusions (1)
Evaluation of Persuasive interfaces:
specific methodologies have to be used /
defined.
Design of an experiment using measures
coming from Social Psy.
14
- WP8 workshop, Trento, 17/18 Nov.
Conclusions (2)
Specific question: Can display of emotion
enhance the persuasive force of a
message in HCC?
Advantages: evaluation study of the
specific system (Promoter) & contribution
for a general model for evaluation of
automated persuasion systems
15
- WP8 workshop, Trento, 17/18 Nov.
Future work (1)
Preamble: We can expect, from studies in
HHI and HCI, that each persuasive
strategy alone works (COMS, fitting princ.)
Broader result: not just the replication of
studies already done.
 Test the interaction of different strategies:
there is no results about their mixing.
16
- WP8 workshop, Trento, 17/18 Nov.
Future Work (2)
Promoter uses four main content-ordering
persuasive strategies: the experiment
replicated conceptually three times, one
for each strategy (after COMS1, also
COMS2, COMS3, COMS4).
Study the interaction between
principle and COMS strategies.
fitting
17
- WP8 workshop, Trento, 17/18 Nov.
Ethical Issues
 Challenge 1: agents that are ethically aware (i.e.
capable of reasoning on the ethicality of their
actions and possibly to correct their behaviour)
 Challenge 2: a “layered” model of ethical
persuasion: ethicality of the act of persuading +
ethicality of the induced action
 Ethical capabilities to be encoded at the metareasoning level
18