Image act theory
C. J. Reynolds
University of Tokyo
Engineering Building 6, Room 230
7-3-1 Hongo, Bunkyo-ku
Tokyo 113-8656, Japan
Phone: +81 3-5841-6937
Fax: +81 3-5841-6952
[email protected]
Abstract
Image Act Theory is an ethical theory that evaluates image manipulations in terms of social actions.
In doing so, Image Act Theory borrows the approach of Austin's Speech Act Theory. Austin argued
that certain utterances are preformative; that "in saying something, we do something." Likewise,
Image Act Theory argues that when we change the way something appears, we perform a social
action. An image manipulation may accuse, misrepresent, persuade and entertain depending upon
the audience, illustrator, and whom is depicted. The benefit of this theory is that social actions are
already amenable to analysis using a variety of ethical systems. This paper will survey historical
and recent examples of image manipulation, and provide commentary about the social action performed.
Keywords
image manipulation, ethics, speech act theory, social acts, communication
INTRODUCTION
When an image is manipulated and this manipulation is recognized, a social action is performed. To
support this theory, this paper builds from Reinach’s ontology concerning social acts. In doing so
the argument borrows heavily on the analysis of television by Lundsten (1998). Still the theory is
both inspired and named after Austin’s Speech Theory (1975).
SOCIAL ACT THEORY
The ontological view proposed by Reinach distinguishes between physical objects and states of
affairs that exist between them. The states of affairs are used to characterize obtainable states. More
importantly states of affairs also serve as the domain of communicable information. Reinach uses
this division of the world into objects and states of affairs as the basis for description of social acts.
Lundsten describes the Reinachian model as involving “four main characteristics in addition to
the necessary physical expression which is needed because human beings are bound to the physical
world: (1) social acts are directed at an intentional correlate, and (2) they are doings of somebody,
furthermore, (3) they generate changes in the world but, in order to do this, (4) they need to be recognised by a counterpart.” Reinach’s social acts were a precursor to structuralist models of communication such as Jakobson’s analysis of verbal communication (1960).
As we shall see, image manipulations can be examined using this ontological stance. However,
before proceeding it is also enlightening to consider speech act theory as another viewpoint for the
analysis of communication.
SPEECH ACT THEORY
Wittgenstein famously noted that “words are deeds.” In doing so he succinctly describes the main
proposition of speech act theory. This is (to use Austin’s description) that “saying something does
something.”
An oft-used example of speech act theory is the utterance “duck” to mean “look out!” In yelling
the word duck to another English-speaker, I utter a statement but also (if heard and understood)
perform the social action of warning the listener.
Austin’s analysis decomposes speech utterances into different varieties of speech acts (Wilson,
2003):
• locutionary act: The meaning of the statement itself (e.g. saying "step back" is to tell
someone to step back).
• illocutionary act: The contextual function of the act (e.g. by telling someone to step back,
you are warning them of a falling object).
• perlocutionary act: The results of the act upon the listener (e.g. alerting the listener to the
falling object, in the sense that the listener became knowledgeable of the impending danger.
Austin’s work had influence on Searle (1970) who continued the line of analysis to develop a
form of illocutionary logic (1985). However, both Austin’s and Searle’s analyses remained focus on
the problems of philosophy of language. In the next section I will discuss Image Act Theory as a
crude extension of these ideas.
IMAGE ACT THEORY
Image Act Theory’s main proposition is that recognized image manipulations are social actions.
Image manipulations are a subset of Reinach’s social acts. They are:
1. directed at an intentional correlate (in this case an image or representation)
2. the doings of somebody (or some agent)
3. they generate changes in the world (but, in order to do this)
4. they need to be recognised by a counterpart
In this analysis, image manipulations (in order to be effective) must be recognized by another
counterpart. Thus, if a computer program alters images but does not communicate them to another
social actor, then the image manipulations do not have a social action. There are many potential
parties to the social action but at minimum there is a manipulating agent and at least one recipient
of the manipulated media.
Image act theory is a also useful as a starting point for ethical analysis. In transforming a image
manipulation (which is hard to analyze) into a social action (which is already amenable to ethical
analysis) image manipulations are mapped onto familiar domain for moral inquiry. The theory suggests interpreting an image manipulation as an action that might accuse, misrepresent, persuade, or
entertain. These actions and their relationship to the state of affairs in the world help better conceptualize the ethical impact of altering imagery.
HISTORICAL EXAMPLES
In the process of creating images before photography, often painters, illustrators, and engravers
would alter and revise their works. However some of these alterations perform social actions. If an
artist exaggerated the features of a person depicted (thereby caricaturing them) social actions such
as mocking or criticizing may have been performed.
Figure 1: Hogarth’s (1743) demonstration of the difference between characters and caricatures.
Caricatures do not aim to realistically depict a person but to illustrate some aspect of their psychology or character. As such they provide a variety of visual commentary on the person in addition
to simply depicting them. Caricaturists however are, in some sense, meant to be caught in their alterations. Their alterations are exaggerated and do not conceal the action of lampooning. Of particular interest though, are manipulations whose detection is difficult, leading the viewer to form a distorted perspective of the state of the world.
Pictured below are a portrait of Abraham Lincoln and a portrait of John C. Calhoun. The portrait
of Lincoln on the left was produced after his assassination by altering the portrait on the right of
Calhoun (Mitchell, 1992).
Figure 2: a portrait of Lincoln (left) constructed by altering a portrait of Calhoun (right)
As an aside, this is somewhat ironic in that Calhoun was an architect of the breakaway confederacy which Lincoln battled during the U.S. civil war. Mitchell’s The Reconfigured Eye (1992) provides more historical detail:
After Lincoln was assassinated, new pictures of the dead president
were created by pasting his head, from a famous Mathew Brady photograph (the one engraved on the five-dollar bill), onto an appropriately
statesmanlike full-length portrait of Calhoun...Lincoln’s head had to be
mirrored in order to make it fit; the deception was discovered when
somebody noticed that the late president’s highly recognizable mole
was on the wrong side.”
This manipulation along with many others were collected by Farid (2006) as part of work on digital
tampering and computational methods to identify manipulated photos.
The machinery Image Act Theory encourages us to examine the manipulated portrait in terms of
what action was performed. The grafting of Lincoln’s head served onto his political opponent’s
body maps onto the action of misrepresenting. It might be argued that the portrait serves good by
providing a heroic image of a revered political figure or it might be argued that the portrait sets up a
bad state of affairs by misleading historical viewers.
Figure 3: original (left) and manipulated (right) photos of Lenin. Trotsky, who can be seen
standing below Lenin but above the crowd, was removed from the image after his exile.
Photography’s realistic depiction objects in the world provided the possibility of making photographs who misled viewers about the state of affairs in the world. For instance, picture above is a
pair of pictures of a speech by Lenin (Tukh, 2002). Although subtle, one can see that some people
have been removed for the photo on the right. Trotsky, one of the people depicted was labeled as an
enemy of the people and deported in 1929. The action of exiling extended to removing him from
pictures documenting his role in establishing the Soviet Union. Consequently, Trotsky’s representation being excised from photographs mirrored his physical expulsion from the Soviet Union.
Another example of photographic manipulation from the Soviet Union centers around Nikolai
Yezhov (Dillon, 2006). During the purges of Stalinist Russia, he was a member of the NKVD police
forces and briefly worked as commissar of waterways before being accused treason by his sucessor
Beria. Following his execution, as with Trotsky, his images were deleted from photographs. And so
masking his presence in various photographs performs the social action of concealing his involvement and association with members of Stalin’s government.
Figure 4: original (left) and manipulated (right) photos of Stalin.
Nikolai Yezhov was both executed and removed from the photo.
The preceding examples show that in altering images we can perform a variety of social acts. The
coupled execution or exile and erasure of figures from official photography provides a stark example of social acts upon both objects in the world and states of affairs; namely a person and their representation was made to disappear.
MODERN EXAMPLES
Image Act Theory is of relevance to computer ethics because modern computing systems have
made it possible for large numbers of individuals to easily and quickly manipulate imagery. Indeed,
moderns systems are capable of altering both images and video streams in a manner which can be
difficult to detect.
Figure 5: original (left) and manipulated (right) cover photos of O.J. Simpson. The Time cover photograph was altered to appear darker and blurred.
Following his arrest, O.J. Simpson (a celebrity and sportsman) was featured on the covers of two
prominent U.S. news magazines. The photo was composited with headlines, but it in the case of
Time, it was altered to appear darker and more blurry. During a panel discussion on ethics and image manipulation (Carlson et al., 2006) Aude Oliva commented that the psychological effect of this
manipulation is to make Simpson appear more sinister. An analogous social action would be to accuse Simpson of the murder for which he was later tried.
Figure 6: original (left) and altered (right) image of Condoleezza Rice in which lighting and contrast were manipulated to create an unnatural appearance,
In the image above, published in USA Today (2005) Condoleezza Rice’s appearance was manipulated. After the manipulation was discovered the following editor’s note was attached to the article:
The photo of Condoleezza Rice that originally accompanied this story
was altered in a manner that did not meet USA TODAY's editorial
standards. The photo has been replaced by a properly adjusted copy.
Photos published online are routinely cropped for size and adjusted for
brightness and sharpness to optimize their appearance. In this case,
after sharpening the photo for clarity, the editor brightened a portion of
Rice's face, giving her eyes an unnatural appearance. This resulted in a
distortion of the original not in keeping with our editorial standards.
By introducing unusual and disquieting contrasts in lighting the image appears striking and unflattering. One might argue that the action is to demonize the appearance of Rice. It is common to create drawn caricatures of political figures; however digital manipulation allows rapid creation of
photorealistic representations of political figures that can make them appear unnatural in a much
more subtle manner.
Figure 7: a pair of photos (left) composited into a different scene (right) depicting a British soldier
in Basra, Iraq. The above graphic depicting all three images together was designed by Karlsson.
The above picture of a solider in Iraq ran on the front page of the Los Angeles Times (2006). After the photo was discovered to be manipulation, the paper ran an editor’s note apologizing for its
use:
The primary subject of the photo was a British soldier directing Iraqi
civilians to take cover from Iraqi fire on the outskirts of Basra. After
publication, it was noticed that several civilians in the background appear twice. The photographer, Brian Walski, reached by telephone in
southern Iraq, acknowledged that he had used his computer to combine
elements of two photographs, taken moments apart, in order to improve
the composition.
Times policy forbids altering the content of news photographs. Because
of the violation, Walski, a Times photographer since 1998, has been
dismissed from the staff...
In terms of social action outside of “improving the composition” the photographer also altered the
perception of the attitudes of the depicted people toward one another. The man in blue in the center
of the altered image appears to be heeding the instructions of the soldier while in the original photographs the relationship between the soldier and man in blue is less clear.
Figure 8: original (left) and digitally manipulated scene (right) showing Beirut’s suburbs during
The imagery above from the 2006 conflict between Israel and Lebanon and was published on the
Reuters news wire by photographer Adnan Hajj with the following caption: ”Smoke billows from
burning buildings destroyed during an overnight Israeli air raid on Beirut’s suburbs August 5, 2006.
Many buildings were flattened during the attack” (Johnson 2006). The picture was later removed
from circulation and Moira Whittle of Reuters made the following comments:
This represents a serious breach of Reuters' standards and we shall not
be accepting or using pictures taken by him...The photographer has
denied deliberately attempting to manipulate the image, saying that he
was trying to remove dust marks and that he made mistakes due to the
bad lighting conditions he was working under.
The altered photo contains darker and larger quantities of billowing smoke. The social action performed here is less obvious. I would argue the scene had been made to look more harrowing or
grim. This misrepresentation may have been intended to persuade viewers that the situation was
more severe than actuality. In discussions about this photograph, it was pointed out that the use of
digital manipulation might have been rhetorical or an attempt to show a closer representation of the
state of affair as opposed to what was photographed.
Thus far photographic manipulation has primarily been discussed. However, convincing manipulations of moving images are also amenable to analysis with Image Act Theory. A variety of hoax
movies and videos have been created (perhaps in an attempt to gain notoriety or to amuse). These
should be distinguished from special effects in fictions film works that manipulate images but in a
context where such manipulations are assumed and not taken to represent the real world. In contrast, a recent video showing a graffiti artist painting a message on Air Force One is reportedly a
hoax (Kurtzman, 2006).
Figure 9: still from a hoax video that depicts a graffiti artist “tagging” Air Force One.
!"!#$%&!$!'()*##+,!#-.(/-0#&1)/-0#)2!!.,#)"'(,!)1)#3-)#$-4!#31(,#(3%#)/..!))1&!#$%4/0!)5#(,!#!"#$%&
It is interesting to consider the social action performed in this case. Had the act of tagging Air
'%!()*%+),#-!+./0,1#(-!+'(##67778#-'4#(,!#$.,2(3$+.4-*4)$0(*%*-$%4/0!*##+,!#777#$%92,)#:!(3!!'#-#
Force
been29%(%("2!#
performed
in real
life some
might have'!3@#
viewed
it as vandalism
might
)$-00#
)!(#One
%;# 6<=8#
$%/(,#
1$->!)#
(%# )"'(,!)1?!#
29!&1%/)0"#
/')!!'#while
$%/(,#others
.%';1>/9-(1%')*##
describe
it
as
a
political
statement
or
expression.
But
in
the
more
complex
situation
emerges
since
+,!#(9-A!.(%9"#)"'(,!)1)#$%4/0!#)"'(,!)1?!)#)$%%(,#(9-A!.(%91!)#1'#777#)2-.!#;%9#!-.,#/((!9-'.!*##+,!#
the
video
and
associated
website
attempt
to
convince
the
viewer
of
an
action
which
did
not
really
2-9-$!(!9)#%;#(,!#(9-A!.(%9"#)"'(,!)1)#$%4/0!#3!9!#(9-1'!4#-/(%$-(1.-00"#;9%$#(,!#9!.%94!4#.%92/)#/)1'>#
occur.4!).!'(#
And so 0!-9'1'>*#
the action#performed
be both
to deceive
and29%.!))#
to convey
a political
message.
>9-41!'(#
B# 4!(-10!4#might
-..%/'(#
%;# (,!#
-'1$-(1%'#
.-'#
:!# ;%/'4#
1'# C??-(# ($- ,#5#
6DEED8*#
Video manipulation is becoming more convincing. A recent research project, Mary101, provides
“videorealistic”
facial animation. The system creates visual speech animations for arbitrary senF%9#(,!#2/92%)!#%;#(,1)#)(/4"@#(,!#(9-1'1'>#%;#(,!#2-9-$!(!9)#%;#(,!#(9-A!.(%9"#)"'(,!)1)#3-)#$-4!#%'0"#%'#
tences; in/((!9-'.!)#
short, the system
be made
to say%;#anything.
The 3-)#
designers
of this %'#
system
performed
)1'>0!G3%94#
1'# (,!# can
.%92/)*#
# +!)(1'>#
(,!# )")(!$#
2!9;%9$!4#
)1'>0!#
3%94)# -'4#
experiments to see if participants could distinguish between real and synthetic images and if per)!'(!'.!)#'%(#1'.0/4!4#1'#(,!#(9-1'1'>#)!(*##+,!#)"'(,!)1)#3-)#2!9;%9$!4#%'#(,!#$%/(,#9!>1%'@#3,1.,#3-)#
formance
in lip-reading
differed
well. In analyzing
the experiment,
the$%&!$!'(*#
designers conclude
that
(,!'#
2-)(!4# :-.H#
%'(%# %91>1'-0#
;-.!#as)!I/!'.!)#
31(,# '-(/9-0#
,!-4# -'4# !"!#
# +,1)# -'1$-(1%'#
participants “could not tell the synthetic from the real sequences above chance level” (Geiger et al.,
29%.!))#-00%3)#9!-'1$-(1%'#%;#'%&!0#/((!9-'.!)#(,-(#3!9!#'%(#1'.0/4!4#1'#(,!#%91>1'-0#9!.%941'>*#
# 2003).
#
#
#
#
Figure 10: actual (top) rendered (middle) and synthetic (bottom) images created by Mary 101.
#
!"#$%&' (5# # 6(,#- ,)'- *4)$0($%3- %!,7(*- +8- 9,.4- :;:5- - <+/=- >- .(,#- %!,7(&*(?"()3(- @%$0- ),$".,#- 0(,'- ,)'- (4(!+A(!()$5- - 9%''#(=- >- *(?"()3(- 7()(.,$('- 8.+!- +".- ,)%!,$%+)- *4*$(!B- @%$0- $0(- '(*%.('- !+"$0- !+A(!()$- ,)',//.+/.%,$(-!,*C%)75--D+$$+!=-<0(-8%),#-3+!/+*%$('-*4)$0($%3-%!,7(&*(?"()3(-@%$0-$0(-'(*%.('-!+"$0-!+A(!()$B-1"$@%$0-$0(-),$".,#-0(,'-,)'-(4(-!+A(!()$*-+8-$0(-1,3C7.+")'-*(?"()3(5#
#
#
)*&'+&,&%-.'/00%1-2*3'
#
By allowing the creation of altered videos which are imperceptibly different from actual videos, a
variety of social actions can be performed. A person can be made to convincingly say anything that
the video creator desires. A system might cause a representation of a person to say something contrary to what the person believes or says. The representation becomes a convincing simulation of
the real person as was argued by Baudrillard (1995).
Figure 11: Professor Hiroshi Ishiguro (standing right) shown with Repliee Q1 Expo (seated left in
both pictures)
Indeed, image manipulation in the future may not be confined to photography and video. Recent
work in robotics has sought to create visually convincing androids (Chamberlain, 2005). Picture
above is a robotic system created to appear as natural and life-like as is possible. Its creator, Professor Hiroshi Ishiguro of Osaka University has also created an android clone of himself. It may be
that convincing body-double robots will manipulate perceptions about the state of the world in the
manner that pictures and video already have.
A PICTURE SPEAKS A THOUSAND HALF TRUTHS
Building for the ideas of Austin and Reinach this paper seeks to define and demonstrate Image Act
Theory. As the ability to alter representations of the state of affairs around us becomes increasingly
common our understanding of the ethical implications of these actions ought to be cultivated. This
paper seeks to convince that image manipulations are not simply visual alterations (such as altering
heightening contrast, or compositing) but are linked to social alterations (defaming, deceiving, exiling). Such social actions can be mediated by a variety of visual channels like pictures, videos, or
even potentially robotic displays. Viewed from a different perspective and taken together these manipulations perform an action; they cause us to be more wary of representations, regardless of how
real they appear.
ACKNOWLEDGEMENTS
The author would like to thank Michael Ananny and Gaurav Sood who provided helpful early feedback to a presentation of this material. Alvaro Cassinelli took part in an initial discussion that led to
the notion of an image act theory. Masatoshi Ishikawa has also facilitated and encouraged work in
applied ethics. This research is partly supported by Special Coordination Funds for Promoting Science and Technology, "IRT Foundation to Support Man and Aging Society."
REFERENCES
Austin, J.L. (1975) How to do things with words. Harvard University Press, Cambridge, Massachusetts, USA.
Baudrillard, J (1995) Simulacra and Simulation. University of Michigan Press.
Carlson, K., DeLevie, B., and Oliva, A. (2006) Ethics in image manipulation In ACM SIGGRAPH
2006 Panels (Boston, Massachusetts, July 30 - August 03, 2006). SIGGRAPH '06. ACM Press,
New York, NY.
http://doi.acm.org/10.1145/1179171.1179176
Chamberlain, T. (2005) Ultra-Lifelike Robot Debuts in Japan
http://news.nationalgeographic.com/news/2005/06/0610_050610_robot.html
Dillon, B. (2006) The revelation of erasure. In: Tate, etc. Issue 8, Autumn 2006
http://www.tate.org.uk/tateetc/issue8/erasurerevelation.htm
Farid, H. (2006) Digital tampering in the
http://www.cs.dartmouth.edu/farid/research/digitaltampering/
Media,
Politics
and
Law.
Geiger, G., Ezzat, T., and Poggio, T. (2003) Perceptual Evaluation of Video-realistic Speech. CBCL
Paper #224/ AI Memo #2003-003, Massachusetts Institute of Technology, Cambridge, MA.
http://cerboli.mit.edu:8000/publications/AIM-2003-003.pdf
Los Angeles Times (2006) Editor’s note, March 31, 2006.
http://www.sree.net/teaching/lateditors.html
Lundsten, L. (1998) Towards a Reinachian Ontology of Television. Conference on Applied Ontology, April 24 - 25, 1998, Buffalo, New York, USA.
Jakobson, R. (1960) Closing Statement: Linguistics and Poetics. In Sebeok, A. (ed.) Style in Language. Cambridge, Mass.: The MIT Press.
Johnson, C. (2006) Reuters Doctoring Photos from Beirut?
http://littlegreenfootballs.com/weblog/?entry=21956
Kurtzman, D. (2006) Air Force One Graffiti Hoax Video.
http://politicalhumor.about.com/od/funnyvideos/v/airforceone.htm
Mitchell, W. J. (1992) The Reconfigured Eye: Visual Truth in the Post-Photographic Era. The MIT
Press, Cambridge, Massachusetts, USA.
Reinach, Adolf. (1989) Sämtliche Werke: Kritische Ausgabe mit Kommentar. B. Smith & K.
Schuhmann (eds.). Munich: Philosophia Verlag.
Searle, J.R. (1970) Speech Acts. Cambridge University Press, Cambridge, UK.
Searle, J.R. & Vanderveken, D. (1985) Foundations of Illocutionary Logic. Cambridge University
Press, Cambridge, UK.
Tukh, O. (2002) Falsification of History.
http://www.tc.umn.edu/~hick0088/classes/csci_2101/false.html
USA Today (2005) Rice won't rule out U.S. troops in Iraq in 10 years, October 26, 2005.
http://www.usatoday.com/news/washington/2005-10-19-rice-congress_x.htm
Wilson, G. (2003) Notes: How to Do Things with Words.
http://www.garretwilson.com/books/reviews/howtodothingswithwords.html
BIOGRAPHY
Carson Reynolds is a project assistant professor in the Department of Creative Informatics of the
University of Tokyo. He holds a Doctor of Philosophy and Master of Science from the Massachusetts Institute of Technology upon recommendation by the Program in Media Arts and Sciences in
the School of Architecture and Planning. His research work there was performed at the Media
Laboratory in the Affective Computing Group. Carson also holds a Bachelor of Science in Technical Communication with a Minor in Philosophy from the University of Washington at Seattle.