SceneMaker:
Multimodal Visualisation
of Natural Language Film Scripts
Eva Hanser
Prof. Paul Mc Kevitt
Dr. Tom Lunney
Dr. Joan Condell
Dr. Minhua Eunice Ma
School of Computing & Intelligent Systems
Faculty of Computing & Engineering
University of Ulster, Magee, Northern Ireland
School of Computing and Mathematics
Faculty of Business, Computing and Law
University of Derby, England
[email protected],
{p.mckevitt, tf.lunney, j.condell}@ulster.ac.uk
[email protected]
PRESENTATION OUTLINE
Aims & Objectives
Related Projects
SceneMaker Design and Implementation
Relation to Other Work
Conclusion and Future Work
: AIMS & OBJECTIVES
AIMS
Input: Screenplay
SceneMaker
System
Output: Animation
• Automatically generate well-designed and affective
virtual scenes from screenplays
• Realistic visualisation of emotional aspects
• Multimodal representation with
3D animation, speech, audio and cinematography
• Enhance believability of virtual actors
and scene presentation
: AIMS & OBJECTIVES
OBJECTIVES
• Processing/inferencing emotions and semantic
information within story context
• Common sense, affective and cinematic knowledge
ontologies reflecting human cognitive reasoning rules
• Automatic genre recognition from text
• Design, implementation and evaluation
of SceneMaker
: RELATED PROJECTS
SEMANTIC TEXT PROCESSING
INT. M.I.T. HALLWAY -- NIGHT
Lambeau and Tom come around a corner.
His P.O.V. reveals a figure in
silhouette blazing through the proof on
the chalkboard. There is a mop and a
bucket beside him. As Lambeau draws
closer, reveal that the figure is Will,
in his janitor's uniform. There is a
look of intense concentration in his
eyes.
LAMBEAU
Excuse me!
WILL
Oh, I'm sorry.
LAMBEAU
What're you doing?
WILL
(walking away)
I'm sorry.
Screenplay Extract from ‘Good Will Hunting (1997)’
• Standardized format
and language of
screenplays
• Automatic annotation
of formal screenplay
elements (Jhala 2008)
• Semantic information
on location, timing,
props, actors, events,
manners, dialogue and
camera direction
: RELATED PROJECTS
VISUAL AND EMOTIONAL SCRIPTING
• Emotion recognition from text:
keyword spotting, lexical affinity, statistical models, fuzzy logic rules,
machine learning, commonsense knowledge, cognitive models
• XML-based annotations
defining visual appearance of animated characters and scenes:
BEAT – Behaviour Expression Animation Toolkit
(Cassell et al. 2001)
MSML – Movie Script Markup Language
(Van Rijsselbergen et al. 2009)
<GAZE word=1 time=0.0 spec=AWAY_FROM_HEARER>
<GAZE word=3 time=0.517 spec=TOWARDS_HEARER>
<R_GESTURE_START word=3 time=0.517 spec=BEAT>
<EYEBROWS_START word=3 time=0.517>
: RELATED PROJECTS
MODELLING AFFECTIVE BEHAVIOUR
• Automatic physical transformation and
synchronisation of 3D models reflecting emotion
• Manner influences intensity, scale, force, fluency
and timing of an action
• Multimodal annotated affective video or
motion captured data (Gunes and Piccardi 2006)
Greta
Personality & Emotion Engine
(Pelachaud 2005)
(Su et al. 2007)
: RELATED PROJECTS
VISUALISING 3D SCENES
• WordsEye – Scene composition
(Coyne and Sproat 2001)
• ScriptViz – Screenplay visualisation
(Liu and Leung 2006)
• CONFUCIUS – Action, speech & scene animation
(Ma 2006)
• CAMEO – Cinematic and genre visualisation
(Shim and Kang 2008)
WordsEye
ScriptViz
CONFUCIUS
CAMEO
: RELATED PROJECTS
AUDIO GENERATION
• Emotional speech synthesis (Schröder 2001)
- Prosody rules
• Music recommendation systems
- Categorisation of rhythm, chords, tempo,
melody, loudness and tonality
- Sad or happy music and genre membership (Cano et al. 2005)
- Associations between emotions and music (Kuo et al. 2005)
: DESIGN AND IMPLEMENTATION
KEYOBJECTIVES
• Context consideration through natural language
processing, common sense knowledge and
reasoning methods
• Extract genre and moods from screenplays
• Influence on all elements of visualisation
• Enhance naturalism and believability
• Text-to-animation software prototype, SceneMaker
: DESIGN AND IMPLEMENTATION
ARCHITECTURE OF SCENEMAKER
Screenplay
Script
Editor
Text &
Language
Processing
Animatio
n Player
Context
Interpretation
Genre
}
Emotio
n
Action
}
Multimedia
Generation
: DESIGN AND IMPLEMENTATION
SOFTWARE AND TOOLS
Natural Language Processing
& Script Segmentation
Movie
Script
Gate
(1)
ANNIE
Onto-Gazetteer
Genre
Movie
Ontology Ontology
Context + Emotion
Reasoning
Event
Synchronisation
3D Rendering +
Multimedia
Concept
Net(3)
MSML(5)
Unity(6)
/SMIL
3D Engine
(JavaScript,XML)
Common Sense
Knowledge
WordNetAffect(4)
Automatic
Sound & Music
Selection
3D Models
(3D Studio Max)
RDFS/OWL
RDFS/OWL
Script
Format
Ontology
LVSR(2)
Festival(7)
Lexical Visual
Semantic
Representation
Speech
Synthesiser
(1) http://gate.ac.uk (2) Ma 2006 (3) Liu and Singh 2004 (4) Strapparava and Valitutti 2004
(5) Van Rijsselbergen et al. 2009 (6) http://unity3d.com (7) http://www.cstr.ed.ac.uk/projects/festival
: DESIGN AND IMPLEMENTATION
EVALUATION OF SCENEMAKER
Evaluating 4 aspects of SceneMaker:
Aspect
Evaluation
Correctness of
screenplay analysis &
visual interpretation
Hand-animating scenes
Effectiveness of
output scenes
Existing feature film
scenes
Suitability for genre type
Scenes of unknown
scripts categorised by
readers
Functionality of interface
Testing with drama
students and directors
RELATION TO OTHER WORK
Text to Animation System
Year
CONFUCIUS (Ma 2006)
ScriptViz (Liu and Leung 2006)
CAMEO (Shim and Kang 2008)
P&E Engine (Su et al. 2007)
2006
2007
2007
2007
Text Input:
Genre Context Emotion Animation
Movie Script
Reasoning
(3D )
–
–
–
–

–
–
–
–



–


–
–
P&E rules


2007
– (Dialogue)
–

–

2009


–

–

–

external

Behaviour Generation System
(Breitfuss et al. 2007)
MSML (Van Rijsselbergen et al.
2009)
SceneMaker
CONCLUSION AND FUTURE WORK
•
Automatic expressive multi-media animation of screenplays
•
Focus on:
– automatic reasoning about story context
and emotional interpretation
– based on world knowledge and context memory
– emotions influencing scene compositions and event execution
– scene direction refined by genre-specifics
•
Analysis of script format to access semantic information
•
Automatic genre specification from script
•
Heightened expressiveness, naturalness and artistic quality
•
Assist directors, actors, drama students, script writers
•
Future work: Implementation & Testing of SceneMaker
Thank you.
QUESTIONS OR
COMMENTS?