The Art and Science of Depiction
1
The Art and
Science of
Depiction
Frédo Durand
MIT
Lab for Computer Science
1
From CG and geometry…
e
v
Q
P
Arc of the skeleton
Visual event ev
Q
P
Array indexed by the polygons
The Art and Science of Depiction
Search tree
3
… to make-up and swimming-suits
The Art and Science of Depiction
4
2
Overview of the talk
• No computer-graphics research
• Preliminary and not-so-preliminary reflections
about images
• 1st part:
– Context and goals
(Raising expectations)
• 2nd part:
– Overview of the class
The Art and science of depiction
– (Failing the expectations)
The Art and Science of Depiction
5
Motivations: Post-PhD blues…
• Why do our image lack aesthetic?
• What’s our goal?
• Where Do We Come From? What Are We?
Where Are We Going?
The Art and Science of Depiction
6
3
Motivations
•
•
•
•
What is “Realism”? What is “Photorealism”?
Are photographs realistic?
Are photographs photorealistic?
What is Non-Photorealistic Rendering?
The Art and Science of Depiction
7
Non-Photorealistic Rendering
•
•
•
•
•
A variety of awesome techniques and solutions
But what are the issues?
Difficulty of classification
Each paper deals with several problems
Lack of inter-operability
The Art and Science of Depiction
8
4
Why make images?
•
•
•
•
•
•
•
•
•
Educational
Tell story
Simulation
Design
Sign
Guide task
Visualization
Search
Analysis
•
•
•
•
•
•
•
•
•
Create shape
Expression
Beauty
Shock
Medical
Humor
Faith
Prevention
Etc.
• Not one single class of images
• Thus, there may be many ways to make images
• CG focuses too much on one of them
The Art and Science of Depiction
9
Non-realism vs. realism
• Non-realism is MORE than degraded realism
– E.g. clarity, selection, abstraction, etc.
The Art and Science of Depiction
10
5
Realism vs. realism
• A realistic image can be MORE than realistic
• E.g. dodging and burning
– During the print
– Locally darken or lighten using a mask
The Art and Science of Depiction
11
Dodging and Burning
• Ansel Adams
• Clearing Winter Storm
The Art and Science of Depiction
12
6
Generic pictorial issues
• A lot of issues are universal
• E.g. oil painting / photograph
Titian
The Art and Science of Depiction
13
Generic pictorial issues
• Contrast is reinforced at the occlusion silhouette
• Tone modification / haze
Titian
The Art and Science of Depiction
14
7
Computer Graphics Imagery
•
•
•
•
Rendering is efficient
Hardware is fast
3D content creation becomes the bottleneck
Most CG images are still not very compelling
The Art and Science of Depiction
15
The one-way pipeline
• Rendering pipeline, rendering equation
• From 3D model to image
• No feedback
3D geometry
Material attributes
Image
Light simulation
Light sources
Viewpoint
Projection
Rasterization, etc.
The Art and Science of Depiction
16
8
Feedback and Darwinian selection
• Picture production is a trial and error process
• The artist tries pictorial techniques, constantly
judges the current state of the picture and reacts
accordingly
The Art and Science of Depiction
The Bull by Picasso
17
What can we do?
• Simulate the feedback
– Optimization approaches
– Perception/artistic-based “metric”
– What are the goals?
• Bypass the feedback
– What are the pictorial issues?
– What are the pictorial techniques?
– Hopefully inverse the feedback
• Try to understand what is going
on and what we’re trying to do!
The Art and Science of Depiction
18
9
What and whom for?
• Trained image makers
– Understand what they need
– Provide more relevant tool
• Image-dummies
– Automatic and semi-automatic
– E.g. “gorgeous image” for CAD
– E.g. “digital photo beautifier”
• Computers (100% automatic)
– E.g. can we transfer the art and
craft of cinema into games?
The Art and Science of Depiction
19
Can artists tell us?
• Well, a lot of translation is necessary
• Their feedback loop is extremely good
• But it is often an inaccessible black box
The Art and Science of Depiction
20
10
Can art historians tell us?
•
•
•
•
•
Well, a lot of translation is necessary
They know what is good in an image
Their language is descriptive, not generative
Often limited to 14th-19th century Western art
But recent art history works with general images
The Art and Science of Depiction
21
Can perception scientists tell us?
• Well, a lot of translation is necessary
• Experimental results
– E.g. colors, perspective perception
• Provide framework to pictorial techniques
– Corresponding human visual mechanisms
– Unfortunately, often no quantitative prediction
– Suggests relevant “dimensions” or issues
• Unfortunately complex images are…
too complex
The Art and Science of Depiction
22
11
Can perception & art history tell us?
• Well, some translation is necessary
• But very fruitful
• That is the topic of this talk!
The Art and Science of Depiction
23
Can we tell them something?
•
•
•
•
Well, a lot of translation is necessary
Raise interesting questions
Generative approach to pictures
Systematic validation
– We cannot omit anything (you get what you ask for)
– Parallel: computer-vision showed the complexity of
human vision
The Art and Science of Depiction
24
12
Towards picture sciences ?
• Perception
• Art History
• Visual Arts
• Computational Vision
• Computer Graphics
• A paradigm: Language sciences
The Art and Science of Depiction
25
The Art and Science of Depiction
• Graduate class at MIT (but undergrads as well)
• Multidisciplinary
• Students from Architecture, Computer Science,
Cognitive Sciences, Media Art & Science
The Art and Science of Depiction
26
13
Other goals
• Multidisciplinary, make connections
• Different viewpoint on picture
• Correct scientific errors in art books
• Excuse to talk
about cool stuff
• Excuse to look
at cool pictures
The Art and Science of Depiction
27
Disclaimer & epistemology
• What will be described are
connections, correlations,
NOT truths, causalities,
explanations
• Moreover, they are often
hypothesis not yet verified
• But we hope they offer
insights
The Art and Science of Depiction
28
14
Plan
• Visual system and art
• Limitations of medium:
compensation
and accentuation
• Representation system
The Art and Science of Depiction
29
Beware of the El-Greco Fallacy
• El-Greco, elongated characters
• Were supposed due to astigmatism
• However, pictures and real people
would have been stretched equally
• Almost as fallacious as assuming
painting should be inverted because
our eyes invert what we see
The Art and Science of Depiction
30
15
Vision as information processing
• Input: retinal image
• Output: 3D layout, object recognition, etc.
Retinal
image
Processing
Intermediate
Data
Processing
Scene
understanding
The Art and Science of Depiction
31
Computational theory of vision
•
•
•
•
Marr’s stages (extended by Palmer et al.)
Human and Computer Vision
Classification of different kinds of processes
Has proved fruitful in art studies
View-centered
Extrinsic
The Art and Science of Depiction
Object-centered
Intrinsic
32
16
Retinal image
• Intensity
Cup
The Art and Science of Depiction
33
Image-based (primary sketch)
• Contrast, edge detection
Cup
The Art and Science of Depiction
34
17
Surface-based
• Visible surfaces, organization
• Distance, orientation
Cup
Local orientation
The Art and Science of Depiction
35
Object-based
• 3D properties, structure
• Nature of the description highly discussed
Cup
The Art and Science of Depiction
36
18
Category-based
• Recognition, category, function
Cup
The Art and Science of Depiction
37
Feedback
• Bottom-up and top-bottom
Cup
The Art and Science of Depiction
38
19
Evolution of children’s drawings
• First draw what they
know (object-based)
• Then what they see
(towards retinal)
• Asked to draw a table
Class of drawing
& average age
7.4
9.7
11.9
13.6
14.3
13.7
Child’s view
The Art and Science of Depiction
39
3D and 2D attributes
• Show a dice to children (~6-7)
• They usually draw a rectangle
• The rectangle could stand for one face
The Art and Science of Depiction
40
20
3D and 2D attributes
• Show coloured or numbered dice to children (6-7)
• The still draw a rectangle
• But different colours or many points
The Art and Science of Depiction
41
3D and 2D attributes
•
•
•
•
•
Show coloured or numbered dice to children (6-7)
The still draw a rectangle
But different colours or many points
The rectangle stands for the whole dice
The notion of 3D object with corners is translated
as a 2D object with corners
The Art and Science of Depiction
42
21
Relation to pictures
• Different classes of pictures for different stages
• Not a strict classification, not a cultural judgment
View-centered
Extrinsic
Object-centered
Intrinsic
The Art and Science of Depiction
43
Relation to pictures
• Chinese painting refuse extrinsic, only essential
• No shadow
View-centered
Extrinsic
The Art and Science of Depiction
Object-centered
Intrinsic
44
22
Retinal image
• Impressionism
• Photography
The Art and Science of Depiction
45
Retinal image
• Impressionism
• Not so simply classified
The Art and Science of Depiction
46
23
Image-based
• Line Drawing
The Art and Science of Depiction
47
Intermediate
• View-based
• Cues for surface-based
feature extraction
are enhanced
– Depth cues
– Orientation cues
• No subjective feature
(e.g. lighting)
The Art and Science of Depiction
48
24
Higher level
•
•
•
•
Primitive art
Cubism
Schema
“What I know”
The Art and Science of Depiction
49
Higher level
•
•
•
•
Primitive art
Cubism
Schema
“What I know”
The Art and Science of Depiction
50
25
Expressionism
• “What I feel”
Other
mode
The Art and Science of Depiction
51
Gaze movement
• We need to align the fovea with relevant features
The Art and Science of Depiction
52
26
Gaze movement
• We need to align the fovea with relevant features
• Saccadic exploration
– Path
– Fixation time
The Art and Science of Depiction
53
Gaze attraction & focal zone
• E.g. Through contrast
The Art and Science of Depiction
54
27
One focal zone
• Arthus-Bertrand
• Striking but…
The Art and Science of Depiction
55
Two focal zones
• Robert Mapplethorpe
Self-portrait, 1988
• More dynamic
The Art and Science of Depiction
56
28
Triple focus and subject gaze
• Robert Doisneau
Les Gosses
de la place
Hebert
• More lively
The Art and Science of Depiction
57
Turner’s Loire journey
• The gaze follows the journey
The Art and Science of Depiction
58
29
Focal point conflict
• Bottom-up is different
from top down
• Makes image dynamic
The Art and Science of Depiction
59
Fixation time & style
• Depends on style “complexity”
• Shorter fixation for more complex style
The Art and Science of Depiction
60
30
Advertisement and focal points
• Evolution of saliency
The Art and Science of Depiction
61
Plan
• Visual system and art
• Limitations of medium:
compensation
and accentuation
• Representation system
The Art and Science of Depiction
62
31
Limitations of the medium
•
•
•
•
•
Flatness
Finite size, frame
Unique viewpoint
Static
Contrast and gamut
• Can be eliminated
• Can be compensated
• Can be accentuated
The Art and Science of Depiction
63
Eliminating flatness
• Stereo display
The Art and Science of Depiction
64
32
Enhancing depth through contrast
The Art and Science of Depiction
65
Accentuating flatness
• Monet
• Occlusion
boundaries
are barely
visible
• Retinal
stage rather
than surface
The Art and Science of Depiction
66
33
Accentuating – dissonance
• Magritte
• Occlusions
are reversed
The Art and Science of Depiction
67
The contrast is limited
• Real world: 10-6 to 106
• Picture: 1 to 50, 1 to 300 at best
10-6
106
High dynamic range
10-6
106
Low contrast
The Art and Science of Depiction
68
34
Low contrast is also an advantage
• W. Eugene Smith photo of Albert Schweitzer
• 5 days to print!
• Things can be related because the intensity is
more similar
• Balance,
composition
The Art and Science of Depiction
69
The image is static – Motion Blur
The Art and Science of Depiction
70
35
Multiple Snapshots
• Marcel Duchamp
Nude Descending
a Staircase
1912
The Art and Science of Depiction
71
Static image & composition
• Tak Kwong Chan
The Horse –
Away He Goes 1980
• Both static and dynamic
qualities :
the limitation
is a richness
The Art and Science of Depiction
72
36
Plan
• Visual system and art
• Limitations of medium:
compensation
and accentuation
• Representation system
The Art and Science of Depiction
73
Plan
• Visual system and visual arts
• Limitations of medium:
compensation
and accentuation
• Representation system
The Art and Science of Depiction
74
37
Representation systems: goals
•
•
•
•
Coarse-grain description of style
Independent systems, independent decisions
Description and comparison
Potentially generative (for NPR)
The Art and Science of Depiction
75
A paradigm: Cartography
•
•
•
•
A map is a depiction of a reality
Can be more efficient than photo
There is no perfect universal solution
The systems are usually explicit!
The Art and Science of Depiction
76
38
Map making
• Which information
will be represented
• Projection
• Which kind of
symbols will be used
• Colour codes
The Art and Science of Depiction
77
Projection
The Art and Science of Depiction
78
39
Projection: Topological
• Beck’s map of London underground, 1931
The Art and Science of Depiction
79
Projection: Topographical
• London underground
The Art and Science of Depiction
80
40
“Metaphoric” Projection
The Art and Science of Depiction
81
Error:Columbus's map
The Art and Science of Depiction
82
41
Error/choice/distortion
• Descelier, 1546
The Art and Science of Depiction
83
Error/choice/distortion
• Descelier,
1546
• E.g. huge
elephant
The Art and Science of Depiction
84
42
Denotation (Primitive & Dimensions)
The Art and Science of Depiction
85
Denotation (Primitive & Dimensions)
The Art and Science of Depiction
86
43
Colour system
• E.g. elevation
The Art and Science of Depiction
87
Representation system
• Drawing
• Denotation
• Tone & color
The Art and Science of Depiction
88
44
Final conclusions and future work
• What the hell can we do with all this?
The Art and Science of Depiction
89
To Be Continued…
The Art and Science of Depiction
90
45
Representation systems
• Drawing and projection
• Denotation
• Tone & colour
• The two first systems are classical
• Because painting was the only recognized art
form
The Art and Science of Depiction
91
Drawing system
• Linear perspective
• Orthographic
• Topological
• Other
The Art and Science of Depiction
92
46
Denotation system
• Silhouette: 2D (regions)
• Line Drawing: 1D (lines)
• Optical: 0D (points)
The Art and Science of Depiction
93
Tone & colour system
• Extrinsic (outgoing light)
• Intrinsic (reflectance properties)
• Other, e.g. symbolic
The Art and Science of Depiction
94
47
Marks vs. primitive
• Mosaic
• Primitives = lines
• Marks = points
(or small regions)
The Art and Science of Depiction
95
Marks vs. primitive
• Paul Siemsen
Picasso
The Art and Science of Depiction
96
48
Style
• Coarse-grain style
– Different categories of drawing, denotation, tone
• Finer-grain
• Local style
• Parameterization
• Capture
– Automatically deduce style from 3D renderings
– (semi)-Automatically capture style from image(s)
The Art and Science of Depiction
97
Shading and highlighting
The Art and Science of Depiction
98
49
Corrective Make Up
• Depending on
the shape of the
face
The Art and Science of Depiction
99
Corrective lighting
The Art and Science of Depiction
100
50
An example
The Art and Science of Depiction
101
Is it fair?
The Art and Science of Depiction
102
51
Touch-up: too dark face
The Art and Science of Depiction
103
Touch-up: silhouette
The Art and Science of Depiction
104
52
Touch-up: undesirable lines
The Art and Science of Depiction
105
Touch-up: stretch and arm
The Art and Science of Depiction
106
53
Deforming lens
• Deform one part of the
frame
• Stretched arm and legs
The Art and Science of Depiction
107
Perspective distortion
• The sphere is projected
as an ellipse
The Art and Science of Depiction
108
54
Perspective distortion
• The sphere is projected
as an ellipse
The Art and Science of Depiction
109
Leonardo & contradictions
• Wide angle vision
The Art and Science of Depiction
110
55
Leonardo & contradictions
• Wide angle vision
• Lateral recession
The Art and Science of Depiction
111
Perspective distortion
• Portrait: distortion with wide angle
Wide angle
The Art and Science of Depiction
Standard
Telephoto
112
56