Towards a computational
account of art cognition
Unifying perception, visual art, and
music through Bayesian inference
Jorge Aurelio Menéndez
Centre for Mathematics, Physics and Engineering in the Life Sciences and Experimental Biology
University College London
Human Vision and Electronic Imaging 2016
What is art?
Human Vision and Electronic Imaging 2016
What is art?
What are the computations
that give rise to art?
Human Vision and Electronic Imaging 2016
Art perception =
computing & evaluating
predictions
Human Vision and Electronic Imaging 2016
Predictions in Music
Human Vision and Electronic Imaging 2016
Predictions in Music
do re mi fa so la ti …
{do, di, re, me, mi, …}
Human Vision and Electronic Imaging 2016
Predictions in Music
do re mi fa so la ti …
{do, di, re, me, mi, …}
Human Vision and Electronic Imaging 2016
Predictions in Music
do re mi fa so la ti …
I - IV - ii - I64 -V - …
{do, di, re, me, mi, …}
{I, ii, iii, IV, …}
Human Vision and Electronic Imaging 2016
Predictions in Music
do re mi fa so la ti …
I - IV - ii - I64 -V - …
{do, di, re, me, mi, …}
{I, ii, iii, IV, …}
Human Vision and Electronic Imaging 2016
Predictions in Music
Leonard Meyer (1957)
Antecedent
Consequent
do re mi fa so la ti …
I - IV - ii - I64 -V - …
{do, di, re, me, mi, …}
{I, ii, iii, IV, …}
Human Vision and Electronic Imaging 2016
Predictions in Music
Leonard Meyer (1957)
Antecedent
Consequent
do re mi fa so la ti …
I - IV - ii - I64 -V - …
{do, di, re, me, mi, …}
{I, ii, iii, IV, …}
Examples
Tristan und Isolde
Goodbye Blue Skies
Also sprach Zarathustra
Human Vision and Electronic Imaging 2016
Predictions in Music
Leonard Meyer (1957)
Antecedent
Consequent
do re mi fa so la ti …
I - IV - ii - I64 -V - …
{do, di, re, me, mi, …}
{I, ii, iii, IV, …}
Examples
Tristan und Isolde
Goodbye Blue Skies
Also sprach Zarathustra
Human Vision and Electronic Imaging 2016
Predictions in Music
Leonard Meyer (1957)
Antecedent
Consequent
do re mi fa so la ti …
I - IV - ii - I64 -V - …
{do, di, re, me, mi, …}
{I, ii, iii, IV, …}
Examples
Tristan und Isolde
Goodbye Blue Skies
Also sprach Zarathustra
Human Vision and Electronic Imaging 2016
Predictions in Music
Leonard Meyer (1957)
Antecedent
Consequent
do re mi fa so la ti …
I - IV - ii - I64 -V - …
{do, di, re, me, mi, …}
{I, ii, iii, IV, …}
Examples
Tristan und Isolde
Goodbye Blue Skies
Also sprach Zarathustra
Human Vision and Electronic Imaging 2016
Predictions in Music
Evidence?
Human Vision and Electronic Imaging 2016
Predictions in Music
Evidence?
phenomenology
Human Vision and Electronic Imaging 2016
Predictions in Music
Evidence?
phenomenology
style
Human Vision and Electronic Imaging 2016
Predictions in Music
Evidence?
phenomenology
style
universals
Human Vision and Electronic Imaging 2016
Predictions in Music
Evidence?
phenomenology
style
universals
individual differences
Human Vision and Electronic Imaging 2016
Predictions in Music
Evidence?
phenomenology
style
universals
individual differences
history of Western music
Human Vision and Electronic Imaging 2016
Predictions in Music
Evidence?
phenomenology
style
universals
individual differences
history of Western music
20th century music
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
Interpretation
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
Interpretation
“this painting is
about sadness”
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
prediction
Interpretation
“this painting is
about sadness”
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
[there should be a lot of blue]
prediction
Interpretation
“this painting is
about sadness”
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
[there should be a lot of blue]
prediction
Interpretation
“this painting is
about sadness”
Top-down
attention
[look for blue]
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
[there should be a lot of blue]
prediction
Top-down
attention
Interpretation
“this painting is
about sadness”
[look for blue]
evaluation
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
[there should be a lot of blue]
prediction
Interpretation
“this painting is
about sadness”
Top-down
attention
[look for blue]
evaluation
[if no blue, then probably not about sadness]
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
Evidence?
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
Evidence?
individual differences
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
Evidence?
individual differences
top-down effects
Human Vision and Electronic Imaging 2016
Predictions in Visual Art
Evidence?
individual differences
top-down effects
Western art history
Human Vision and Electronic Imaging 2016
Art perception =
computing & evaluating
predictions
Human Vision and Electronic Imaging 2016
Art perception =
computing & evaluating
predictions
Human Vision and Electronic Imaging 2016
Art perception =
computing & evaluating
predictions
Bayesian inference
Human Vision and Electronic Imaging 2016
{
probability of the hypothesis,
given the observed data
Human Vision and Electronic Imaging 2016
{
Bayes' theorem
probability of the hypothesis,
given the observed data
Human Vision and Electronic Imaging 2016
Bayes' theorem
{
[it is a box]
probability of the hypothesis,
given the observed data
Human Vision and Electronic Imaging 2016
Bayes' theorem
{
retinal input
[it is a box]
probability of the hypothesis,
given the observed data
Human Vision and Electronic Imaging 2016
Bayes' theorem
retinal input
[it is a box]
Probability of a box
generating this
retinal input
{
{
probability of the hypothesis,
given the observed data
Human Vision and Electronic Imaging 2016
Bayes' theorem
retinal input
[it is a box]
Probability of a box
“prior” probability
generating this
of a box
retinal input
{
{
{
probability of the hypothesis,
given the observed data
Human Vision and Electronic Imaging 2016
{do, di,…}
do re mi fa so la ti
Human Vision and Electronic Imaging 2016
{do, di,…}
do re mi fa so la ti
{sadness,
happiness,…}
{blueness,
curved edges,…}
Human Vision and Electronic Imaging 2016
{do, di,…}
do re mi fa so la ti
style
{sadness,
happiness,…}
{blueness,
curved edges,…}
Human Vision and Electronic Imaging 2016
Predictive Coding
Rao & Ballard (1999)
Human Vision and Electronic Imaging 2016
Evolutionary rationale
Uncertainty
- noisy environment
- noisy internal signals
- learning
- decision outcomes
Human Vision and Electronic Imaging 2016
Spans all Marr’s levels!
Human Vision and Electronic Imaging 2016
Spans all Marr’s levels!
Computational: why do we enjoy art?
because our brains are built to compute predictions
Human Vision and Electronic Imaging 2016
Spans all Marr’s levels!
Computational: why do we enjoy art?
because our brains are built to compute predictions
Algorithmic: how do we enjoy art?
Bayesian inference, predictive coding
Human Vision and Electronic Imaging 2016
Spans all Marr’s levels!
Computational: why do we enjoy art?
because our brains are built to compute predictions
Algorithmic: how do we enjoy art?
Bayesian inference, predictive coding
Implementational: how does our brain do it?
neural pathways encoding predictions and prediction error
Human Vision and Electronic Imaging 2016
Art perception =
computing & evaluating
predictions
Human Vision and Electronic Imaging 2016
Art perception =
computing & evaluating
predictions
…through Bayesian
inference
Human Vision and Electronic Imaging 2016
Art perception =
computing & evaluating
predictions
…through Bayesian
inference
(…via predictive coding?)
Human Vision and Electronic Imaging 2016
Need to build
computational models!!
Human Vision and Electronic Imaging 2016
Thank you!
Human Vision and Electronic Imaging 2016
Towards a computational
account of art cognition
Unifying perception, visual art, and
music through Bayesian inference
Jorge Aurelio Menéndez
Centre for Mathematics, Physics and Engineering in the Life Sciences and Experimental Biology
University College London
Human Vision and Electronic Imaging 2016