Metropolis Light
Transport and Manifold
Exploration
Matthew Meyn
5/2017
Problem:
 How
do we turn a 3D scene into an
image?
Problem: How do we turn a 3D scene
into an image?

Solution #1:
 Rasterization/Scanline
Conversion!
Problem: How do we turn a 3D scene
into an image?

Solution #1:
 Rasterization/Scanline

What do we like about it?
Conversion!
Problem: How do we turn a 3D scene
into an image?

Solution #1:
 Rasterization/Scanline

What do we like about it?

Fast

Easy (…?)

Lots of hardware built for it
Conversion!
Problem: How do we turn a 3D scene
into an image?

Solution #1:
 Rasterization/Scanline

Conversion!
What do we like about it?

The rasterizer only needs one triangle at a time, plus the entire image
and associated depth information for all pixels

A rasterization-based renderer can stream over the triangles, no need to
keep entire dataset around

– Allows parallelism and optimization of memory systems
Source: Matusik, MIT Open Courseware
https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-837-computer-graphics-fall-2012/lecturenotes/MIT6_837F12_Lec21.pdf
Problem: How do we turn a 3D scene
into an image?

Solution #1:
 Rasterization/Scanline

What DON’T we like about it?
Conversion!
Problem: How do we turn a 3D scene
into an image?

Solution #1:
 Rasterization/Scanline

Conversion!
What DON’T we like about it?

Can only render planar polygons (i.e., triangles)

Only get a single “bounce” from light source


1 sample at each pixel

Does not model indirect lighting
Shadows, Reflections, Refractions, etc. not inherently part of the model


We have to hack these effects
Makes Eugene upset
Problem: How do we turn a 3D scene
into an image?

Solution #1:
 Rasterization/Scanline

Conversion!
What DON’T we like about it?
It’s fake!

Not a real simulation of light rays.
Problem: How do we turn a 3D scene
into an image?

Solution #2:
 Path

tracing
modeling light rays
Problem: How do we turn a 3D scene
into an image?

Solution #2:
 Path

tracing
What do we like about it?

Reflections, refractions, shadows created as part of the simulation

Global/indirect illumination

Simulates physical light transport…
Problem: How do we turn a 3D scene
into an image?

Solution #2:
 Path

tracing
What do we like about it?
It’s

a real simulation!
Produces more realistic results
Problem: How do we turn a 3D scene
into an image?

Solution #2:
 Path

tracing
What don’t we like about it?
Problem: How do we turn a 3D scene
into an image?

Solution #2:
 Path

tracing
What don’t we like about it?

Computationally expensive

Can be hard to implement in hardware

(Not a big problem now with GP GPUs)
Source: Matusik, MIT Open Courseware
https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-837-computer-graphics-fall-2012/lecturenotes/MIT6_837F12_Lec21.pdf
Review: BRDF
𝐿0
𝐵𝑅𝐷𝐹 =
𝐸𝑖
𝐿0
=
𝐿𝑖 cos(𝜃𝑖 ) 𝑑𝑤𝑖
Light Reflected in
direction 𝑤0
Light arriving from
direction 𝑤𝑖
Review: BRDF
(BDRF?)
Review: BRDF
Bi-directional Reflectance Distribution Function
The Rendering Equation
An infinte-dimensional integral—
oh, great!
Review: Path Tracing
Finding the radiance reflected toward the camera
• We need the radiance incident at x from all directions
• Also need the material properties of the surface
Source: Marc Sunet,
https://www.slideshare.net/jeannekamikaze/introduction-to-path-tracing
Forward/Backward Tracing

Where to start?
“Backward” Tracing

If we start from the light source…

The ray may or may not find the lens.

Not very efficient.
Forward (Backward?) Tracing

Let’s start from the lens.
Forward Tracing

Path Tracing (simple)
• Shoot a ray from camera in random
direction
• Bounce ray from the first surface it
hits to a light source
• For indirect lighting, spawn another
ray in “random” direction
•
stochastically chosen based on the light
scattering properties of the surface
material
• Bounce that ray to a light source,
spawn another ray, etc….
Forward Tracing

Path Tracing (simple)
• Okay, this is faster, but has some
problems:
• Not as good at indirect
lighting
• The more bounces, the less
confidence about path (lower
probability => more noise)
What can we do?
Bidirectional Path Tracing
generates path from light source and eye and then combines each pair of vertices
Source: Cao Jiayin
https://agraphicsguy.wordpress.com/2016/01/03/naive-bidirectional-path-tracing/
Bidirectional Path Tracing
Trace 1 light bounce, then connect to eye path.
OK, but how about more bounces from the light?
Bidirectional Path Tracing
Multiple light bounces:
We can also do one bounce from camera.
Or no bounces from camera, or no bounces from light source…
Bidirectional Path Tracing
Combine the results of all these passes:
Bidirectional Path Tracing
•
Okay, this is better still, but there are still effects it doesn’t handle well.
•
Refaction
•
Caustics
•
Complex reflections
Bidirectional Path Tracing
•

Okay, this is better still, but there are still effects it doesn’t handle well.
•
Refaction
•
Caustics
•
Complex reflections
How can we do better?
What was the rendering equation again?
What was the rendering equation again?
For measurement j:
We(j) is a weight indicating how much light arriving at x’ from x contributes to
the value of the measurement
If we expand the recursion…
Source: Morgan McGuire,
http://graphics.cs.williams.edu/courses/cs371/f14/gallery/8-Final/mlt-presentation.pdf
If we expand the recursion…

In all cases, the integrand is simply the contribution of the path from x0 to
xn, so we can express this entire integral as simply an integral over path
space, with f the path contribution function, and mu an appropriate measure,
which expands to the proper product.
Source: Morgan McGuire,
http://graphics.cs.williams.edu/courses/cs371/f14/gallery/8-Final/mlt-presentation.pdf
So, how do we perform this
integration???
So, how do we perform this
integration???

We can sample randomly (Monte Carlo methods)
Source: Barbara Cutler, https://www.cs.rpi.edu/~cutler/classes/advancedgraphics/S11/
So, how do we perform this
integration???
So, how do we perform this
integration???

We can sample randomly (Monte Carlo methods)

Step 1. Choose a ray given (x,y,u,v,t)

weight = 1

Step 2. Trace ray to find point of intersection with the nearest surface.

Step 3. Randomly decide whether to compute emitted or reflected light.

Step 3A. If emitted, return weight * Le

Step 3B. If reflected, weight *= reflectance

Randomly scatter the ray according to the BRDF pdf

Go to Step 2.
Source: John F. Hughes, http://cs.brown.edu/courses/cs224/papers/mc_pathtracing.pdf
Monte Carlo Integration
Source: Barbara Cutler, https://www.cs.rpi.edu/~cutler/classes/advancedgraphics/S11/
Monte Carlo Integration
Source: Barbara Cutler, https://www.cs.rpi.edu/~cutler/classes/advancedgraphics/S11/
Monte Carlo Integration
Metropolis
Light Transport
to the rescue!
Metropolis Light Transport to the rescue!

Metropolis Light Transport (MLT) is an unbiased global illumination algorithm
originally introduced by Veach and Guibas in 1997.
Metropolis Light Transport to the rescue!

Metropolis Light Transport (MLT) is an unbiased global illumination algorithm
originally introduced by Veach and Guibas in 1997.
(in graphics)

The Metropolis Algorithm is a way of sampling hard-to-sample probability
distributions, that dates back to a paper in Chemical Physics in 1953 by
Metropolis et. al.
Metropolis Light Transport to the rescue!

Idea: Define a Markov chain that has the function to be integrated as its
stationary distribution

if the chain is run for a long time the distribution of states it visits will be
proportional to the desired distribution.
Source: Wenzel Jakob, Steve Marschner,
http://www.cs.cornell.edu/projects/manifolds-sg12/
Stationary Distribution

After many transitions, the distribution of states visited
will converge toward 𝜋(𝐱)  stationary distribution
Metropolis Light Transport to the rescue!

The Metropolis Algorithm works by sampling the space through a Markov Chain
of states

establishing probabilities for transitioning to the next state.
Metropolis Light Transport to the rescue!

The Metropolis Algorithm works by sampling the space through a Markov Chain
of states: establishing probabilities for transitioning to the next state.

Given some initial state Ω, a sequence of random variables , X1, X2, …, Xn , is
a called Markov chain if the probability density function of is only dependent
on Xi-1 according to some transition function P( · | xi-1).
What??
Metropolis Light Transport

So: we’re going to take a random walk through the state space!

Just walk around at random?
Metropolis Light Transport

So: we’re going to take a random walk through the state space!


Just walk around at random?
No, silly! Of course not. Our walk is going to be guided by our Markov
probabilities.
Metropolis Light Transport

A discrete random walk consists of the following steps.

Step 1. Create a random particle in state i with probability pi0 .

Step 2. With probability p∗i , terminate in state i.


Score particle in state i by incrementing the counter for state i

Go to Step 1.
Step 3. Randomly select new state according to the transition probability
distribution.

Set i to new j.

Go to Step 2.
Metropolis Light Transport to the rescue!

Metropolis transition rule:
 1. Choose x according to the probability distribution T (x, x ).
′
′
x′ with probability min(1, π(x′)/π(x))

2. xnext =
{
x otherwise
Metropolis Light Transport to the rescue!

Hastings acceptance probability:

T (x, x′ ) must be nonzero exactly when T (x′ , x) is nonzero.
Metropolis transition probabilities

transition probability should approximate the radiance contribution

paths with more light flowing along them should be chosen more often
Source: Wenzel Jakob, Steve Marschner,
http://www.cs.cornell.edu/projects/manifolds-sg12/
Metropolis transition probabilities

How do we sample paths that contribute?

Use path tracing to get initial paths

mutations change the structure of the path

perturbations move the vertices by small distances while preserving the
structure
Source: Wenzel Jakob, Steve Marschner,
http://www.cs.cornell.edu/projects/manifolds-sg12/
Metropolis Light Transport

OK, so what? How do we use that to solve the rendering equation?
Metropolis Light Transport

OK, so what? How do we use that to solve the rendering equation?
Metropolis Light Transport

trying to approximate that integral over path space, so we want to take a
finite number of samples, chosen randomly.

However, it makes intuitive sense that we do not want to sample uniformly, as
most paths will have 0 contribution.

So, we sample by picking an initial path out of some distribution: we get this
distribution by doing path tracing, as that gives us a lot of paths which we
know have non-zero transport.
Metropolis Light Transport

trying to approximate that integral over path space, so we want to take a
finite number of samples, chosen randomly.

However, it makes intuitive sense that we do not want to sample uniformly, as
most paths will have 0 contribution.

So, we sample by picking an initial path out of some distribution: we get this
distribution by doing path tracing, as that gives us a lot of paths which we
know have non-zero transport.

Then, we proceed to do a random walk around path space, mutating paths
through insertions and deletions, and accepting only based on a calculated
acceptance probability that is proportional to the contribution of that path.
Metropolis Light Transport

All this random walking is subject to a condition known as detailed balance.
Detailed Balance

We want to preserve equilibrium:

“The idea is to invent or construct a transition function K whose resulting
stationary distribution will be proportional to the given f, and which will
converge to f as quickly as possible.”

“…consider the density of transitions between any two states x ̄ and y ̄. From
x ̄ to y ̄, the transition density is proportional to f(x ̄) T(y ̄|x ̄) a(y ̄|x ̄), and a
similar statement holds for the transition density from y ̄ to x ̄.”
--Veach, Guibas, “Metropolis Light Transport”
Detailed Balance
• We have some equilibrium function f(x) as target.
• We need to construct a Markov Chain such that in the limit as n -> ∞, the x’s
are eventually distributed proportionally to f
Detailed Balance

To preserve detailed balance, we need to make sure that for any two states,
𝑥, 𝑦 ∈ Ω,
𝑓 𝑥 𝑝 𝑦 𝑥 = 𝑓 𝑦 𝑝(𝑥|𝑦)

Detailed balance states that the expected flow of “energy” between any two
states should be equal, where energy refers to the value of target function.
Source: Travis Fischer,
https://cs.brown.edu/research/pubs/theses/ugrad/2009/fischer.pdf
Path Mutations

OK, so how do we mutate a path?
Path Mutations

OK, so how do we mutate a path?

For a bidirectional mutation, we delete some subpath (chosen at random with
some probability) of the initial path.
Bidirectional Mutations

OK, so how do we mutate a path?

For a bidirectional mutation, we delete some subpath (chosen at random
with some probability) of the initial path.

Then, we need to fill in the gap, so we choose at random some number of
edges to add from each side.

Then we scatter, so, sampling BSDFs, in order to pick new internal points
in the path, growing towards the center

Finally, we reconnect the two paths if the two points are mutually visible:
otherwise, the path will have 0 contribution, and will be immediately
thrown out.
Source: Morgan McGuire,
http://graphics.cs.williams.edu/courses/cs371/f14/gallery/8-Final/mlt-presentation.pdf
Bidirectional Mutations

OK, so how do we mutate a path?
How do we decide to accept a mutation?

We want to respect the equilibrium implied by detailed balance.
𝑓 𝑥 𝑝 𝑦 𝑥 = 𝑓 𝑦 𝑝(𝑥|𝑦)

make a(y ̄|x ̄) and a(x ̄|y ̄) as large as possible, by letting
𝑓 𝑦 𝑇 𝑥𝑦
𝑝 𝑦 𝑥 = min(1,
)
𝑓 𝑥 𝑇 𝑦𝑥
Veach, Guibas, “Metropolis Light Transport”
How do we decide to accept a mutation?

make a(y ̄|x ̄) and a(x ̄|y ̄) as large as possible, by letting
𝑓 𝑦 𝑇 𝑥𝑦
𝑝 𝑦 𝑥 = min(1,
)
𝑓 𝑥 𝑇 𝑦𝑥

The resulting Markov Chain will converge to the desired stationary distribution
if we ensure that | 0 , Ω with 0 and 0.
Source: Travis Fischer,
https://cs.brown.edu/research/pubs/theses/ugrad/2009/fischer.pdf
Metropolis Light Transport

Great! So, how do we get started?

Starting at a single random path will likely result in a black image: the initial path
will have 0 contribution.

To fix this, we start with a distribution of contributing paths (from path tracing)

Run the algorithm several times and average the results
I’m still skeptical! Does it really work?

Yes!

Depends on 2 properties:
I’m still skeptical! Does it really work?

Yes!

Depends on 2 properties:

Stationary Distribution
 Our
acceptance probabilities guarantee this
I’m still skeptical! Does it really work?


Yes!
Depends on 2 properties:

Stationary Distribution
 Our

acceptance probabilities guarantee this
Ergodicity
 If
the allowable mutations are too restricted, it is possible for
the random walk to get “stuck” in some subregion of the path
space
Source: Veach and Guibas,
https://graphics.stanford.edu/papers/metro/metro.pdf
Ergodicity
Source: Veach and Guibas, https://graphics.stanford.edu/papers/metro/metro.pdf
Ergodicity

want to ensure that the random walk converges to an ergodic state.

This means that no matter how X0 is chosen, it converges to the same
stationary distribution p.

To do this, it is sufficient to ensure that T (y ̄ | x ̄) > 0 for every pair of states
x ̄, y ̄ with f(x ̄) > 0 and f(y ̄) > 0

Source: Veach and Guibas, https://graphics.stanford.edu/papers/metro/metro.pdf
Does it really work?

The MLT algorithm is unbiased, which means that it actually will converge in the
limit.

Ergodicity means it can never be stuck

acceptance probabilities are chosen such that we get a stationary distribution, so once
we reach a correct sampling distribution, we never deviate from it.
Source: Morgan McGuire,
http://graphics.cs.williams.edu/courses/cs371/f14/gallery/8-Final/mlt-presentation.pdf
What do we get?

Kelemen-MLT after 10 seconds of rendering at 1280x720

by Dietger van Antwerpen, IGAD
Manifold Exploration
Manifold Exploration
Loosely, manifolds are topological spaces that look locally like
Euclidean space.
 A little more precisely it is a space together with a way of
identifying it locally with a Euclidean space which is compatible
on overlaps.


Source: Tomasz S. Mrowka, MIT: https://ocw.mit.edu/courses/mathematics/18-965geometry-of-manifolds-fall-2004/lecture-notes/lecture1.pdf
Manifold Exploration

A (real) n-dimensional manifold is a topological space M for which
every point x ∈ M has a neighbourhood homeomorphic to Euclidean
space Rn.

Definition 1. (Coordinate system, Chart, Parameterization) Let M be
a topological space and U ⊆ M an open set. Let V ⊆ Rn be open. A
homeomorphism φ : U → V, φ(u) = (x1(u), . . . , xn(u)) is called a
coordinate system on U , and the functions x1 , . . . xn the coordinate
functions. The pair (U , φ) is called a chart on M. The inverse map
φ−1 is a parameterization of U

Source: Jenny Wilson, https://web.stanford.edu/~jchw/WOMPtalk-Manifolds.pdf
Manifold Exploration

Source: Todd Rowland, http://mathworld.wolfram.com/Manifold.html
Manifold Exploration

The problem:
Certain classes of light paths have traditionally been a
source of difficulty in conducting Monte Carlo simulations of
light transport.
 A well-known example is specular-diffuse-specular paths,
such a tabletop seen through a drinking glass sitting on it, a
bottle containing shampoo or other translucent liquid, or a
shop window viewed and illuminated from outside.

--Wenzel Jakob, Steve Marschner,
http://www.cs.cornell.edu/projects/manifolds-sg12/
Manifold Exploration

The problem: how to do this
http://www.cs.cornell.edu/projects/manifolds-sg12/
Manifold Exploration

The problem:

Basically, ideally specular and nearly specular (“glossy” material) paths are
difficult to render using Monte Carlo techniques.
--Wenzel Jakob, Steve Marschner,
http://www.cs.cornell.edu/projects/manifolds-sg12/
Manifold Exploration

The problem:

to generate proposals, previous algorithms rely on local path sampling
strategies (i.e. path tracing).

choose the next interaction vertex along a light path by sampling from a
directional distribution associated with the current vertex, followed by an
intersection search
--Wenzel Jakob, Steve Marschner,
http://www.cs.cornell.edu/projects/manifolds-sg12/
Manifold Exploration

Insight: the set of paths undergoing specular reflection or refraction is
lower in dimension than the full path space.
Manifold Exploration

Insight: the set of paths undergoing specular reflection or refraction is
lower in dimension than the full path space.

Light is reflected only at certain angles

Apply these specular constraints and get a lower dimension space:

Specular Manifold
Manifold Exploration

Normally, unbiased rendering just “throws darts” at the path space.

We want to explore the specular manifold systematically.

Specular constraint
constant 𝑛 = 𝐻 𝜔𝑖 + 𝜔𝑜
𝐻 𝜔𝑖 + 𝜔𝑜 ←
(𝜔𝑖 +𝜔𝑜 )
𝜔𝑖 + 𝜔𝑜
Manifold Exploration

Normally, unbiased rendering is just “throwing darts” at the path space.

We want to explore the specular manifold systematically.

Given a path with k specular bounces, we have k constraint equations.

Stack those constraint equations into a vector-valued function C.

Then what are our valid paths?
𝑆 = 𝑥1 , … 𝑥𝑛 ∈ Ω 𝐶 𝑥1 , … 𝑥𝑛 = 𝟎 }

Implicitly defined manifold
Manifold Exploration

Implicitly defined manifold
𝑆 = 𝑥1 , … 𝑥𝑛 ∈ Ω 𝐶 𝑥1 , … 𝑥𝑛 = 𝟎 }

Implicit function theorem:
1.
Local parameterization of S exists
2.
Compute tangent space by differentiating C
Manifold Exploration

Implicitly defined manifold
𝑆 = 𝑥1 , … 𝑥𝑛 ∈ Ω 𝐶 𝑥1 , … 𝑥𝑛 = 𝟎 }

Each specular constraint depends only on 3 vertices.

Differentiate with respect to any vertex, get new tangent vertices!
Manifold Exploration

Implicitly defined manifold
𝑆 = 𝑥1 , … 𝑥𝑛 ∈ Ω 𝐶 𝑥1 , … 𝑥𝑛 = 𝟎 }

Differentiate with respect to any vertex, get new tangent vertices!

Actually, it’s only a linear approximation.

adding an offset to each vertex will move the path off the specular
manifold.
Manifold Exploration
Projecting back to the specular manifold
𝑆 = 𝑥1 , … 𝑥𝑛 ∈ Ω 𝐶 𝑥1 , … 𝑥𝑛 = 𝟎 }

Ray tracing provides a deterministic means of projecting an off-specular
path back onto the space of valid configurations based on its first two
vertices. Given x1 and x2, we can trace a sequence of rays xi → xi+1, at
each step performing a specular reflection or refraction, and this leads to
corrected positions.
Short Demonstration
http://www.cs.cornell.edu/projects/manifolds-sg12/
(7:20)