STAT774: Statistical Computing
5. Monte Carlo Integration and Variance Reduction
Chunsheng Ma
Department of Mathematics, Statistics, and Physics
Wichita State University, Kansas 67260, USA
E-mail: [email protected]
5.1. Monte Carlo
Monte Carlo officially refers to an administrative area
of the Principality of Monaco, specifically the ward of
Monte Carlo, where the Monte Carlo Casino is
located.
Informally the name also refers to a larger district,
the Monte Carlo Quarter (corresponding to the
former municipality of Monte Carlo), which besides
Monte Carlo also includes the wards of La
Rousse/Saint Roman, Larvotto/Bas Moulins, and
Saint Michel. The permanent population of the ward
of Monte Carlo is about 3,500, while that of the
quarter is about 15,000.
Monaco has four traditional quarters. From west to
east they are: Fontvieille (the newest), Monaco-Ville
(the oldest), La Condamine, and Monte Carlo.
Las Vegas
An internationally renowned
major resort city known primarily
for gambling, shopping, fine
dining and nightlife and is the
leading financial, commercial,
and cultural center for Southern
Nevada.
The city bills itself as The Entertainment Capital of the World, and is famous for its mega
casino, hotels and associated entertainment. A growing retirement and family city, Las Vegas is
the 29th-most populous city in the United States, with a population of 603,488 at the 2013
United States Census Estimates. The 2013 population of the Las Vegas metropolitan area was
2,027,828. The city is one of the top three leading destinations in the United States for
conventions, business, and meetings. In addition, the city’s metropolitan area has more AAA
Five Diamond hotels than any other city in the world, and is a global leader in the hospitality
industry. Today, Las Vegas is one of the top tourist destinations in the world.
Established in 1905, Las Vegas was incorporated as a city in 1911. At the close of the 20th
century, Las Vegas was the most populous American city founded in that century (a similar
distinction earned by Chicago in the 19th century). The city’s tolerance for numerous forms of
adult entertainment earned it the title of Sin City, and has made Las Vegas a popular setting for
films, television programs, and music videos.
Los Alamos National Laboratory, Los Alamos, New Mexico
One of two laboratories in the United States where classified
work towards the design of nuclear weapons has been
undertaken (the other being the Lawrence Livermore National
Laboratory).
One of the largest science and technology institutions in the
world.
It is a United States Department of Energy national laboratory, managed and operated by Los
Alamos National Security (LANS). It conducts multidisciplinary research in fields such as
national security, space exploration, renewable energy, medicine, nanotechnology, and
supercomputing. The laboratory was founded during World War II as a secret, centralized facility
to coordinate the scientific research of the Manhattan Project, the Allied project to develop the
first nuclear weapons. The work of the laboratory culminated in the creation of several atomic
devices, one of which was used in the first nuclear test near Alamogordo, New Mexico,
codenamed "Trinity", on July 16, 1945. The other two were weapons, "Little Boy" and "Fat
Man", which were used in the attacks on Hiroshima and Nagasaki.
Nicholas Constantine Metropolis (1915 - 1999)
An American physicist.
BSc (1937), PhD (1941), Physics, University of Chicago.
Robert Oppenheimer recruited him from Chicago, where he
was at the time collaborating with Enrico Fermi and Edward
Teller on the first nuclear reactors, to the Los Alamos National
Laboratory. He arrived in Los Alamos in April of 1943, as a
member of the original staff of fifty scientists.
After World War II, he returned to the faculty of the University
of Chicago as an assistant professor.
He came back to Los Alamos in 1948 to lead the group in the Theoretical Division that designed
and built the MANIAC I computer in 1952 that was modeled on the IAS machine, and the
MANIAC II in 1957. (He chose the name MANIAC in the hope of stopping the rash of such
acronyms for machine names, but may have, instead, only further stimulated such use.) From
1957 to 1965 he was Professor of Physics at the University of Chicago and was the founding
Director of its Institute for Computer Research. In 1965 he returned to Los Alamos where he was
made a Laboratory Senior Fellow in 1980.
John von Neumann (1903 - 1957)
A Hungarian-American pure and applied mathematician,
physicist, inventor, computer scientist, and polymath.
Major contributions: mathematics (foundations of
mathematics, functional analysis, ergodic theory, geometry,
topology, and numerical analysis), physics (quantum
mechanics, hydrodynamics and quantum statistical
mechanics), economics (game theory), computing (Von
Neumann architecture, linear programming, self-replicating
machines, stochastic computing), and statistics.
A consultant to the Los Alamos Scientific Laboratory from
1943 to 1955.
Von Neumann was a founding figure in computing. Von Neumann’s hydrogen bomb work was
played out in the realm of computing, where he and Stanislaw Ulam developed simulations on
von Neumann’s digital computers for the hydrodynamic computations. During this time he
contributed to the development of the Monte Carlo method, which allowed solutions to
complicated problems to be approximated using random numbers. His algorithm for simulating a
fair coin with a biased coin is used in the "software whitening" stage of some hardware random
number generators. Because using lists of "truly" random numbers was extremely slow, von
Neumann developed a form of making pseudorandom numbers, using the middle-square method.
Though this method has been criticized as crude, von Neumann was aware of this: he justified it
as being faster than any other method at his disposal, and also noted that when it went awry it
Monte Carlo methods
Monte Carlo methods (or Monte Carlo experiments) are a broad class of
computational algorithms that rely on repeated random sampling to obtain
numerical results.
In principle, Monte Carlo methods can be used to solve any problem having a probabilistic
interpretation. In physics-related problems, Monte Carlo methods are quite useful for simulating
systems with many coupled degrees of freedom, such as fluids, disordered materials, strongly
coupled solids, and cellular structures. Other examples include modeling phenomena with
significant uncertainty in inputs such as the calculation of risk in business and, in math,
evaluation of multidimensional definite integrals with complicated boundary conditions.
Monte Carlo methods are mainly used in three distinct problem classes:
optimization,
numerical integration,
generating draws from a probability distribution.
5.2. Theoretical supports
Chebyshev’s inequality
Pafnuty Lvovich Chebyshev (1821 - 1894)
A prominent Russian mathematician, professor on algebra,
number theory, and probability at St. Petersburg University
and member of many Academies.
His contributions to the science include
distribution of Prime number theory,
proof of fundamental limit theorems in probability theory,
theory of polynomial approximations to functions,
theory of interpolation,
the theory of moments and the approximate calculus of
definite integrals and others.
Chebyshev’s inequality
For a random variable X with mean µ and variance σ 2 , the inequality
P(|X − µ| > ε) ≤
holds for any ε > 0.
σ2
ε2
Chebyshev’s inequality
For a random variable X with mean µ and variance σ 2 , the inequality
P(|X − µ| > ε) ≤
σ2
ε2
holds for any ε > 0.
Proof.
Z
P(|X − µ| > ε) =
1dF (x )
|x −µ|>ε
Chebyshev’s inequality
For a random variable X with mean µ and variance σ 2 , the inequality
P(|X − µ| > ε) ≤
σ2
ε2
holds for any ε > 0.
Proof.
Z
P(|X − µ| > ε) =
1dF (x )
|x −µ|>ε
Z
≤
|x −µ|>ε
x −µ
ε
2
dF (x )
Chebyshev’s inequality
For a random variable X with mean µ and variance σ 2 , the inequality
P(|X − µ| > ε) ≤
σ2
ε2
holds for any ε > 0.
Proof.
Z
P(|X − µ| > ε) =
1dF (x )
|x −µ|>ε
Z
≤
|x −µ|>ε
≤
1
ε2
Z
x −µ
ε
2
dF (x )
∞
−∞
(x − µ)2 dF (x )
Chebyshev’s inequality
For a random variable X with mean µ and variance σ 2 , the inequality
P(|X − µ| > ε) ≤
σ2
ε2
holds for any ε > 0.
Proof.
Z
P(|X − µ| > ε) =
1dF (x )
|x −µ|>ε
Z
≤
|x −µ|>ε
≤
=
1
ε2
Z
σ2
.
ε2
x −µ
ε
2
dF (x )
∞
−∞
(x − µ)2 dF (x )
Weak law of large number (Khintchine’s law)
If X1 , X2 , . . . , Xn , . . . is a sequence of independent and
identically distributed random variables with mean µ and
variance σ 2 , then
X1 + X2 + . . . + Xn
− µ > ε = 0,
lim P n→∞
n
∀ε > 0.
That is, the sample average converges in probability towards
the expected value.
Aleksandr Khinchin(1894-1959)
One of the founders of modern probability theory, discovering the law of the
iterated logarithm in 1924, achieving important results in the field of limit
theorems, giving a definition of a stationary process and laying a foundation for
the theory of such processes.
Strong law of large number
If X1 , X2 , . . . , Xn , . . . is a sequence of independent and identically distributed
random variables with mean µ, then
X1 + X2 + . . . + Xn
= µ = 1.
P lim
n→∞
n
That is, the sample average converges almost surely towards the expected value.
Examples
Example 1.
R2
0
sin(cos(sin x ))dx =?
R2
Treat the integral as µ = EX =
random variable on (0, 2)
0
g(x )f (x )dx , where f (x ) is the density function of a uniform
f (x ) =
n
1
,
2
0,
0 < x < 2,
otherwise,
and
g(x ) = 2 sin(cos(sin x )).
If X1 , X2 , . . . , Xn is a random sample from f (x ), then an estimate of µ is
µ̂ =
1
n
n
X
i=1
g(Xi ).
Examples
Example 1.
R2
0
sin(cos(sin x ))dx =?
R2
Treat the integral as µ = EX =
random variable on (0, 2)
0
g(x )f (x )dx , where f (x ) is the density function of a uniform
f (x ) =
n
1
,
2
0,
0 < x < 2,
otherwise,
and
g(x ) = 2 sin(cos(sin x )).
If X1 , X2 , . . . , Xn is a random sample from f (x ), then an estimate of µ is
µ̂ =
1
n
n
X
i=1
g=function(x) { 2*sin(cos(sin(x))) }
x=runif(1000, 0, 2)
mean(g(x))
g(Xi ).
Example 2.
R8
0
sin
√ x dx
x
=?
Treat the integral as µ = EX =
random variable on (0, 8)
R8
0
g(x )f (x )dx , where f (x ) is the density function of a uniform
f (x ) =
n
1
,
8
0,
0 < x < 8,
otherwise,
and
8 sin x
√ .
x
If X1 , X2 , . . . , Xn is a random sample from f (x ), then an estimate of µ is
g(x ) =
µ̂ =
1
n
n
X
i=1
g(Xi ).
Example 2.
R8
0
sin
√ x dx
x
=?
Treat the integral as µ = EX =
random variable on (0, 8)
R8
0
g(x )f (x )dx , where f (x ) is the density function of a uniform
f (x ) =
n
1
,
8
0,
0 < x < 8,
otherwise,
and
8 sin x
√ .
x
If X1 , X2 , . . . , Xn is a random sample from f (x ), then an estimate of µ is
g(x ) =
µ̂ =
1
n
n
X
i=1
g=function(x) { 8 * sin(x)/sqrt(x) }
x=runif(1000, 0, 8)
mean(g(x))
g(Xi ).
Example 2 (Continued).
R8
0
Treat the integral as µ = EX =
sin
√ x dx
x
R8
0
=?
g(x )f (x )dx , where f (x ) is a density function
f (x ) =
√1
2 2x
,
0,
0 < x < 8,
otherwise,
and
√ sin x
g(x ) = 2 2 √ .
x
If X1 , X2 , . . . , Xn is a random sample from f (x ), then an estimate of µ is
µ̂ =
1
n
n
X
i=1
g(Xi ).
Example 2 (Continued).
R8
0
Treat the integral as µ = EX =
sin
√ x dx
x
R8
0
=?
g(x )f (x )dx , where f (x ) is a density function
f (x ) =
√1
2 2x
,
0,
0 < x < 8,
otherwise,
and
√ sin x
g(x ) = 2 2 √ .
x
If X1 , X2 , . . . , Xn is a random sample from f (x ), then an estimate of µ is
µ̂ =
1
n
n
X
i=1
g=function(x) { 2*sqrt(2)* sin(x)/x }
x=?
mean(g(x))
g(Xi ).
Time usage
start.time=Sys.time()
g=function(x) { 8* sin(x)/sqrt(x) }
x=runif(1000, 0, 8)
mean(g(x))
Sys.time() - start.time