S6880 #5
Random Number Generation #1: Generate Uniform Random
Number
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Outline
1
Congruential Methods
Multiply-with-carry
Combined Linear Congruent Generator
2
Multiple Recursive Generators
MRGs and Combined MRGs
Lagged-Fibonacci Generators
3
FSR Methods
Feedback Shift Register (FSR) Methods
Generalized FSR
Mersenne Twister
4
Coupled Generators
Coupled Generators
(WMU)
S6880 #5
S6880, Class Notes #5
2 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
General Formula
xi ≡ (αxi−1 + c) ( mod m), i = 1, 2, 3, · · ·
where xi , α, c, m are integers, 0 ≤ xi < m. Start out at x0 (random
seed).
xi
Then ui =
: U[0, 1)
m
It is called mixed-congruential generator when c 6= 0. When c = 0, it is
called multiplicative-congruential generator (Lehmer, 1951).
For convenience, denote xi ≡m (αxi−1 + c).
(WMU)
S6880 #5
S6880, Class Notes #5
3 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
An Example
m = 9, c = 7, α = 10.
x0 = 2, x1 ≡9 (10 × 2 + 7) ≡9 0,
x2 ≡9 (10 × 0 + 7) ≡9 7,
x3 ≡9 5, x4 ≡9 3, x5 ≡9 1, x6 ≡9 8, x7 ≡9 6, x8 ≡9 4,
x9 ≡9 2.
Hence, 9 is the period of the generator.
(WMU)
S6880 #5
S6880, Class Notes #5
4 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Theorem
A mixed-congruential generator has a period m iff
(i) c is relatively prime to m
(ii) α ≡p 1 for every prime factor p of m
(iii) α ≡4 1 if 4 divides m.
(WMU)
S6880 #5
S6880, Class Notes #5
5 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Theorem
A mixed-congruential generator has a period m iff
(i) c is relatively prime to m
(ii) α ≡p 1 for every prime factor p of m
(iii) α ≡4 1 if 4 divides m.
Note: A congruential generator should have period as large as
possible, at least 230 , a multiplier α chosen to give period m or m − 1.
(WMU)
S6880 #5
S6880, Class Notes #5
5 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Example
the infamous RANDU
a generator used widely on IBM 360/370 and PDP-11 machines (late
1960s and 1970s):
m = 231 , α = 216 + 3 = 65539, c = 0.
Can show
Ui+2 − 6Ui+1 + 9Ui is an integer,
(Ui , Ui+1 , Ui+2 ) lies on one of 15 planes in the unit cube =⇒ quite
predictable if (Ui−1 , Ui−2 ) is known even to limited accuracy.
(WMU)
S6880 #5
S6880, Class Notes #5
6 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Example
the infamous RANDU
a generator used widely on IBM 360/370 and PDP-11 machines (late
1960s and 1970s):
m = 231 , α = 216 + 3 = 65539, c = 0.
Can show
Ui+2 − 6Ui+1 + 9Ui is an integer,
(Ui , Ui+1 , Ui+2 ) lies on one of 15 planes in the unit cube =⇒ quite
predictable if (Ui−1 , Ui−2 ) is known even to limited accuracy.
(WMU)
S6880 #5
S6880, Class Notes #5
6 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Example
the infamous RANDU
a generator used widely on IBM 360/370 and PDP-11 machines (late
1960s and 1970s):
m = 231 , α = 216 + 3 = 65539, c = 0.
Can show
Ui+2 − 6Ui+1 + 9Ui is an integer,
(Ui , Ui+1 , Ui+2 ) lies on one of 15 planes in the unit cube =⇒ quite
predictable if (Ui−1 , Ui−2 ) is known even to limited accuracy.
(WMU)
S6880 #5
S6880, Class Notes #5
6 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Multiply-with-carry
by George Marsaglia
Recall that, in LCG, c (the carry) is constant so that with an initial seed
x0 , the subsequent number xn is generated by taking the residue of
modulus b. A lag-1 Multiply-with-carry (MWC) is one that modifies the
carry:
start with initial seed x0 and initial carry c0
the subsequent number xn is as before
the carry is modified each time by keeping the quotient as the new
carry
αxn−1 + cn−1 , n = 1, 2, · · ·
cn =
b
(WMU)
S6880 #5
S6880, Class Notes #5
7 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Lag-r MWC, base m, usually m = 232
Initial Step: Begin with a set of r + 1 random seed values:
x0 , x1 , . . . , xr −1 and an initial carry cr −1 < α
Subsequent Step: the lag-r MWC sequence:
xn ≡m (αxn−r + cn−1 ), cn =
αxn−r + cn−1 , n ≥ r.
m
Output Sequence: xr , xr +1 , xr +2 , · · ·
Usually, m = 232 − 1 is used instead.
(WMU)
S6880 #5
S6880, Class Notes #5
8 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Complementary-multiply-with-carry Generators
base m
The carry uses the same calculation, but the number is replaced by
taking the complement (m − 1) − xn :
xn ≡m (m − 1) − (αxn−r + cn−1 ),
The MWC/CMWC is said to have period ranging from 260 to 22000000 .
Marsaglia-Multicarry is used as an option in RNG in R.
(WMU)
S6880 #5
S6880, Class Notes #5
9 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Combined LCGs, Wichman & Hill 1982, 1984
Initial Step: Length-3 vector as seed values: (x0 , y0 , z0 ), each component
should be between 1 and 30000.
Subsequent Step:
xn ≡30269 171xn−1
yn ≡30307 172xn−1
zn ≡30323 171xn−1
Return
un ≡1
xn
yn
zn
+
+
30269.0 30307.0 30323.0
Such generator has period approximately 7 × 212 (corrected in 1984)
not 2.8 × 213 as claimed originally in 1982. (Wichmann-Hill is used as
an option in RNG in R)
(WMU)
S6880 #5
S6880, Class Notes #5
10 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Combined LCGs, Wichman & Hill 2006
64-bit integer arithmetic:
xn ≡2147483579 11600xn−1
yn ≡2147483543 47003xn−1
zn ≡2147483423 23000xn−1
tn ≡2147483123 33000xn−1
xn
yn
Return un ≡1
+
+
2147483579.0 2147483543.0
zn
tn
+
2147483423.0 2147483123
(WMU)
S6880 #5
S6880, Class Notes #5
11 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Outline
1
Congruential Methods
Multiply-with-carry
Combined Linear Congruent Generator
2
Multiple Recursive Generators
MRGs and Combined MRGs
Lagged-Fibonacci Generators
3
FSR Methods
Feedback Shift Register (FSR) Methods
Generalized FSR
Mersenne Twister
4
Coupled Generators
Coupled Generators
(WMU)
S6880 #5
S6880, Class Notes #5
12 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
MRGs and Combined MRGs
An MRG is defined by
xn ≡m (α1 xi−1 + · · · + αk xi−k
with output un =
xn
m.
Knuth(39): α1 = 271828183, α2 = 314159269, and k = 2
Marsa-LFIB4 (1999): xn ≡232 (xn−55 + xn−119 + xn−179 + xn−256 ).
MRG32k3a proposed by L’Ecuyer (1996): used by SAS.
(WMU)
S6880 #5
S6880, Class Notes #5
13 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Lagged-Fibonacci Generators, LFib(m, r , k , ◦) uses
xn ≡m (xn−r ◦ xn−k )
where ◦ can be one of +, −, ×, xor. With the first two operators being
MRGs.
Unix/Linux’s random(): LFib(232 ,7,3,+), LFib(232 ,15,1,+),
LFib(232 ,31,3,+), and LFib(232 ,63,1,+) with the least significant bit
of each random number dropped.
Knuth-ranf_array2: LFib(230 ,100,37,+), a.k.a.,
Knuth-TAOCP-2002 used as an option in the RNG used by R.
(WMU)
S6880 #5
S6880, Class Notes #5
14 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Outline
1
Congruential Methods
Multiply-with-carry
Combined Linear Congruent Generator
2
Multiple Recursive Generators
MRGs and Combined MRGs
Lagged-Fibonacci Generators
3
FSR Methods
Feedback Shift Register (FSR) Methods
Generalized FSR
Mersenne Twister
4
Coupled Generators
Coupled Generators
(WMU)
S6880 #5
S6880, Class Notes #5
15 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
FSR (Tausworthe, 1965)
(∗) bi ≡2 (a1 bi−1 + · · · + ad bi−d ), where
ai ’s are binary constants.
ui
=
L
X
2−s bit+s = 0.bit+1 · · · bit+L (binary)
s=1
= L-bit binary fractions taken t apart.
t is called the decimation, it is proper if t and 2p − 1 are relatively prime.
Note: for a proper decimation, ui has period 2p − 1.
(WMU)
S6880 #5
S6880, Class Notes #5
16 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
FSR, cont’d
The equation (∗) is associated with the polynomial
f (x) = x d + a1 x d−1 + · · · + ad
Usually consider trinomials 1 + x q + x p with 1 ≤ q < p, so
bi = bi−p xor bi−(p−q)
where xor is the exclusive-or bit-wise operator. The result of x xor y is
1 (TRUE) only if the two operands (bits) x and y are of opposite bits
(1-0 or 0-1).
(WMU)
S6880 #5
S6880, Class Notes #5
17 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Generalized FSR, Lewis & Payne, 1973
introduces delays in some bits:
E.g., Yi = bi bi−`1 · · · bi−`L for delays `1 , . . . , `L .
Yi = Yi−p xor Yi−(p−q) , ui = 2−L Yi
L is called the word length.
(WMU)
S6880 #5
S6880, Class Notes #5
18 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Mersenne Prime
A Mersenne number (or Mersenne prime), named after Marin
Mersenne (a French monk who began the study of these numbers in
the early 17th century), is a positive integer that is one less than a
power of two:
Mp = 2p − 1
Theory shows that Mp must be a prime.
(WMU)
S6880 #5
S6880, Class Notes #5
19 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Largest Mersenne Prime to Date
The GIMPS Project
The Great Internet Mersenne Prime Search (GIMPS) is a distributed
computing project devoted to finding Mersenne prime. It found, as of
March 2011, the largest Mersenne prime M43,112,609 (12,978,189
digits).
(WMU)
S6880 #5
S6880, Class Notes #5
20 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Mersenne Twister (a.k.a., MT19937)
a pseudo uniform random number generator developed in 1997 by Makoto
Matsumoto and Takuji Nishimura. It is basically a twisted GFSR which is
proven to be equi-distributed in 623 consecutive dimensions (over the
whole period). The seed values consist of 624-dimensional set of 32-bit
integers plus a current position in that set. And the random numbers are
obtained by basically shifting through the entries of the incomplete array. It
has a period of 219937 − 1.
Mersenne-Twister is the default RNG in R. It is also used by SPSS.
With the add-on package randtoolbox installed, run this R
command to see an introduction to Mersenne-Twister:
vignette(topic="randtoolbox",package="randtoolbox")
(WMU)
S6880 #5
S6880, Class Notes #5
21 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
WELL
Well Equidistributed Long-period Linear generators
Panneton et al. (2006) proposed WELL generator that improved
MT19937.
(WMU)
S6880 #5
S6880, Class Notes #5
22 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
SFMT
SIMD-oriented Fast Mersenne Twister
Matsumoto & Saito (2008) enhances MT algorithm by incooperating
SIMD (Single Instruction Multiple Data) operations to work
conceptually with 128 bits integers. SFMT allows the use of Mp other
than M19937 (the original MT19937). Other Mp ’s: with p = 607, 1279,
2281, 4253, 11213, 44497, 86243, 132049 and 216091.
(WMU)
S6880 #5
S6880, Class Notes #5
23 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Outline
1
Congruential Methods
Multiply-with-carry
Combined Linear Congruent Generator
2
Multiple Recursive Generators
MRGs and Combined MRGs
Lagged-Fibonacci Generators
3
FSR Methods
Feedback Shift Register (FSR) Methods
Generalized FSR
Mersenne Twister
4
Coupled Generators
Coupled Generators
(WMU)
S6880 #5
S6880, Class Notes #5
24 / 25
Congruential Methods
Multiple Recursive Generators
FSR Methods
Coupled Generators
Coupled Generators
use a combination of two or more generators. For example,
super-duper generator (George Marsaglia, 1973; used in S-Plus).
1
Generate xi by a multiplicative generator:
xi+1 ≡232 69069xi
2
period = 230 and the bottom bit must always be odd.
Generate yi by a FSR generator:
yi : 32-bit integer
bi = bi−p xor bi−(p−q) , p = 32, q = 15
3
4
period = 4,292,868,097 < 232 − 1
zi = xi xor yi — 32-bit integer
ui = (top 31 bits of zi )/231 : [0, 1)
(WMU)
S6880 #5
S6880, Class Notes #5
25 / 25