Rank Correlation in
Crystal Ball Simulations
(or How We Overcame Our Fear of Spearman’s R
in Cost Risk Analyses)
Mitch Robinson & Sandi Cole
June 11-14, 2002
Fear
Why should you fear
rank correlation
in cost risk analyses?
1
Fear2
Cost risk researchers have recently questioned the
use of Monte Carlo tools that simulate cost driver
correlations using “rank correlation methods”:
“Crystal Ball and @Risk use rank [correlation methods]. Rank
correlation is easier to simulate than Pearson correlation;
however, as we've seen, rank correlation is not appropriate
for cost risk analyses.” Sources: 67th Military Operations Research
Symposium, 1999; 32nd Annual DoD Cost Analysis Symposium, 1999; ISPA/SCEA
Joint Meeting, 2001
Why do they fear rank correlation?
2
Rank Correlation
Rank correlation measures how consistently one
variable increases (or decreases) in a second
variable—”monotonicity” between two variables:
= 1 one of two variables strictly increases in the other;
= -1 if one of two variables strictly decreases in the other;
 (-1,1) if one of two variables is constant or variably
increases and decreases in the second.
The Spearman r is one rank correlation measure.
3
Spearman Rank Correlation
Y strictly decreases in X; Spearman r = -1.00.
Y = 1/X
0.001 1000.000
0.003 333.333
0.005 200.000
0.007 142.857
0.009 111.111
0.030
33.333
0.050
20.000
0.070
14.286
0.090
11.111
0.300
3.333
0.500
2.000
0.700
1.429
0.900
1.111
1.000
1.000
10.000
0.100
1000
Y
X
-200
-1
X
4
10
Product Moment Correlation (1)
However, … our statistical tools generally require
“linear association” measures—how consistently do
two variables covary linearly?
= 1 if the two variables covary on a positively-sloped line;
= -1 if the two variables covary on a negatively-sloped line;
 (-1,1) if one variable is constant in the second variable or
the two don’t covary linearly.
The Pearson r addresses linear covariation.
5
Pearson Product Moment Correlation
Same numbers. Pearson r = -0.18.
Y = 1/X
0.001 1000.000
0.003 333.333
0.005 200.000
0.007 142.857
0.009 111.111
0.030
33.333
0.050
20.000
0.070
14.286
0.090
11.111
0.300
3.333
0.500
2.000
0.700
1.429
0.900
1.111
1.000
1.000
10.000
0.100
1000
Y
X
The regression line modeling “linearity” is
about Y = 142 –19  X.
-200
-1
X
6
10
Spearman vs. Pearson
Monotonicity does not imply linearity!
Y
0.001 1000.000
0.003 333.333
0.005 200.000
0.007 142.857
0.009 111.111
0.030
33.333
0.050
20.000
0.070
14.286
0.090
11.111
0.300
3.333
0.500
2.000
0.700
1.429
0.900
1.111
1.000
1.000
10.000
0.100
1000
Spearman r = -1.00
Pearson r = -0.18
Y
X
-200
-1
X
7
10
What’s Up with Crystal Ball™ (1)
Crystal Ball implements Iman and Conover’s
(1982) method for simulating rank correlation.
Iman and Conover’s step-by-step mathematical logic
proves their algorithm for simulating Spearman
correlations.
8
What’s Up with Crystal Ball™ (2)
However, Crystal Ball nominally uses the ImanConover algorithm to simulate Pearson rs.
• This assumption does not follow from the Iman-Conover
logic and is thus sensibly suspect.
• We’ve seen the potential for bad disconnects between
Spearman rs and Pearson rs for the same sets of numbers.
• We should thus want to know, Does Crystal Ball  produce
correlations that accurately match our intended Pearsonsense target correlations?
9
This Presentation
 Can Crystal Ball accurately simulate Pearson
correlations?
 Are there conditions or practices that contribute
to better or worse accuracy performance?
10
The General Approach (1)
– Define 33 inputs and respective probability distributions in an
Excel spreadsheet using Crystal Ball—“assumption cells”.
– Define 33 outputs in an Excel spreadsheet using Crystal
Ball, each equal to an assumption cell—“forecast cells”.
11
The General Approach (2)
– Define a target correlation matrix.
Var 2
Var 3
Var 32
1
0.75
0.75
0.75
0.75
1
0.75
0.75
0.75
1
0.75
0.75
1
0.75
Var 33
Var 1
Var 1
Var 2
Var 3
Var 32
Var 33
1
– Configure the simulation using the Crystal Ball “Run
Preferences” menu.
12
The General Approach (3)
– Run 10,000 simulation trials.
– Calculate the simulated Pearson correlations by applying the
Excel correlation tool (in “Tools-Data Analysis”) to the
“extracted,” forecast cell outputs.
– Compare the simulated Pearson correlations with their
respective target correlations.
13
The First Study (1)
– 33 variables—yielded 528 correlations for accuracy
tests.
– Identical target correlations among the variables.
– Identical triangular (0,0.25, 1.0) probability
distributions—slightly right skewed with mode = 0.25,
mean = 0.42.
14
The Tr (0, 0.25, 1.0) Distribution
2.0
0.0
0
0.25
0.5
15
0.75
1
The First Study (2)
– 10,000 trials—10,000 numbers for each variable.
– “Correlation sample” = 10,000—i.e., apply the
correlation algorithm to the entire set of numbers.
If correlation sample = 1000 Crystal Ball would apply the algorithm 10
times, to successive “batches” comprising 1000 trials per variable and
33 variables.
– 3 x 4 study design—run the 10,000 simulation trials
under each of 12 separate conditions:

target correlation = 0.25, 0.50, or 0.75.

starting seed = 1; 2; 1,048,576; or 2,097,152—the four number streams
are mutually nonoverlapping over their first 2 million members.
16
First Study Results (1)
– More than 98% of the 6336 simulated correlations
were within 0.03 of the target; all but 5 were within
0.05 of the target; all were within 0.06 of the target.
– Seed = 2 produced 73 of the 99 simulated correlations
that missed their target by more than 0.03; all of the 5
that missed their target by more than 0.05.
– Nearly 75% of the simulated correlations were less
than their target; this varied only negligibly over the
three targets; seed = 2 produced a 68%/32% split.
17
First Study Results (2)
Mean absolute error (n = 528)
Target:
Seed
1
2
1,048,576
2,097,152
0.25
0.50
0.75
0.007
0.008
0.010
0.012
0.009
0.009
0.008
0.007
0.011
0.011
0.009
0.009
18
The Second Study (1)
The first study related every variable to every
other variable.
Did this dense “correlation network”—528
nonzero correlations interconnecting all 33
variable pairs—drive the correlation accuracy
results?
19
The Second Study (2)
– Assigned nonzero correlations only to (x1, x2),
(x3, x4), … (x31, x32), reducing the correlation
yield from 528 to 16 in each replication—i.e.,
3 targets x 4 seeds.
– Other second study design choices are
identical to their first study counterparts.
20
Second Study Results (1)
– All of the 192 simulated correlations were within
0.02 of their target.
– All of the simulated correlations were less than the
target.
21
Second Study Results (2)
(First Study Results)
Mean absolute error (n = 64)
Target:
Seed
1
2
1,048,576
2,097,152
0.25
0.50
0.75
0.005
0.008
0.009
(0.007)
(0.010)
(0.009)
0.006
0.008
0.008
(0.008)
(0.012)
(0.009)
0.006
0.009
0.008
(0.008)
(0.011)
(0.009)
0.006
0.009
0.008
(0.007)
(0.011)
(0.009)
22
The Third Study (1)
Are there conditions or practices that worsen
accuracy performance?
“[Simulating correlations] requires a large sample of random
values generated ahead of time. The values in the samples
are rearranged to create the desired correlations. [If the
“correlation sample size” is smaller than the total number of
trials] a next group of samples is generated and correlated.”
Crystal Ball 2000 Users Manual. pp. 246-7.
23
The Third Study (2)
Are there conditions or practices that worsen
accuracy performance?
“The sample size is initially set to 500. … While any sample
size greater than 100 should produce sufficiently acceptable
results, you can set this number higher to maximize accuracy.
The increased accuracy resulting from the use of larger
samples, however, requires additional memory and reduces
overall system responsiveness. If either of these become an
issue, reduce the sample size.”
Crystal Ball 2000 Users Manual. pp. 246-7.
24
The Third Study (3)
– “Correlation sample size” = 100—configure Crystal
Ball to apply the correlation algorithm 100 times,
to successive batches comprising 100 trials per
variable and 33 variables.
– Other third study design choices were identical to
their first study counterparts.
25
Third Study Results (1)
(First Study Results)
– About 31% (98%) of the 6336 simulated correlations
were within 0.03 of the target; 2817 (5) were outside
0.05 of the target—all were within 0.29 (0.06) of the
target.
– About 92% (74%) of the simulated correlations were
less than the target.
26
Third Study Results (2)
Error distribution by target (% of n = 2112)
Error band: target
to
Target
0.03
0.03
0.05
0.10
0.15
0.20
0.25
to
to
to
to
to
to
0.05
0.10
0.15
0.20
0.25
0.30
0.25
98
2
0
0
0
0
0
0.50
21
32
42
5
0
0
0
0.75
2
11
39
31
14
3
>0
27
Third Study Results(4)
(First Study Results)
Mean absolute error (n = 528)
Target:
Seed
0.25
0.50
0.75
1
0.012
0.052
0.107
(0.007)
(0.010)
(0.009)
0.012
0.058
0.110
(0.008)
(0.012)
(0.009)
0.011
0.048
0.096
(0.008)
(0.011)
(0.009)
0.011
0.052
0.097
(0.007)
(0.011)
(0.009)
2
1,048,576
2,097,152
28
The Fourth Study (1)
Does increasing the correlation sample size
from 100 to 500 well improve accuracy?
500 is Crystal Ball’s “installation default” for the correlation
sample—“The sample size is initially set to 500.” Source: Crystal
Ball 2000 Users Manual. pp. 246-7; also see the “Trials” tab in the Crystal Ball
“Run Preferences” menu.
29
The Fourth Study (2)
– “Correlation sample size” of 500—i.e., configure
Crystal Ball to apply the correlation algorithm 20
times, to successive batches comprising 500 trials
per variable and 33 variables.
– Examine only target correlation = 0.75, for which
we catastrophically lost accuracy in the third
study.
– Other fourth study design choices were identical
to their first and third study counterparts.
30
Fourth Study Results (1)
(First/Third Study Results for Target = 0.75)
– About 79% (99%/2%) of the 2112 simulated
correlations were within 0.03 of the target;
100% were within 0.09 (0.05/0.29) of the target.
– About 93% (74%/92%) of the simulated correlations
were less than the target.
31
Fourth Study Results (2)
Error distribution by target and sample (% of n = 2112)
Error band: target 0.03 0.05 0.10 0.15 0.20 0.25
to
to
to
to
to
to
to
Target (sample)
0.03 0.05 0.10 0.15 0.20 0.25 0.30
0.75 (500)
79
19
2
0
0
0
0
0.75 (100)
2
11
39
31
14
3
>0
0.75 (10,000)
99
1
0
0
0
0
0
0.25 (100)
99
1
0
0
0
0
0
32
Fourth Study Results (3)
Mean absolute error by target and
sample (n = 528)
Correlation
sample:
Target:
Seed
100
500
10,000
0.75
0.75
0.75
1
2
1,048,576
0.107
0.110
0.096
0.017
0.022
0.021
0.009
0.009
0.009
2,097,152
0.097
0.018
0.009
33
The Extended Fourth Study
– Side-by-side examination of “correlation sample
sizes” 100, 500, 1000, and 10,000 for target
correlation = 0.75 and “accuracy bands”  0.02,
 0.04, and  0.06.
– Other fourth study design choices were identical
to their first and third study counterparts.
34
Extended Fourth Study Results
How many correlation trials? What accuracy do you want?
% of correlations within band
100%
10,000 trials
1,000 trials
80%
60%
40%
500 trials
100 trials
20%
0.75± 0.02
0.75± 0.04
Accuracy band
35
0.75± 0.06
Lesson Learned (1)
 Don’t fear rank order correlation as a general principle;
Crystal Ball produced pretty accurate Pearson
correlations in the first and second studies.
This was surprising given the theory—only showing that we really
don’t understand the theory.
 Correlation accuracy collapsed after minimizing the
correlation sample size.
Accuracy fell apart asymmetrically, concentrating on where we
most need accuracy, among the larger correlations.
36
Lesson Learned (2)
 There was a clear tendency to undershoot target
correlations.
This may have been predictable. Strong Spearman rs can
accompany weak Pearson rs , but not vice-versa.
 Don’t put all of your simulations in one seed basket.
Seed = 2 provided somewhat weaker results than the other
seeds in the first study. Replicating the simulation using other
seeds exposed the weaker, atypical results.
37
Acknowledgements
To Ed Miller
for encouraging this study.
To Eric Wainwright and Decisioneering, Inc.
for supplying us a Crystal Ball.
38
References
Decisioneering, Inc. Crystal Ball 2000 User Manual. 19982000.
Iman, R.L. and W.J. Conover. “A distribution-free approach
to inducing rank correlation among input variables.”
Communications in Statistics, B11 (3), pp. 311-334, 1982.
Kelton, W.D. and A.M. Law. Simulation Modeling & Analysis.
New York: McGraw Hill, 1991.
39
The End
40