ETM 607 – Output Analysis:
Estimation of Relative Performance
• Output comparison between two or more alternative systems
• Common Random Numbers (CRN)
• Comparison of several systems
• Optimization via simulation
ETM 607 - Comparing Alternatives
•
•
•
Usually, simulation is used for more than just a
single model “configuration”
Often want to compare alternatives, select or
search for the best (via some criterion)
Simple processing system: What would happen
if the arrival rate were to double?



Cut interarrival times in half
Rerun the model for double-time arrivals
Make five replications
ETM 607 – Simple Processing
System
Single Server Queue
Arrival Rate ~Exp
Service Rate ~ Norm
Performance Measures:
• Average queue size
• Max queue size
• Average waiting time
• Max waiting time
• Average time in system
• Max time in system
• Server utilization
• Number served
ETM 607 - Results: Original vs. DoubleTime Arrivals
• Original – circles
• Double-time – triangles
• Replication 1 – filled in
• Replications 2-5 – hollow
• Note variability
• Danger of making
•
•
decisions based on one
(first) replication
Hard to see if there are
really differences
Need: Statistical analysis
of simulation output data
ETM 607 - Comparing Two Alternatives
•
General Idea: Approach I





Select some primary performance measure for comparison
(e.g. production rate, average time in system, server
utilization, etc…), denoted as Qi for alternative i.
Run alternative i for Ri replications to find the mean,
standard deviation and confidence interval for performance
parameter Qi.
If the CI’s of each alternative overlap, you cannot conclude
there is statistical difference between the alternatives.
What if you increased the number of replications?
What if you changed the significance level (a) of the CI?
ETM 607 - Comparing Two Alternatives
•
General Idea: Approach II





Select some primary performance measure for comparison
(e.g. production rate, average time in system, server
utilization, etc…), denoted as Qi for alternative i.
Run alternative i for Ri replications to obtain the
performance parameter Qri, where r refers to the
replication.
Find the difference between the performance measure for
each replication (Qr1 – Qr2).
Compute the mean, standard deviation and CI for (Qr1 –
Qr2).
If the CI of (Qr1 – Qr2) contains the value 0, then there is no
statistical difference between the two alternatives.
ETM 607 - Comparing Two Alternatives
Perform In-Class Lab Exercise 11,
Comparison of Two Alternatives
ETM 607 - Comparing Two Alternatives:
Common Random Number (CRN)
• Step 4 of Lab 11 was possible because the same
•
•
random number stream was used to compare each
alternative. (The only change between both systems
was the arrival rate).
This technique is referred to as Common Random
Numbers (CRN).
Usually possible only when changing parameter
values between alternatives. If logic changes, or the
execution of random numbers, then this technique is
not valid.
ETM 607 - Comparing Two Alternatives:
Common Random Number (CRN) cont.
• For CRN case:
Y
.1



s.e. Y.1  Y.2 
where
for differences
and

 Y.2  ta / 2,v s.e. Y.1  Y.2
v  R 1
_
__
__
SD
R

, if Dr  Yr1  Yr 2
D  Yi1  Yi 2 and
__
1
R
S 
( DR  D)

r 1
R 1
2
D
ETM 607 - Comparing Two Alternatives:
Common Random Number (CRN) cont.
• For more general case when CRN not valid:
Y
.1
where


 Y.2  ta / 2,v s.e. Y.1  Y.2


S12 S 22

R1 R2
s.e. Y.1  Y.2 
2
S

S
 R  R 
1
2

v
 S 2  2
  S 2  2

1
2
 R  / R1  1   R  / R2  1
1
2

 

2
1
And

2
2
rounded to integer
ETM 607 – Comparison of Several
System Designs
•
•
How do you compare multiple system designs?
Depends on the goal:
1.
2.
3.
4.
•
•
Estimation of the parameter, Qi for alternative i.
Comparison of Qi to some control Q1 which might be an
existing system.
All pairwise comparisons Qr1 – Qr2.
Selection of the best Qi .
Goals 1,2, 3 all use CI’s. Goal 4 requires a type of
optimization approach.
Performing analysis on multiple alternatives is
somewhat laborious, but some simulation packages
facilitate this process. See In-class Lab 11a.
ETM 607 – Optimization via Simulation
•
•
•
Goal 4 was to find the best Qi .
What if there are many (thousands or millions) of
alternatives to choose from?
One approach is to use meta-heuristics.
1.
2.
3.
Register for ETM 645 to learn more about meta-heuristics.
Attend the last lesson to observe an application of
optimization via simulation in the scheduling of Radar
Warning Receivers.
Arena comes with a tool called “OptQuest”. Download via
Google.
ETM 607 – Optimization via Simulation
Perform InClass Lab 11a –
Comparison of multiple alternatives