A Genetic Algorithm Approach
to Multiple Response
Optimization
Francisco Ortiz Jr.
James R. Simpson
Joseph J. Pignatiello, Jr.
Alejandro Heredia-Langner
Motivation
• Many industrial problems involve many response
solutions
– “It is not unusual to find RSM applications with 20 or more
responses…” (Montgomery, JQT, 1999)
• Current methods don’t always work
• Work by Heredia-Langner, and suggestions from
Carlyle, Montgomery and Runger (JQT, 2000)
Department of Industrial Engineering
Florida A&M and Florida State University
Composites
Production
INPUTS (Factors)
OUTPUTS (Responses)
Fiber Permeability
Resin Flow Rate
Type of Resin
PROCESS:
Tensile Strength
Gate Location
Flexural Strength
Fiber Weave
Mold Complexity
Fiber Weight
Resin Flow Properties
Resin
Transfer
Molding
Dynamic Mechanical Analysis
Compression Strength
Acoustic Properties
Department of Industrial Engineering
Florida A&M and Florida State University
Laser Toner
Development
INPUTS (Factors)
OUTPUTS (Responses)
Spitting
Charge Control Agents
Release Agents
PROCESS:
Background
Developer Roll Mass/Area
Colorants
Toner-to-Cleaner
Surface Additives
Charge Voltage
Transfer Voltage
Toner
Transfer
Powder Flow
Isopel Optical Density
.
.
.
24 responses
Film Onset
Department of Industrial Engineering
Florida A&M and Florida State University
Challenges for Current Multiple
Response Optimization Methods
• As the number of response and decision variables increases
– The combined response function typically can be highly
nonlinear, multi-modal and heavily constrained
– Conventional optimization methods can get trapped at a
local optimum and even fail to find feasible solutions
Department of Industrial Engineering
Florida A&M and Florida State University
Purpose
• This paper develops and evaluates a multiple
response solution technique using a genetic
algorithm in conjunction with an unconstrained
desirability function
Department of Industrial Engineering
Florida A&M and Florida State University
Topics to Cover
• A desirability function for the genetic algorithm
• “Robust-ising” the genetic algorithm for multiple
response optimization
• Performance evaluation and comparison
Department of Industrial Engineering
Florida A&M and Florida State University
Combining Responses
• Most methods convert multiple responses with
different units of measurements into a single
commensurable objective
• Distance or Loss Functions
– Khuri and Conlon, 1981
– Pignatiello, 1993
– Vining, 1998
• Desirability Methods
– Derringer and Suich, 1980
– Del Castillo, Montgomery, and McCarville, 1996
• Desirability approach converts individual response
ŷi into individual desirability di(ŷi)
• Used in combination with optimization algorithm to
locate a single solution
Department of Industrial Engineering
Florida A&M and Florida State University
Desirability Approach
• For example, target as the goal response
• Constraint developed based on response’s utility
• 0 < di(ŷi) < 1
si
ti
 yˆ  L  si
i
 i
 , Li  yˆ i  Ti
 Ti  Li 

ti
 yˆ i  H i 
d i ( yˆ i )  
 , Ti  yˆ i  H i
T

H
i 
 i

0
, otherwise



• Overall desirability
1/ m
D  x    d1  yˆ1  d 2  yˆ 2   d m  yˆ m  
• Optimization using Nelder-Mead simplex or GRG
Department of Industrial Engineering
Florida A&M and Florida State University
Genetic Algorithm
• A genetic algorithm (GA) is a search technique that is based
on the principles of natural selection
– “survival of the fittest”
• Uses the objective function magnitude directly in the search
• The GA generates and maintains a population pool of
solutions throughout
• It then evaluates the quality of each individual chromosome
using a fitness function
– In this case the desirability function = fitness function
Department of Industrial Engineering
Florida A&M and Florida State University
Using the GA in Response
Surface Studies
• A designed experiment produces an empirical system model
yˆ  f ( x1 ,..., xk )
• A coding scheme widely used is to transform the natural
variables into coded variables that fall between -1 and 1
Chromosome
x1
-0.451
x2
0.346
x3
0.518
x4
0.701
x5
-0.573
• Here we have multiple fitted responses
yˆ  [ yˆ1 , yˆ 2 ,..., yˆ m ]
Department of Industrial Engineering
Florida A&M and Florida State University
x6
-0.634
x7
0.937
Gene
x8
-0.448
Limitations of Current Desirability
for the GA
• Multiplicative desirability functions
– Overall desirability is
D  x    d1  yˆ1  d 2  yˆ 2   d m  yˆ m  
1/ m
– If any di (x) = 0, D(x) = 0
– The GA is unable to compare
infeasible solutions
– Perhaps we could extend the
D(x) function
Department of Industrial Engineering
Florida A&M and Florida State University
Formulating a Desirability
Function
• GA can be designed to handle constraint violations by using a
penalty method
D* (x)  DDS (x)  P(x)
• where
P  x    p1 ( yˆ1 ) p2 ( yˆ2 )... pm ( yˆ m ) 

1/ m
Department of Industrial Engineering
Florida A&M and Florida State University
 c

2
Formulating a Desirability
Function
• Every fitted response has an individual desirability di(ŷ) and a
penalty pi(ŷ)
 yˆ  H ti
i
 i
 , Ti  yˆi  H i
 Ti  H i 

si
 yˆi  Li 
di ( yˆi )  
 , Li  yˆi  Ti
T

L
i 
 i

0
, otherwise




c 


pi ( yˆi )  

c 

yˆi  Li
Ti  Li
c
yˆi  H i
Ti  H i
• where c is a small constant (e.g. c= 0.0001)
Department of Industrial Engineering
Florida A&M and Florida State University
,    yˆi  Li
, Li  yˆi  H i
, H i  yˆi  
Formulating a Desirability
Function
• Hence, the overall unconstrained desirability function is
D (x)   d ( y1 )  d ( ym ) 
*
1/ m
  p( y1 )  p( ym ) 

1/ m
 c

2
• The penalty function enables the GA to find feasible solutions
• After feasible solutions are found, P(x) = 0, and no effect on
the D*(x)
Department of Industrial Engineering
Florida A&M and Florida State University
Unbounded Desirability Function
• Overall GA desirability function for a case with one response and
linear weights (s1 = t1= 1) for the d(y1)
D*
Department of Industrial Engineering
Florida A&M and Florida State University
Desirability Function Investigation
• Proposed desirability function D*(x) vs. DDS (x)
Case # 1 Chromosome #1
DDDS (x
(x))
DS
Responses
ŷ1
ŷ2
ŷ3
ŷ4
Value
60
75
50
30
Low
50
50
47
40
Target
60
75
50
45
High
70
100
53
50
di
1
1
1
0
Overall Desirability
0
D * ( x)
di
1
1
1
0
D * ( x)
pi
0.000001
0.000001
0.000001
2.000001
-1.34E-09
Case # 1 Chromosome #2
DDDS (x
(x))
DS
Responses
ŷ1
ŷ2
ŷ3
ŷ4
Value
60
45
60
30
Low
50
50
47
40
Target
60
75
50
45
High
70
100
53
50
Overall Desirability
Department of Industrial Engineering
Florida A&M and Florida State University
di
1
0
0
0
0
D * ( x)
di
1
0
0
0
D * ( x)
p i (penalty)
0.000001
0.200001
2.333334
2.000001
-9.66E-04
GA Performance Investigation
• Proposed desirability function D*(x) vs. DDS (x)
Desirability vs. Generations
1.2
1
Desirability
0.8
0.6
GA DF
D-S DF
0.4
0.2
0
0
5
10
15
20
25
30
-0.2
Generations
Department of Industrial Engineering
Florida A&M and Florida State University
35
40
45
50
Tuning the GA Parameters
• The GA can be sensitive to the many parameter choices that must
be made for its application
• The GA parameters considered for this study are
– Population size
– Parent-to-Offspring ratio
– Selection type
– Mutation type
– Mutation rate
– Crossover rate
x1
-0.451
x2
0.346
x3
0.518
x4
0.701
x5
-0.573
x6
-0.634
x7
0.937
x8
-0.448
x1
-0.257
x2
-0.914
x3
0.028
x4
0.383
x5
0.429
x6
-0.111
x7
-0.207
x8
-0.812
Department of Industrial Engineering
Florida A&M and Florida State University
Creating Multiple Response
Problem Scenarios
• Done by incorporating four problem environment parameters
incorporated into a designed experiment framework
• Considered noise variables for the purposes of robust design
Factor
Problem Parameters
A
Number of Decision Variables
B
Number of Responses
C
Percent of 2nd order Models
D
Constraints (%of target)
Department of Industrial Engineering
Florida A&M and Florida State University
Low
4
4
25
5
High
8
16
75
15
Creating Multiple Response
Problem Scenarios
• Certain rules were used to ensure that the problems
generated mimic what typically is found in a real world
situation
• Specifications
– The decision variables that appear in each response are
chosen randomly
– For second order models, ½ of the terms are linear (main
effects), ¼ of the terms are interactions, ¼ of the terms
are pure quadratic
– The exact values of the regression coefficients are
selected from a uniform probability distribution U(5, 20)
– Model hierarchy is always maintained
Department of Industrial Engineering
Florida A&M and Florida State University
Creating Multiple Response
Problem Scenarios
• For example, a case with 4 responses and 4 decision
variables, 75% of the responses are second order models
yˆ1  50  16 x1  6 x2  12 x1 x2  7 x12
yˆ 2  50  19 x2  8 x3  13 x2 x3
yˆ3  50  19 x1  18 x2  8 x1 x2  11x32
yˆ 4  50  7 x3  19 x4  10 x3 x4  14 x42
Department of Industrial Engineering
Florida A&M and Florida State University
Robust Designed Experiment
• Robust parameter design using a combined array 210-4
resolution IV (80 runs includes 16 pseudo-centers)
Factor
A
B
C
D
E
F
G
H
J
K
GA Parameters
Parent Pop. Size
Parent/Offspring Ratio
Selection Type
Crossover Rate
Mutation Type
Mutation Rate
Problem Parameters
Number of Decision Variables
Number of Responses
Percent of 2nd order Responses
Constraints (% of target)
Department of Industrial Engineering
Florida A&M and Florida State University
Low
20
1:1
Ranking
0.5
Uniform
0.1
Low
4
4
25
5
High
50
1:07
Tournament
0.85
Gaussian
0.4
High
8
16
75
15
Type
Control
Control
Control
Control
Control
Control
Noise
Noise
Noise
Noise
Robust Designed Experiment
Results
• GA performance metrics were the number of evaluations until
– Feasible
– Within 10% of optimal, or D*(x) > 0.90
• Response model of the form
6
6
yˆ  x, z   b0   bi xi  
i 1
6

i 1 j 1, j i
6
4
bij xi x j  ci zi   dij xi z j
• Significant effects included
– Noise factor main effects
– Control x control interactions
– Control x noise interactions
Department of Industrial Engineering
Florida A&M and Florida State University
4
i 1
i 1 j 1
Robust Designed Experiment
Results
• Response: Achieving D*(x) > 0.90
Term
AC
Term Type
Control x Control
AE
Control x Control
AF
Control x Control
BK
Control x Noise
EG
Control x Noise
EK
Control x Noise
AEK
Control x Noise
CFK
Control x Noise
Finding
For parent population = 20, selection type does not matter, whereas if the
parent population = 50, choose tournament selection
Better performance is achieved with parent population = 20 and Gaussian
mutation
Performance is enhanced using the parent population = 20 and a mutation
rate = 0.4
An offspring ratio of 1:7 is best with tight constraints (5%), but as the
constraints widen, the offspring ratio is unimportant
Gaussian mutation works better with only 4 decision variables and both
mutation methods perform equally with 8 decision variables
Gaussian performs better with tight constraints (5%), but as the constraints
widen, the mutation type is insignificant
For mutation rates of 0.1, use a parent population = 20 and Gaussian
mutation. For all other mutation rates, factor AE is not significant
For tournament selection, use mutation rate = 0.4 regardless of constraint
setting. For ranking selection, the FK interaction is not significant
Department of Industrial Engineering
Florida A&M and Florida State University
Robust GA Parameter Settings
• All but one determined by response surface models
Factor
Parameters
Setting
Rationale
A
Parent Population
20
Designed experiment result
B
Parent/Offspring Ratio
1:7
Designed experiment result
C
Selection Type
D
Crossover Rate
0.85
Diversify offspring pool
E
Mutation Type
Gaussian
Designed experiment result
F
Mutation Rate
0.4
Designed experiment result
Department of Industrial Engineering
Florida A&M and Florida State University
Tournament Designed experiment result
Performance Evaluation
• Evaluate and compare proposed GA method
• Considered the multiple response problem scenarios
– Dropped percentage of second order models
– 23 factorial plus a center point
Case #
1
2
3
4
5
6
7
8
9
Factor A
Factor B
No. of Decision Variables No.of Responses
4
4
8
4
4
16
8
16
4
4
8
4
4
16
8
16
6
10
Department of Industrial Engineering
Florida A&M and Florida State University
Factor C
Constraint Range
5
5
5
5
15
15
15
15
10
GRG Performance
• Results of investigation using 30 starting point locations
Pct of GRG Solutions
Case # No. of Variables No.of Responses Constraints
Optimal
1
4
4
5
0.30
2
8
4
5
0.07
3
4
16
5
0.07
4
8
16
5
0.00
5
4
4
15
0.83
6
8
4
15
0.27
7
4
16
15
0.57
8
8
16
15
0.00
9
6
10
10
0.07
Department of Industrial Engineering
Florida A&M and Florida State University
GA Performance Investigation
• Performance of proposed GA using four replicates
Case #
1
2
3
4
5
6
7
8
9
No. of Variables No.of Responses Constraints Mean
4
4
5
0.989
8
4
5
0.969
4
16
5
0.997
8
16
5
0.987
4
4
15
0.998
8
4
15
0.994
4
16
15
0.987
8
16
15
0.958
6
10
10
0.979
Department of Industrial Engineering
Florida A&M and Florida State University
(95.0% )Confidence Intervals
St. Dev.
High
Low
0.004
0.994
0.985
0.011
0.979
0.958
0.002
0.998
0.995
0.005
0.991
0.982
0.001
0.998
0.997
0.003
0.997
0.992
0.004
0.991
0.983
0.027
0.984
0.931
0.016
0.995
0.962
Why Use the GA for Multiple
Response Optimization?
• Current study shows consistent, effective performance
• Can be effectively combined with direct search methods
(e.g. GA with GRG) to improve run-time performance
• Flexible to handle a host of objective functions
– Distance or loss functions can all be applied as long as
covariance information is available
• Able to perform reasonable mapping of response function
over design space
Department of Industrial Engineering
Florida A&M and Florida State University
•
2.4 Genetic Algorithm
– Coding
00001010000110000000011000101010001110111011
is partitioned into 2 halves
0000101000011000000001 and 1000101010001110111011
these strings are then converted from base 2 to base 10 to yield:
x1 = 165377 and x2 = 2270139
The following is an example of real-value coding
0.125
-1.000
0.525
Department of Industrial Engineering
Florida A&M and Florida State University
•
Recombination
–
Exchange information/genes between parent chromosome to make new offspring
Parent 1
2500
85
1780 103 1200
95
1900
98
2500 168
500
40
Crossover Point
Parent 2
1850
53
2200
99
785
67
1000 102
750
75
900
69
2500
85
1780 103 1200
95
1900
750
75
900
69
Offspring
• GA cannot rely on recombination alone.
Department of Industrial Engineering
Florida A&M and Florida State University
98
•
Mutation
–
–
–
Uniform mutation
Multiple uniform mutation
Guassian mutation
• All entities of the chromosome are mutated such that the resulting chromosome lies
somewhere within the neighborhood of it parent.
X1
X2
Department of Industrial Engineering
Florida A&M and Florida State University