Fuzzy inference
Cold
Warm
17
20
Hot
26
Cloudy
Partial
30
29
Sunny
50
100
Implication
Fuzzyfication
Low
Medium
48
High
Defuzzyfication
.
.
Fuzzy inference systems
Knowledge base
Database
Rule base
Crisp
Input
Fuzzifier
Fuzzy
Input
Inference
Engine
Fuzzy
Output
Defuzzifier
Crisp
Output
Fuzzyfier: translates crisp inputs into fuzzy values
Inference engine: applies reasoning to compute fuzzy outputs
Defuzzyfier: translates fuzzy outputs into crisp values
Knowledge base: defines rules and membership functions
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Network−like view of a fuzzy system
.
Low
Pressure
&
High
High
OR
Gas
&
Low
Cold
OR
Temp.
Hot
&
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
Operational parameters
Low
Pressure
&
High
High
OR
Gas
&
Low
Cold
OR
Temp.
Hot
&
Membership function values
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
Connective parameters
Low
Pressure
&
High
High
OR
Gas
&
Low
Cold
OR
Temp.
Hot
&
Weights
Consequents
Antecedents
Rules
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
Structural parameters
Low
Pressure
&
High
High
OR
Gas
&
Low
Cold
OR
Temp.
Hot
&
{
Number of
Relevant variables
Rules
Membership functions
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
Logical parameters
Low
Pressure
&
High
High
OR
Gas
&
Low
Cold
OR
Temp.
Hot
&
Reasoning mechanism
Fuzzy operators
Membership function types
Defuzzification method
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Parameters of a fuzzy system
Class
Parameters
.
Component
Reasoning mechanism
Fuzzy operators
Logic
Inference engine
Membership function types
Fuzzi- and defuzzifier
Defuzzification method
Defuzzifier
Relevant variables
Number of membership functions
Structural
Knowledge base
Number of rules
Antecedents of rules
Connection
Consequents of rules
Rulebase
Rule weights
Operational
Membership function values
Database
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
Dual external nature
Numeric
Linguistic
If Temperature is COOL
then Ventilator is Off
If Tempearture is WARM then Ventilator is Low
If Temperature is HOT
then Ventilator is Medium
If Temperature is VERY−HOT then Ventilator is High
If Temperature is HELLISH then Ventilator is Off, and...
... Let’s go to the lake!!!
Precision
Interpretability
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Numeric issues
●
●
●
●
.
Numeric mapping: Crisp inputs / Crisp outputs
Nonlinear behavior, but linearity not excluded
Universal approximator
Uncertainty management: noise and low quality of data
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
Interpretability considerations: semantic criteria
.
Semantics: the study of meanings
● Distinguishability: Each linguistic label has semantic meaning
● Number of elements: Compatible with human capabilities
● Coverage: Any element belongs to at least one fuzzy set
● Normalization: At least one element has unitary membership
● Complementarity: For each element, the sum of memberships is one
Cold
Cool
Warm
Hot
1
Cold
Cool
Temperature
0
Warm
Hot
1
Temperature
0
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Interpretability considerations: syntactic criteria
.
Syntax: the way in which linguistic elements are put together
● Completeness: for any input, at least one rule must fire
● Rule−base simplicity: Set of rules as small as possible
● Rule readability: small number of conditions in rule antecedents
● Consistency: rules firing simultaneously must have similar consequents
R7
R8
R9
R4
R5
R6
R4
R5
R1
R2
R3
R1
R2
RB
RB
R5
RA
RA
R0
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Strategies to satisfy interpretability criteria
.
● Linguistic labels shared by all rules
● Normal, orthogonal membership functions
● Don’t care conditions
● Default rule
Cold
Warm
RB
Hot
R5
RA
R0
17
20
26
29
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
The general modeling problem
.
What do you know about the modeled system?
i.e. what is predefined and what looked for?
Search space
How do you search?
i.e. do a well suited and/or well known technique exists?
Search method
Have you preferences or restrictions to the model?
i.e. do issues like size, speed, or simplicity matter?
Constraints
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
Search space in fuzzy modeling
The number of parameters is too high to perform a full search,
some parameter pre−definition is thus required.
According with the searched parameters we can have:
Operational parameters:
Knowledge tuning.
Knowledge base
Database
Rule base
Fuzzifier
Inference
Engine
Defuzzifier
P1 P2 P3
P4
Connective parameters:
Behavior learning.
If V1 is Low AND ....
Structural parameters:
Structure learning.
R1, ... , Rn
f1, ... , fm
Logical parameters:
System design.
and, or, not, ...
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Search methods: Fuzzy modeling techniques
.
Knowledge engineering
"Classic" identification methods
Machine learning approaches
Neuro−fuzzy systems
Evolutionary fuzzy modeling techniques
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Usual constraints in fuzzy modeling
.
What is the fuzzy system expected to do?
− Classification: Classification performance, quadratic error.
− Control: Dynamic response, adaptability, robustness, etc.
− Diagnostic: Overall performance, sensitivity, specificity
− Data mining: Completness, complexity.
How is the system expected to do it?
− Speed: Real−time constraints, computing resources.
− Size: Available memory, computing platform.
Who is going to interact with the system?
− Interpretability: Allowed complexity.
− Availability: Continuity of explanations (time to provide them)
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Interpretability−related constraints
.
Rule−specific MFs are not allowed
all rules share the same MFs
Orthogonal MFs with well defined
null and unity membership
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
Fuzzy modeling: direct approach
Domain
expert
Knowledge
engineer
Fuzzy
model
Design loop
Validation loop
This approach is also called knowledge engineering
Fuzzy modeling: data-driven approaches
Domain data
Building algorithm
Fuzzy
model
Design loop
Validation loop
Domain expert
This approaches are also denominated
knowledge discovering
.
Fuzzy modeling: some data−driven approaches
Identification−based
Neuro−fuzzy systems
Fuzzy system
Human
design
Logic
Structural
Estimation
algorithm
Connective
Operational
.
Fuzzy system
ANN−like training algorithm
Constructive−learning
Fuzzy system
Structural
Connective
Operational
Data
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Evolutionary knowledge tuning (database)
X2
.
Big
Q3
R3
Normal
Q2
R2
Small
Q1
- Fixed rule base (completness)
R1
R6
R9
- Rules of type:
R5
R8
R4
R7
if X1=Low and X2=Normal then Output = Ci
- Knowledge is tuned by evolution,
which searches for membership function values
P1
P2
P3
X1
- Genome encodes values P, Q, and C
Low Mid
High
(3*P + 3*Q + 9*C) * 5 bits = 75 bits
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
Evolutionary behavior learning (rule base)
- Number of rules exploses rapidly
- Two strategies for reducing this number:
Don’t care conditions and default rule
- Evolution can be used to find a minimal (or fixed size) rule base,
- Three main approches to evolutionary behavior learning
Michigan
Pittsburgh
Individual = One rule
Population = Rule base
(i.e. fuzzy system)
Individual = Entire system
(rule base or knowledge base)
Population of systems
Iterative Rule Learning
Evolution finds the best rule
Incremental construction of
the knowledge base
R1
R2
R1
R2
R3
Ri
R3
Ra, Rb, Rc ...
Ri
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
Evolutionary behavior learning: An example
.
- 4 input variables, 5 membership functions per variable
{Tiny, Small, Normal, Big, Huge}
- Space of 625 rules (1295 including don’t care conditions)
IF V1 is Tiny AND ... AND V4 is Normal then Out = Huge
- Evolution searches for a subset of N rules (fixed by the designer)
- Genome encodes rules: Antecedents and consecuents
N rules * 15 bits
R1
R2
....
Ri
....
Rn
15 bits
A1
A2
A3
A4
C
5 * 3 bits
5 functions + 3 don’t care
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
Evolutionary knowledge base learning
(Knowledge base = rule base + database)
Parameter
class
Modeling
type
Usual
quantity
Type of
values
Fuzzy system
attribute
Connective
Behavior learning
10 - 1000
Symbolic
Rule base
Operational
Knowledge tuning
10 - 1000
Real-valued
Database
Critical issues for applying evolution:
- Parameter representation
- Tight interdependency
- Size of the search space
- Computation requirements
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
Evolutionary knowledge base learning
- A basic approach: Single population, single evolution
Example: Breast cancer diagnosis problem (Peña and Sipper 99)
- 9 inputs, 1 output, 2 membership functions per variable
- A simple genetic algorithm searches for the knowledge base
- Genome encodes: Rule antecedents and membership function parameters
9 Variables * 6 bits
V1
V2
....
Vi
....
Nr rules * 18 bits
V9
R1
R2
....
6 bits
Ri
....
Rn
....
A9
Low
High
18 bits
P
d
3 bits
3 bits
A1
....
Ai
9 antecedents * 2 bits
P
d
Sample ru;e: IF V1 is Low and V4 is High THEN Diagnostic is Benign
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
Evolutionary knowledge base learning
.
- A variation: Single population, double evolution
Example: Evolving fuzzy rule based classifiers with GA–P (García et al. 99)
- Genome encodes: Complete rule base and membership function parameters
- A simple genetic algorithm searches for the database
- The rule base is evolved using genetic programming
1 0 0 1 0 0 0 1 1 0 0
Numeric part: Database
Symbolic part: Rule base
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
Evolutionary knowledge base learning
- Hybrid learning: Evolved rule base, learned database
Example: Breast cancer diagnosis (J.-F. Philagor, student project SPG, 1999)
- Evolution searches for a fixed-size rule base
Genome encodes rules: Antecedents and consequent
- Database is tuned using a neuro-fuzzy approach
A fuzzy self-organizing map searches for P and Q values
....
Ri
....
Rn
19 bits
A1
A2
...
A9
C
Q3
R3
R2
Q2
R2
Small
R1
R1
Q1
N rules * 19 bits
Normal Big
X2
P1
9 * 2 bits + 1 bit
P2
Low Mid
P3
X1
High
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
The database
The test
The features
The Wisconsin Breast Cancer Database
.
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
Proposed Fuzzy System
...
...
...
...
...
R1: if (V1 is Low) and (V2 is High) and ... and (V9 is Low) then (output is Benign)
R2: if (V1 is Low) and (V2 is Low) and ... and (V9 is None) then (output is Benign)
else (Output is Malignant)
Low
High
Benign
Low
.....
Low
Low
None
Benign
.....
V1
V2
V9
Malignant
P1
P1+d 1
P2
P2+d 2
P9
P9+d 9
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
Genome encoding
9 Variables * 6 bits
V1
V2
....
Vi
....
Nr rules * 18 bits
V9
R1
R2
....
6 bits
d
3 bits
3 bits
d = [1;8]
....
Rn
....
A9
18 bits
P
P = [1;8]
Ri
A1
....
Ai
9 antecedents * 2 bits
Ai = 1 (Benign)
Ai = 2 (Malignant)
Ai = 0 or 3 (Variable not assigned)
Total genome length = 54 + 18Nr bits
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
Fitness function
.
Fc : Classification performance,
the most important performance measure
F = Fc + a* Fv + b*Fe
Fv : Number of variables
measures the interpretability
Fe : Quadratic error
selection pressure to fine tune parameters
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
Results: Classification performance (Number of variables)
Rules Setiono (96)
1
Setiono
Liu (96)
Taha
Ghosh (97)
95.42% (2)
2
3
97.14% (4)
Peña
Sipper (98)
This
work (99)
96.35% (3)
97.07% (4)
96.65% (7)
97.36% (3)
97.80% (4.7)
97.21% (4)
4
5
.
97.80% (4.8)
97.51% (3.4)
96.19% (1.8)
Learned Boolean rules
Evolved fuzzy rules
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
.
The best single-rule system
Low
Low
v1
Low
v2
Low
v6
Benign
v8
Malignant
IF the clump of cells is not very thick,
AND the cell’s size is quite uniform,
AND there are few bare nuclei,
AND nucleoli are not highly abnormal,
THEN the case is benign;
ELSE the case is malignant.
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
Proposed approach: two elements
.
1 A system model: Fuzzy systems
2 A building algorithm: Cooperative coevolution
Fuzzy
Database
CoCo
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Fuzzy modeling: a coevolvable problem
Parameter
class
Modeling
type
Usual
number
Logical
System design
3 − 10
Structural (size)
Structure learning
5 − 20
Integer
Connective (rules)
Behavior learning
10 − 1000
Symbolic
Operational (labels)
Knowledge tuning
10 − 1000
Real−valued
The required solutions can be very complex,
they can be decomposed in distinct components,
represented by different types of values,
and which are very interdependent.
.
Type of
values
These features render
fuzzy modeling an
adequate target for
COOPERATIVE
COEVOLUTION
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Fuzzy CoCo:
.
A cooperative coevolutionary approach to fuzzy modeling
Membership functions
Evaluation
Rules
Two evolutionary algorithms
searching for:
membership functions
Evaluation
and rules.
Selection
Modification
Advantages:
Selection
Modification
− Divide−and−conquer strategy
− Better search power
− Lesser computational cost
− More−flexible setup
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Fitness evaluation in Fuzzy CoCo
1. Cooperators for generation g
are selected from generation g−1
both fitness−dependent and
randomly
Rules
MFs
Rules
Fitness
Cooperators
MFs
g−1
Generation
Cooperators
Fitness
2. Individuals are combined with cooperators
to form fuzzy systems.
g
Selected cooperators
3. These fuzzy systems are evaluated, and
individual fitness is then calculated.
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
Interpretability strategies in Fuzzy CoCo
.
● Shared membership functions: reinforced by the existence of a separate species
● Normal, orthogonal membership functions: constrained representation
● Don’t care conditions: encourage shorter rules
● Default rule: guarantees complete coverage of the input space
● Linguistic fitness: when used, increases selective pressure for interpretability
Cold
Cool
Warm
RB
Hot
1
R5
RA
R0
Temperature
0
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
.
Features
Classes
(1) SL Sepal length
(2) SW Sepal width
(3) PL Petal length
(4) PW Petal width
(1) setosa
(2) versicolor
(3) virginica
The database
The variables
Fisher’s Iris Data
Case
SL
SW
PL
PW
Class
1
2
5.1
4.9
3.5
3.0
1.4
1.4
0.2
0.2
Setosa
Setosa
150
5.9
3.0
5.1
1.8
Virginica
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Iris: Variable analysis
.
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Iris proposed solution: Controller−type
Fuzzy Subsystem
Input
Class
Knowledge base
Database
Stair−function
Subsystem
.
Class
Rule base
estimation
Crisp
Input
Fuzzifier
Fuzzy
Input
Inference
Engine
Defuzzifier
Fuzzy
Output
Crisp
Output
The fuzzy subsystem estimates a continuous "class" value
The selection unit approximates it to the nearest class
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Iris controller−type: Proposed Fuzzy System
.
R1: if (SL is A11) and (SW is A12) and (PL is A13) and (PW is A14) then (output is Class1)
...
...
...
...
...
R2: if (SL is A21) and (SW is A22) and (PL is A23) and (PW is A24) then (output is Class2)
Rn: if (SL is An1) and (SW is An2) and (PL is An3) and (PW is An4) then (output is Classn)
else (Output is Class0)
Low
High
Low
setosa
.....
1
Low
Medium
None
2
3
versicolor
.....
SL
SW
PW
1
2
3
virginica
P11 P21 P31
P12 P22 P32
P14
P24
1
2
3
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Iris proposed solution: Classifier−type
µ (setosa)
Fuzzy Subsystem
Input
Knowledge base
µ (versicolor)
Database
Maximum and
Threshold Subsystem
.
Class
Rule base
Crisp
Input
Fuzzifier
Fuzzy
Input
Inference
Engine
Defuzzifier
Fuzzy
Output
Crisp
Output
µ (virginica)
The fuzzy subsystem estimates a continuous membership
value for each class
The selection unit chooses the most active class
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Iris classifier−type: Proposed Fuzzy System
.
R1: if (SL is A11) and ... and (PW is A14) then (setosa is Yes),(versicolor is No),(virginica is No)
...
...
...
...
...
R2: if (SL is A21) and ... and (PW is A24) then (setosa is No),(versicolor is Yes),(virginica is Yes)
Rn: if (SL is An1) and ... and (PW is An4) then (setosa is No),(versicolor is No),(virginica is Yes)
else (setosa is No),(versicolor is Yes),(virginica is No)
Low
Low
.....
Medium
None
setosa
versicolor
virginica
No Yes
No Yes
No Yes
setosa
versicolor
virginica
No Yes
No Yes
No Yes
setosa
versicolor
virginica
No Yes
No Yes
No Yes
.....
SL
P11 P21 P31
PW
P14
P24
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
Iris: the genomes
.
Membership functions
Rules (Controller/Classifier)
Nr rules * 19 bits
4 Variables * 15 bits
SL
SW
PL
SW
P
.
R1
...
3 x 5 bits
Ri
...
Rn
2/3 bits
10/11 bits
P1
P2 P3
5
5
A1
4 * 2 bits
5
Genome length = 60 bits
....
Co
A4
Ci
2/3 bits
Genome length = 10/11*Nr + 2/3 bits
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Iris: Fuzzy CoCo set−up
.
1. Fitness function
Fc * Fmb
{
F=
(Fc + a*Fv) * Fmb
Fc : Classification performance,
the most important
Fm : 1 − mse (mean square error)
encourages not−so−bad errors
Fv : Number of variables
measures the interpretability
2. Fuzzy CoCo parameters
Population size
Maximum generations
Crossover probability
Mutation probability
Elitism rate
"Fit" cooperators
Random cooperators
{60, 70}
500 + 100*Nr
1
{0.02, 0.05, 0.1}
{0.1, 0.2}
1
{1, 2}
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Controller
Iris results: classification (average rule)
2
Fuzzy CoCo
ICANNGA’01
99.33% (2)
3
100 % (1.7)
Rules
4
Simple GA
Shi et al (1999)
98 % (2.6)
100 % (2.5)
100 % (3.3)
5
Classifier
FuGeNeSys
Russo (1998)
Constructive Learning Methods Neurofuzzy
Rules Hong (00)
Wu (99)
Hung (99)
2
96.2 % (4)
3
97.4 % (4)
4
8
.
Fuzzy CoCo
ICANNGA’01
98 % (1.5)
99.33% (2.3)
99.33% (2)
97.3 % (2)
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Iris controller−type: A three−rule system
SL
SW
PL
PW
Class
SL
SW
PL
PW
Class
SL
SW
PL
PW
Class
.
Class
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
.
Iris classifier−type: A three−rule system
SL
SW
PL
PW
setosa
versic.
virgin.
SL
SW
PL
PW
setosa
versic.
virgin.
SL
SW
PL
PW
setosa
versic.
virgin.
setosa
versic.
virgin.
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Proposed
solution
The test
The database
Breast cancer diagnosis: the WBCD problem
.
Input
Fuzzy Subsystem
Appraisal
Threshold Subsystem
Diagnostic
malignant
benign
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
The genomes
Membership functions
Rules
9 Variables * 6 bits
V1
V2
....
Vi
....
Nr rules * 19 bits
V9
R1
...
6 bits
Ri
...
Rn
19 bits
P
d
3 bits
3 bits
A1
....
A9
9 * 2 bits
Ai = 1 (Low)
P = [1;8]
Ai = 2 (High)
d = [1;8]
Ai = 0 or 3 (None)
Genome length = 54 bits
Co
1 bit
Ci
1 bit
Ci = 1 (Benign)
Ai = 2 (Malignant)
Genome length = 19*Nr + 1 bits
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Fuzzy CoCo set−up
.
Fitness function
F = Fc − a* Fv
Fc : Classification performance,
the most important performance measure
Fv : Number of variables
measures the interpretability
Fuzzy CoCo parameters
Population size
Maximum generations
Crossover probability
Mutation probability
Elitism rate
"Fit" cooperators
Random cooperators
[30−90]
1000 + 100*Nr
1
[0.02−0.3]
{0.1−0.6]
1
{1,2,3,4}
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
WBCD results: classification (longest rule)
Rules
NeuroRule
Setiono (2000)
Fuzzy−genetic
AIM 1999
Fuzzy CoCo − IEEE TFS 2001
Average
Best
97.07% (4)
97.36% (4.0)
97.36% (4)
97.36% (3)
97.73% (3.9)
98.54% (5)
97.80% (6)
97.91% (4.4)
98.54% (4)
97.80% (−)
98.12% (4.2)
98.68% (3)
97.51% (−)
98.18% (4.6)
98.83% (5)
6
98.10% (−)
98.18% (4.3)
98.83% (5)
7
97.95% (−)
98.25% (4.7)
98.98% (5)
1
97.36% (4)
2
3
98.10% (4)
4
5
98.24% (5)
Learned Boolean rules
Evolved fuzzy rules
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
Two−rules evolved system
if (v1 is Low) and (v3 is Low) and (v5 is Low) then (output is Benign)
if (v1 is Low) and (v4 is Low) and (v6 is Low) and (v8 is Low) and (v9 is Low) then (output is Benign)
else (output is Malignant)
v1
v3
v4
v5
v6
v8
v9
Classification rate = 98.54%
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
Computing requirements
.
Fuzzy GA: Single population (Peña & Sipper 99)
Number of fitness evaluations = Np * Gmax
200 * (2000 + 500Nr)
Single-rule systems: 500.000 fitness evaluations
Five-rule systems: 900.000 fitness evaluations
Fuzzy CoCo: Cooperative coevolution (CEC-2000)
Number of fitness evaluations = 2 * Np * Gmax * (Ncf + Ncr)
32.000 * (1000 + 100Nr) {worst case, Ncr=3}
Single-rule systems: 352.000 fitness evaluations
Five-rule systems: 480.000 fitness evaluations
Carlos Andrés Peña-Reyes
.
Logic Systems Laboratory - Swiss Federal Institute of Technology Lausanne
.
.
The problem: mammography interpretation
mammogram
reading
protocol
Database
{
516 readings
187 malignant (pos)
329 benign (neg)
.
COBRA system:
computer−assisted
case interpretation
biopsy
recommendation
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
.
COBRA system: internal view
Web−based user interface
Reading form
Reading
input
Fuzzy system
Malignancy
Threshold unit
appraisal
Biopsy
Proposal
Diagnostic decision unit
Database
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Understanding the database
Variable type
Binary
Continuous
Discrete
.
Number
4
3
8
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
.
Genome encoding for linguistic labels
Ri: if (v1 is Ai1) and (v2 is Ai2) and (v3 is Ai3) and ... and (v15 is Ai15) then (output is Ci)
Low
High
V1
P’1
P3
Pi
V1
V2
....
Malignant
..... V15
V3
P1
Benign
None
P’3
P15
Binary variables (e.g., V2):
not encoded
P’i
Vi
P’15
....
V15
Continuous variables (e.g., V1):
3 var. x 2 par. x 7 bits = 42 bits
Discrete variables (e.g., V3):
8 var. x 2 par. x 4 bits = 64 bits
DB
Total genome length
= 106 bits
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
.
Genome encoding for rules
A4
A1
A2
+
A3
A6
A7
A10 A11 A12 A13 A14
2
2
2
2
Ac1 Ac2 Ac3 Ar1 Ar2
...
20
20
1
R1
A8
...
A9
if Sr = 0
A15
if Sr = 1
Radiological
Clinical
22
A5
Ri
...
RB
Rn
2
1
1
Ar6
Sr
C
Co
Total genome length = 20 x Nr +1
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Performance measures and fitness function
Sensitivity
TP
TP + FN
Basic fitness (Fbase)
Sensitivity +α Specificity
1+α
Specificity
TN
TN + FP
Accuracy
TP + TN
TP+TN+FP+FN
PPV
.
TP
TP + FN
Accuracy reinforcement
Fbase + β Accuracy
1+β
(note: done only if Accuracy > 0.7)
Carlos Andrés Peña−Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology Lausanne
.
.
.
Fuzzy CoCo results on 65 runs
25
22
20
14
15
14
Nr
10
15
20
25
Average per class
Reff
Fitness
9.17
0.8754
12.03
0.8786
14.15
0.8934
15.78
0.8947
Vr
2.52
2.62
2.59
2.76
Nr
10
15
20
25
Best individual
Fitness
Reff
0.8910
9
0.8978
12
0.9109
17
0.9154
17
Vr
2.22
2.50
2.41
2.70
10
5
5
4
3
2
1
0
0.82
0.83
0.84
0.85
0.86
0.87
0.88
0.89
0.9
0.91
0.92
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
Performance of two selected systems
17−rule
.
9−rule
Measure
Figure
Ratio
Figure
Ratio
Sensitivity
Specificity
Accuracy
PPV
99.47%
68.69%
79.84%
64.36%
186/187
226/329
412/516
186/289
98.40%
64.13%
76.55%
60.93%
184/187
211/329
395/516
184/302
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
The 9−rule system with two different thresholds
Threshold = 2
.
Threshold = 3
Measure
Figure
Ratio
Figure
Ratio
Sensitivity
Specificity
Accuracy
PPV
100.0%
63.22%
76.55%
60.71%
187/187
208/329
395/516
187/308
98.40%
64.13%
76.55%
60.93%
184/187
211/329
395/516
184/302
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.
.
COBRA system: reading form
.
Carlos Andres Pena Reyes
.
Logic Systems Laboratory − Swiss Federal Institute of Technology
.