EFFORT ESTIMATION AND RISK
MANAGEMENT USING THE COBRA®
APPROACH
Galorath International Conference, Eindhoven, Netherlands, Dec. 8, 2011
Dr. Jens Heidrich
Division Manager
Process Management
Fraunhofer IESE
Fraunhofer-Platz 1
D-67663 Kaiserslautern
Deutschland
Tel.:
Fax:
Email:
© Fraunhofer IESE
+49 (0) 631-6800-2193
+49 (0) 631-6800-9-2193
[email protected]
About the Fraunhofer Gesellschaft
„ Named after Joseph von
Fraunhofer (1787-1826), a
researcher and inventor and
entrepreneur
„ 18000 employees
Fraunhofer Institute for
Experimental Software Engineering
(IESE)
Kaiserslautern, Germany
Bremen
Hannover
Oberhausen
Dortmund
Duisburg
Schmallenberg
St. Augustin
Aachen
Euskirchen
Wachtberg
Darmstadt
Kandern
EfringenKirchen
The CoBRA Approach
Leipzig
Dresden
Jena
Chemnitz
Ilmenau
Erlangen
Kaiserslautern Fürth
Freiburg
Berlin
Teltow
Cottbus
Halle
Schkopau
Würzburg
Saarbrücken Karlsruhe
Pfinztal
Ettlingen
Stuttgart
© Fraunhofer IESE
Potsdam
Braunschweig Magdeburg
St. Ingbert
13/12/2011
Lübeck
Bremerhaven
„ Germany’s leading organization
for applied research and
technology transfer
„ 60 institutes
Rostock
Itzehoe
Nürnberg
Freising
München
Holzkirchen
2
Software Cost and Effort Estimation Common Mistakes and Challenges
„ Estimating before knowing basic
requirements
Specified
Context
Quality
„ Creating estimates solely as a rule of
thumb
„ Not considering basic risks
„ Brooks‘s Law: “adding manpower to
a late software project makes it later”
(fixing wrong estimates in runtime is
difficult)
Optimize
Functionality
Productivity
Specified
Unknown
„ Using „Off the shelf“ estimation
models work without calibration
Optimize
Resources
13/12/2011
© Fraunhofer IESE
Effort
The CoBRA Approach
Schedule
3
Existing Cost Estimation Methods
Expert-based
Hybrid
Data-driven
„Rule-of-thumb
„CoBRA®
„SEER
„Wide-band Delphi
„BBN-based
„COCOMO
„Planning Game (XP)
„…
„CART-based
„…
„…
Human Expertise
Typical
Software
Organization
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
Quantitative Data
4
The CoBRA® Method:
Cost Estimation, Benchmarking, and Risk Assessment
„ Creates a custom-tailored estimation model
„ Company-specific size measure
„ Company-specific influencing factors
„ Follows a hybrid approach
„ Makes use of historical data (at least 10 projects)
„ Captures expert knowledge about effort drivers
„ More characteristics
„ Accurate: Estimation error about 9%-14% (MMRE)
„ Reusable: Documents company experiences (white box models)
„ Customizable: Estimation model can be adapted to company needs
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
5
Basics: Cost/Effort Estimation Equation
Probability Distribution
New Projects
Characterization of the new
projects regarding size and
effort drivers
Probability
1.0
0.8
0.6
0.4
0.2
0.0
100
200
300
400
500
600
Effort = 1 / PN x Size x (1 + Effort Overhead)
Effort (e.g., PD)
Productivity Model (e.g., FP/PH)
Historical
Project Data
Quantified Causal Model
Customer
Participation
Effort
Overhead
Nominal
Productivity
Requirements
Volatility
Project Team
Capabilities
Multiplier
Requirements
Volatility
Reliability
Requirements
Customer
Participation
Size (e.g., FP) x Effort Overhead
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
6
Basics: Underlying Idea of the CoBRA Approach
„ Build causal effort model that allows explaining the variance of productivity
over projects in a certain context
Explain
Distribution of actual productivity
Sep. 14-15, 2011
© Fraunhofer IESE
Effort Estimation and Risk Management
Causal effort model
7
Basics: Causal Model and Cost Drivers Interaction
F1: Requirements
Volatility
F2: Disciplined
Requirements
Management
-
Indirect
influence
+
V6.1: Domain
Experience
F3: Customer
Participation
Effort
Overhead
+
-
F6: Key Project
Team
Capabilities
V6.2: Communication
Capabilities
F4: Customer
Competence
+
Decomposition of a
complex 3-dimensional
concept
F5: Meeting Reliability
Requirements
Sep. 14-15, 2011
© Fraunhofer IESE
Effort Estimation and Risk Management
8
Basics: Causal Model and Cost Drivers Interaction
(with Generic Triangular Distributions)
F1: Requirements
Volatility
F2: Disciplined
Requirements
Management
-
+
F2=nominal &
F1=extreme
F4=nominal &
F3=extreme
F3: Customer
Participation
F4&F3=extreme
+
V6.1=extreme
F2&F1=extreme
Effort
Overhead
-
F6: Key Project
Team
Capabilities
V6.2: Communication
Capabilities
V6.2=extreme
F4: Customer
Competence
+
F5=extreme
F5: Meeting Reliability
Requirements
Sep. 14-15, 2011
© Fraunhofer IESE
V6.1: Domain
Experience
Effort Estimation and Risk Management
per Expert
9
Basics: Example Simulation Runs using Random
Sampling (Monte Carlo Simulation)
Probability Density
Probability Function
F1
Probability Distribution
EO(F1)
EO(F2)
F3
∑
Probability
F2
Mean
1.0
0.8
0.6
0.4
0.2
0.0
10
20
30
400 50 60
80 100
% Effort Overhead
EO(F3)
…
Sep. 14-15, 2011
© Fraunhofer IESE
Random selection of expert and
overhead value from triangular
distribution
Effort Estimation and Risk Management
10
Basics: Cost Estimation, Benchmarking & Risk
Assessment
Estimation
Risk Assessment
Mean
Probability
Probability
Mean
Probability
B
Effort/Cost
Estimation (Cumulative)
Probability
Mean
Benchmarking Projects
Effort/Cost
A
Low
Risk
Mean
Risk
High
Risk
Overhead
Risk Minimization
Team
Capabilities
Requirements
Volatility
Customer
Participation
Effort/Cost
13/12/2011
© Fraunhofer IESE
Sensitivity
The CoBRA Approach
11
Basics: CoBRA® Life Cycle
6 Package and Improve
1 Characterize
CoBRA®
Improve
Model and
Package for future use
Define CoBRA® Application Environment
(Context, Size Measure, etc.)
5 Analyze Results
Analyze Model Performance and
Identify Improvement Potentials
2 Set Goals
Define Estimation Goals and
Determine Application Scope
3 Build Model
4 Execute Model
Define CoBRA® Model,
Collect Data, and
Perform Initial Validation
Characterize New Project,
Apply CoBRA® Model, and
Validate Estimation Quality
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
12
Basics: Tool Support via CoBRIX
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
13
Case 1: Oki Electric Ltd., Japan
based on collaboration with Japanese
Information-technology Promotion Agency (IPA) / SEC
„ Goal: Improve estimation accuracy and effectiveness as well as understanding
of basic cost/effort drivers
„ Context
„ 16 historical projects from business applications domain
„ 12 experienced project and quality manager
„ Approach: Iterative application of the CoBRA® approach
„ Results
„ Significant improvement of estimation accuracy (MMRE: 14%)
„ Better understanding of individual cost/effort drivers
„ Improvement of data collection processes
„ High acceptance of the model users
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
14
Case 1: Final Causal Model for Oki Electric Ltd.
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
15
Case 1: Iterative Improvement of the Model
Estimation Error of CoBRA Model
120,00%
MMRE
Custom-tailored models
need to be developed
iteratively
Estimation error
100,00%
80,00%
MdMRE
60,00%
40,00%
20,00%
0,00%
Initial model
1st iteration
2nd iteration
3rd iteration
4th iteration
Refinement Iterations
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
16
Case 2: Mitsubishi Research Institute (MRI), Japan
„ Goal: Benchmarking productivity of projects of Japanese industries and
recommend improvement actions
„ Context
„ 30 historical projects of a Japanese company
„ Approach
„ Use SEER-SEM and the IPA/SEC data white book for benchmarking projects
„ Develop CoBRA causal model with experts for identifying individual
improvement potential
„ Results
„ Industry-proven ranking of projects
„ International (SEER)
„ Japanese (IPA/SEC data white book)
„ Better understanding of individual cost/effort drivers (CoBRA)
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
17
Case 2: Combining Data-Driven and Hybrid Approaches
Combining the “best” of both
worlds:
SEER
SEER for benchmarking and
CoBRA for individual
improvement models
IPA/SE
C
Causal Model
Benchmark
Benchmark Report
(SEER, IPA/SEC)
F2: Disciplined
Requirements
Management
Projects
F3: Customer
Participation
F1: Requirements
Volatility
-
+
Effort
Overhead
+
-
F6: Key Project
Team
Capabilities
F4: Customer
Competence
+
F5: Meeting Reliability
Requirements
Improvement
Recommendations
(COBRA)
Productivity
[PH/LOC]
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
18
Conclusion: Common Challenges in Practice
„ Output measure (sizing)
„ Candidates limited by context
„ LOC depends on the PL and may
be hard to estimate early
„ Functional sizing may be too costly
„ Data collection
„ Data completeness/reliability
„ High measurement effort
„ Influencing factors
„ Hard to identify and quantify
„ Factor dependencies makes
analysis extremely difficult
„ Interpretation may be misleading
13/12/2011
© Fraunhofer IESE
Around 60% of organizations do not
measure and consider influencing factors
explicitly
The CoBRA Approach
[Results from 126 literature studies, 25 IESE industry projects, and
survey among 24 companies]
19
Conclusion: Lessons Learned
„ Effective effort estimation is key for successful project management
„ There is no ultimate „best“ method that works everywhere
„ Success depends on many factors
„ The „right“ method may create many synergies
„ Combining existing methods may maximize your benefits
„ An estimation method is no oracle
„ Results need to be checked and interpreted
„ A good estimation approach is…
„ integrated into organizational processes
„ adapted to the specific needs
„ continually improved
13/12/2011
© Fraunhofer IESE
The CoBRA Approach
20
Contact
Dr. Jens Heidrich
Department Head
Processes, Measurement, and Improvement PMI
Fraunhofer IESE
Fraunhofer-Platz 1
D-67663 Kaiserslautern
Germany
Phone:
Fax:
Email:
13/12/2011
© Fraunhofer IESE
+49 (0) 631-6800-2193
+49 (0) 631-6800-9-2193
[email protected]
The CoBRA Approach
21