Value-Based SE & Empirical SE
Value-Based Project Monitoring an the contribution of QA activities
Experiences, Opportunities, Challenges
ISERN Meeting, Noosa Head, Australia, November 15 2005
Stefan Biffl
Inst. of Software Technology and Interactive Systems
TU Wien, Austria
[email protected]
Experiences with Empirical QA/Inspection Studies
Our experiences with empirical studies in the QA/inspection area
• Reading techniques (perspective-based, usage-based); inspector capability
• Estimation techniques (defects, reliability)
• Recently: value-based requirements tracing (65% less effort with similar benefits)
Focus on process-oriented models of methods, cost, and technical measures
• Often: effectiveness, efficiency; some value orientation with focus on “critical defects”.
• Interest in experimental findings that hold in a more general context.
However, cost-benefit models to justify QA/inspection in a project need to
take project and company context into account:
• Cost drivers can typically be modeled well with data from project monitoring
• However, the value contribution of QA and the impact of QA improvement
is much harder to argue and quantify (especially for a general case).
New book on “Value-Based Software Engineering” with many ISERN authors
• Theory on valuation and decision support; measurement.
• Applications: Architecture, RE selection for release planning, and QA/test management.
• -> Benefits Results Chain helps model QA/inspection contributions to project monitoring.
2
VBSE Key Element: Benefits Results Chain
SE is only part of development for software-intensive systems, e.g., aircraft, cars.
• While software is an important success factor for added value,
the actual benefits come from the application in the systems context.
• Need for communication of value between stakeholders with different backgrounds and
perceptions of "the project“ and key requirements
• Stakeholders from SE and other disciplines need to sufficiently understand
– The impact of IT requirements on SE project and product,
e.g., simplifiers and complicators.
– The impact of SE decisions on the “external” system and its evolution options.
•
•
Value-based SE key element: the “Benefits Results Chain”,
allows linking SE project and product capabilities to business outcomes
(focus on success factors, agility, bottlenecks, value breakers).
VBSE allows cost-effective “internal” trade-off analysis of SE activities
according to value translated from the requirements (most valuable first).
– Weigh conflicting quality attributes
– Find balanced solutions for resource allocation in projects
(how much investment in QA activities is enough?)
3
Opportunity: Value-Based Project Monitoring
Value-based IT project reports on the actual value contributions over time allow:
• Better alignment of stakeholders, project management, and engineers.
• Release planning that allows linking project value to release results.
• A negotiation process between stakeholders, management, and key engineers
on the value contribution of
– release products,
– the strategic options for engineering solutions, and
– their impact on the project and system lifetime cost and benefits;
• Project reporting that links the results of release planning to project progress
and its stakeholder value; allows to more flexibly react to real-world changes
during a release.
Potential contributions of QA activities to value-based project monitoring:
• Plan-driven development:
QA can avoid late rework and provide visibility on intermediate results.
• QA provides opportunities for stakeholders to assess the value of deliverables.
• QA provides tracing between concurrently evolving artifacts.
4
VBSE/EMSE Research Issues and Discussion
Empirically study “best practice” for important SE planning and decision situations
• Benefits Results Chain allows analysis to ensure sufficient strength
for all major contributions and to identify weak links for improvement.
•
•
•
Elicitation of project and company context factors
that influence the value contribution of a SE approach.
– Identify context factors that are main value drivers.
– Stakeholders utility functions may vary strongly -> need for negotiation support.
Investigate measures and models of SE value contribution
in industry on project and business unit level
– E.g., results chains of typical projects and IT initiatives -> frame for GQM.
– Identify best practices for reporting IT project value contributions and
the contribution of SE and QA practices.
Use benefits results chain to
– Set expected levels of SE productivity and quality in a project context ;
e.g., determine sufficient documentation in software lifecycle tailored to
project needs as basis for QA planning.
– Help focus improvement initiatives (QA, SPPI) on best value contributions.
5
Value-based SE: References
•
•
•
Boehm B.W.: Value-Based Software Engineering. Software Eng. Notes 28(2),
2003, 1-12
Boehm, B. W. and Huang, L.G.: Value-Based Software Engineering: A Case
Study, IEEE Computer, March 2003 pp 33–41
Biffl, S., Aurum, A., Boehm, B., Erdogmus H., Grünbacher P.: Value-based
Software Engineering. Springer Verlag, Heidelberg; September 2005
– Chps. 1, 2: Initial VBSE Theory
– Chp. 3: Valuation of Software Initiatives Under Uncertainty
– Chp. 4: Preference-Based Decision Support
– Chp. 6: VBSE Key Elements
– Chp. 7: Stakeholder Value Proposition Elicitation and Reconciliation
– Chp. 9: Industrial Empirical Study on Requirements Selection
– Chp. 11: Value-based Test Management
– Chp. 12: Value-based Release Planning
– Chp. 16: Quantifying the Value of New Technologies
6
Backup Slides
Backup Slides
7
Challenges of Value-Based Project Monitoring
Structure a project to support value delivery
• Early assessment of options for product structure (architecture)
that links to most important win conditions
(success factors, bottlenecks, value breakers; stakeholder incentive alignment)
• Trace requirements value (priorities) to project management, SE, and QA.
Measurement Challenges:
• If individual and group preferences are not known,
inefficient requirements priorities, SE and QA investments may come out.
• Stakeholders utility functions may vary strongly and even conflict.
• It is not always meaningful to aggregate measures from different dimensions
to one utility measure.
• Multi-criteria models allow stakeholders a more direct assessment of options.
However, the more complex data needs better analysis and visualization
support to understand trade-offs between SE, QA, and other activities.
8
Value of Project Deliverables
Project contribution to IT change impact
• Analysis of requirements for their value
impact.
• Identify key win conditions that frame
the project.
• Communicate value priorities to project
participants (project manager,
developers).
• Assess need for QA to complement SE
activities.
Order to delivery
time is an important
buying criterion
Assumption
Contribution
Implement a new
order entry system
Initiative
•
Reduce time to
process order
Reduced order
processing
cycle
Intermediate
outcome
Increase sales
Reduce time to
deliver product
Outcome
Results chain: allows analysis to
ensure sufficient strength
for all major contributions and to
identify weak links for improvement
(bottlenecks; value breakers)
9