CBS Process WG
Participants
David Carney (chair)
M. Al-Said (scribe)
Tony Jordano
Kyung Whan Lee
Jeffrey Poulin
David Klappholz
Glenn Berg
Jongmoon Baik
Rami Razouk
George Huling
Tim Spinney
Steve Cross
Mike Moore
Co-chairs:
Betsy Clark
Dan Port
Tricia Oberndorf
Working Approach
Homework: come in with 3 issues
 Brainstormed, collapsed, clustered
 Prioritized by importance (practitioner &
researcher) & difficulty
 Formulated process-oriented statements
 Discussed prospective CeBASE Top 10 List

Issue 1: CBS Lifecycle Models
and Processes - 1
Is there a new lifecycle model for CBSs?
 Can one (or more) of the existing
lifecycle models be tailored to CBSs?
How do we define a CBS life-cycle
process?
Issue 1: CBS Lifecycle Models
and Processes - 2
Both of these questions in light of:
 A requirements-driven (COTS-supported) vs.
COTS-driven paradigm
 Strategies for handling different kinds of
market segments
 CBS time to market pressures
 Level of service quality needs
 ….
How do you identify [-> metrics group] and
factor in multiple CBS cost drivers?
Issue 2: Post-Deployment Process
In what ways does the CBS postdeployment process differ from:
 the CBS development process
 the custom post-deployment process?
In what ways do the differences affect
cost estimation for CBSs?
Issue 3: Release Planning
What are the process implications of planning
for system releases where COTS product:
releases are not synchronized with each other
 releases are not synchronized with your system
 end-of-life occurs during your system life

Planning includes cost estimation, scheduling,
determining system release content, etc.
Issue 4: Assessment,
Evaluation & Testing Process
How and when do you assess, evaluate,
and test:
COTS products
 COTS-based systems
“When” includes timing (e.g., multiple points in
the process) and system-independent
certification of COTS products.
“How” includes process, techniques, tools,
roles, and metrics.
Related to determination of CBS requirements.

Issue 5: COTS Market and
Identification
What are the processes for





Acquiring and maintaining market knowledge
Analyzing the forces in a market segment
Forecasting trends in both market segments and
specific products
Obtaining & disseminating COTS product
information and product-specific experiences
Establishing and maintaining vendor relationships
Both the active (e.g., influencing) and passive
aspects of these should be considered.
Issue 6: Surprise!
How does a CBS process
accommodate/address large discontinuities
stemming from COTS product surprises?
Surprises result from releases or patches:
that are not backward compatible
 that fail to perform
 whose fundamental properties change

Exacerbated by inadequate documentation.
 [-> architecture issue concerning product
characterization]
Issue 7: Organizational
Assessment Capability
How does an organization assess its
capability to be successful with CBSs?
How does an organization improve its CBS
capability? This includes such issues as:
relationship between CMMI and CBS processes
 metrics for CBS process maturity

Applies to capabilities of CBS acquirers,
developers, and maintainers.
Other Issues







Dealing with extra features
Managing customer expectations (postselection)
World-class SE organizations becoming COTS
integrators
Licensing
Systematic way of deriving system challenges
In-place transition of baselines for
uninterruptible systems
Metrics for degree of product & system risk
TS
vs
.
&
D
M
kt
Id
s
lo
y
rv
rD
n
ef
ic
ie
C
nc
ap
ie
ab
s
ilit
y
As
se
ss
C
os
tD
riv
er
s
Su
p
O
TS
do
O
Ve
n
C
C
e
pr
is
e
t-D
ep
Su
r
Po
s
el
ea
s
A,
E,
&T
R
COTS Process Issue Priorities –1a
Importance to Practitioner
9
8
7
6
5
4
3
2
1
0
0.0
Cost Drivers
Capability
Assess
Vendor
Deficiencies
COTS Sup
vs. Drvn
COTS Mkt &
Id
Surprises
Post-Deploy
A,E,&T
Release
COTS Process Issue Priorities –1b
Importance to Practitioner Scores
3.0
2.5
2.0
1.5
1.0
0.5
ss
e
pr
is
e
As
se
Su
r
bi
lit
y
el
ea
s
lo
y
rv
n
rD
s
ef
ic
ie
nc
C
ie
O
s
TS
M
kt
&
Id
do
D
t-D
ep
R
Po
s
er
s
A,
E,
&T
Dr
iv
vs
.
os
t
Su
p
ap
a
TS
C
O
Ve
n
C
C
COTS Process Issue Priorities –2a
Importance to be Researched Tallies
12
10
8
6
4
2
0
0.0
COTS Mkt &
Id
Vendor
Deficiencies
Surprises
Capability
Assess
Release
Post-Deploy
COTS Sup
vs. Drvn
A,E,&T
Cost Drivers
COTS Process Issue Priorities –2b
Importance to be Researched Scores
3.0
2.5
2.0
1.5
1.0
0.5
ss
rv
n
As
se
D
A,
E,
rD
&T
ef
ic
ie
C
nc
O
ie
TS
s
M
kt
&
Id
do
bi
lit
y
vs
.
s
er
s
e
pr
is
e
Dr
iv
Su
r
os
t
Su
p
ap
a
TS
C
O
Ve
n
C
C
lo
y
el
ea
s
t-D
ep
R
Po
s
COTS Process Issue Priorities –3a
Difficulty Tallies
8
7
6
5
4
3
2
1
0
0.0
COTS Mkt &
Id
Vendor
Deficiencies
A,E,&T
Capability
Assess
COTS Sup
vs. Drvn
Surprises
Cost Drivers
Release
Post-Deploy
COTS Process Issue Priorities –3b
Difficulty Scores
3.0
2.5
2.0
1.5
1.0
0.5
Release, Surprise Votes
Post-deployment, AE&T Votes
Assessment, COTS Id. Votes
COTS-supp/–driven Votes
Priorities (Diff. = Imp.)
8
Priorities (Diff/Imp = 1)
8
Prospective CeBASE Top 10
General reactions:
 Use of list as guidance
 Good (only?) as top 10 list of things
CeBASE will research
 Use of unfamiliar and imprecise
terminology
 Is it valid to have hypotheses on this
list?
Specifics -1
1. Valid within a limited scope only.
2. What else is new? So does everything else!
Add: “…. Just like traditional”
 Or “Cost & schedule overruns are as common
with COTS as with custom development and
they (CBSs) can cost as much.”

Specifics -2
3. What is “CBS type”?




Domain?
COTS-driven vs COTS-supported?
COTS product type?
…..?
4. Should be studied, but we don’t believe
the hypothesis as posed:

How can a parametric model account for the
situation in which the selected products can’t be
integrated?
Specifics -3
5. False, this is one of many parameters
that influence the estimate.

Prefer: ” Architectural mismatch will affect
CBS cost.”
6. “ …just as with traditional systems;
exaggerated with COTS products.”

Change to “CBS post-deployment costs
dominate …”
Specifics -4
7. Intuitively agree; defects will be in
mismatches between products, not
something visible by inspection of line
of code.
A (ADDITION): Writing glue code requires
greater skill than traditional coding.
Often need to discover lots of things.
Specifics -5
8. Agree with “frequently far worse than linear”,
question use of square; heavily dependent on
other factors (e.g.; use of standards, quality
of products, degree of integration)
9. Question “twice”.


Replace with“Risk specific to CBSs must be
managed; CBS risks (and their mitigations) are
different; risk doesn’t disappear.”
Current statement is not a useful hypothesis.
10. Disagree with percentage; agree vaporware
is one of many factors.
Conclusions
Process is foundational for CBSs.
Risk management is still key - and will be
different for CBSs.
It was cold, but we had fun anyway!