What was the problem?
We had a defect tracking tool -- several!
No support for multi-site development.
Maintenance overhead of multiple databases.
No web enabled version.
So naturally, the solution is …
To buy a new tool!
Original Plan
Identify products / vendors
Develop short list
Write evaluation criteria
Evaluate & select product
Negotiate price
Install tool in Development
Install new server and migrate to production
Execution
Things go okay through installing tool in a
tool testing environment.
Usual and predicted hiccups.
–
–
–
–
Getting server configured with Oracle
Installing license server
Calculating total space requirements
Matching field values to old product
Another Track Starts
PIT (Process Improvement Team) starts
– Look at collecting “Defect Root Cause” metrics
– How can we improve defect management during
testing?
No overlapping membership and little
awareness of each other
PIT approach is to look only at metrics
Until…
The “process guy” enters the picture
Process Mapping
How does this process work today?
Who needs what information?
How is the process controlled?
What are the current breakdowns?
– Rework like bad fixes, rebuilds
– Stalled problems like “hot potato” or “not my problem”
– Missed communications like “that fix was completed 6
weeks ago”
Definition: MR=change request or defect
Original Process
Field Defects
Test Defects
MR Meeting
(Testers &
Developers)
"Invalid Defect"
Need info
Reject
Fix
"Invalid Defect"
Someone else's Problem
Build & Submit
Retest
Failed Test
Very simple process model, but rework wasn’t understood
very well
Multiple (24) States
Fixed in STG, Fixed in ITG, Cannot duplicate, Waiting
external verification …….
PIT Progress
Process Guy:
“Defect management is basically a change
management process. I know what change
management looks like.This shouldn’t be too
hard.”
Process guy still is unaware of tool effort.
PIT meets only 1 hour per week and
attendance is irregular
Second Event: An “IAF”
Internal Audit Finding uncovers a mis-match
between the process documentation and the
way the current tool works.
Tool guy calls process guy for review of
changes.
MR process document is 22 pages of text!
Regroup
No longer just a metrics problem
Tool implementation project is stuck
Metrics team is stuck too
MR process has no clear governance
Rules are very complex
Plan
Seems like there are three distinct things to do.
Model
 Design a robust process and metrics.
Pilot
 Find a willing partner, and
 Make sure you have time to fix the things you forgot.
Deploy
 Conversion will be a big deal so it will need a template
and some estimation for the various teams.
Plan -- Model The Process
– Stake holders
> Developers, testers, support, tech writers, customers
– Current activity
> MR meeting, correcting, builds, testing, postponing,...
– Transfer of control, inputs, outputs
> Moving to another group, completed fixes, ...
– Management and escalation
> Postponing, 1 problem that needs 2 groups
– Write
– Review and fix
Model the Process
Use states because MR’s change hands
– Different people do different kinds of work.
– Rules are applied to exit states.
Management of states
– Danger of getting stuck. Resources not balanced.
– State transitions: What enforcement? What audits?
Remember Test and Fix usually works fast
– Can’t let management get in the way.
– So your change control board needs to have rules to
focus the activity on higher problems.
Process Model State Diagram
New
Working
Fixed
Testing
Postponed
Watch
Closed
MR STATE TRANSITION DIAGRAM
[ New ]
Submit a new
MR
9.1.0
9.1.11
[ Close ]
[ Assign ]
NEW
9.1.1
[ Postpone ]
[ Fixed ]
[ Reject ]
9.1.7
9.1.2
WORKING
[ Test ]
[ Postpone ]
FIXED
9.1.3
9.1.9
9.1.5
[ Close ]
9.1.14
[ Reject ]
TESTING
9.1.8
9.1.12
9.1.4
[ Watch ]
[ Resubmit ]
9.1.6
[ Close ]
POSTPONED
CLOSED
9.1.13
[Resubmit]
WATCH
[ Close ]
9.1.10
State: Describe the Activity
New
– Purpose: Determine benefit of change and assign work.
> What products are affected?
> Who gets the assignment?
> Remember: mostly it’s about test and fix, but not always.
Working
– Someone is assigned to change some specific items for
a specific purpose.
State: Describe the Activity
Fixed
– These changed items solve the problem
> Track what was changed and problem number
– But it waits for the build because several changes will
be submitted together for the testers.
Testing
– Sometimes this is an inspection or editorial process.
State: Describe the Activity
Postponed
– We will not assign resources to this request at this
time.
– Product Management decision (part of CCB).
– Takes Product Management to put back onto the
queue.
Watch
– “We think this is fixed bu we cannot tell for certain.”
– Sits for 3 weeks or 2 test cycles.
Closed
Roles
Developer & Developer Manager
Release Engineer (build)
Tester and Tester Manager
Program Manager (project manager)
Product Manager (postponing)
Product Support Representative (field defects)
Tool support
Map Roles
To state activities
– New: Assign work: Tester and Development Mgr.
– Fixed: Build: Developer, Release Engineer, Test
manager
Add process management activities
– Balance resources & Prioritize: Dev. Mgr, Prog. Mgr.,
Prd. Mgr., Prd Support
Tool Support
– New projects, new users, performance, new reports
Current Activities
MR Meeting already exists
– Testers and developers meet in small groups.
– Make assignments.
– Review build contents and test plans.
Improvements
– Agenda is not standard and sometimes revisits the
same problem.
– Escalation and transfer of control between
development groups is awkward.
MR Meeting
Agenda
–
–
–
–
–
New Defects (test failures) for assignment
Prioritize test blockers
Testing progress on fixed items
Move to watch.
Items for escalation to management
(postpone, conflicts, etc.)
Activities -- CCB
EMT: Engineering Management Team
– This group does intergroup coordination, not CCB.
– Very large group, inc. CPS, PLM, Ops, etc.
– It might be ok for smaller projects, but not for big
ones.
– EMT should see MR metrics for risk assessment.
CCB
– Missing!
– Prioritize, postpone, balance resources, resolve
conflicts
Creating the CCB
Purpose
– CCB is a management function. It’s primary
responsibility is to prioritize work and balance the
resources.
– It will also resolve disputes if a problem seems to be
treated like a hot potato.
– The CCB assumes a natural priority on defects so
they can be handled by MR Meeting.
– The CCB assumes full responsibility for prioritizing
feature requests, and process changes.
– All items for postponement go through CCB.
CCB Agenda
Current metrics
– Transition activity, Current State #s
– Stuck State (things that have been sitting)
Other change requests
– Postpone, engineering features, etc.
Prioritize and Rebalance
Process Change Plan
Identify Pilot Team
– Use a team doing new work to minimize impact
> Converting a lot of data will take time
– Representative
> Must cover software and hardware
– Enough resources to absorb pilot effort
– Enough time to fix problems with new MR process
– Amenable to change
> Good managers
Process Change Plan
Pilot Work
–
–
–
–
–
–
Set-up database
Convert existing data
Training using scenarios
Support plan two people covering 6am-6pm
Change control for tool!
Continued development plan
> Have to work on reports and graphs
The Pilot
Team
– Large project of 100+ developers
– 2-year effort
– Hardware engineers have not used defect tracking
Training
– 8 scenarios prepared but class only covers 3
– Lots of questions make course take over 3 hours
– 3 scenarios is enough for first groups
Pilot is successful. CCB works right away.
Pilot Problems
Multi-site does not work as expected
– Performance makes remote use problematic
– Web interface works, but looks really different
– but that’s what we use.
Support Team
– Scripting, tool foibles, Crystal Reports
– Multiple printers affect reports
– Tool developers must do code reviews too!
Substates, rules, and reports.
“Invalid defects” and rejects
– Someone has a problem! It’s just not yours.
– Test case, documentation, and many others
“New” has lots of substates
– Under investigation, failed test, re-submit from watch
Processing of Duplicates
– Tool provided “Duplicate” function, but most should just
close with “Reason=Duplicate”
– External ones needed to be tracked to communicate to
customer.
Great Inventions
Team started to track their process changes
– Had to add a Change Type of “process”
(addition to software, hardware and documentation)
– Needed simplified Testing state
Parent-Child relationship
– Problem that requires effort from multiple groups
– Used to track feature development mid-stream
– and a few complex defects
Management Queries & Reports
CCB has Lots of them
–
–
–
–
–
–
Lots of different combinations of filters
Current State
Transactions during current period
“Stuck State”
Candidates for postponement and bringing them back.
“What are the blockers?” & priority questions
MRRB uses “New”
Leads audit transactions for missing data
Eng. Lead /MR Rev. Team
Requests postponment
New
Process
Documentation
MR Review Team
Agrees on
* Severity
* R&D Priority
* Other MR information
New
<blank>
MR Review Team
Eng. Team
Agrees on
Requires additional information
* Severity
* R&D Priority
Eng. Lead
Accepts the MR
Postponed
<blank>
Eng. Lead
Requests postponment
New
Rejected
Other
Info Needed
MR Review Team
Agree the test case is incorrect Eng. Lead
Needs to do analysis
Eng. Team
Requires additional information
Eng. Team
Requires additional information
New
Reviewed
Eng. Team
Requires additional information
New
Rejected
Bad Test Case
MR Review Team
Agree the test case is incorrect
MR Review Team /
Submitter
Close as 'not a problem'
* Works as intended
State
Diagrams
New
Under Study
MR Review Team /
Submitter
Close as 'not a problem'
* Works as intended
Eng. Lead
Accepts the MR
Eng. Lead
Establishes an agreed
* Severity
* R&D Priority
Accepts the MR
New
Accepted
Eng. Lead
Requests postponment
Eng. Lead
Requests postponment
Closed
Postponed
<blank>
Eng. Lead
1) Assigns the MR
2) Estimates
2a) Level of effort to correct
2b) Risk of creating a serious
regession
2c) Probability this causing a
Field Service call
Working
Eng. Lead
1) Assigns the MR
2) Estimates
2a) Level of effort to correct
2b) Risk of creating a serious
regession
2c) Probability this causing a
Field Service call
Responsibility Matrix
State
New
Working
Fixed
Testing
Watch
Posponed
Posponed
SubState
<blank>
Duplicate
Reject
<blank>
Accepted
Reviewed
UnderStudy
ReSubmitted
<blank>
<blank>
<blank>
<blank>
<blank>
Accepted
Reject Reason
Other
Ambiguous Requirement
Bad Test Case
By Design
Can't Duplicate
Design Limitation
Do Not Fix
Doc Defect
Improper Document
Other
External Test Failed
Integration Test Failed
Regulatory Test Failed
Rejected Fix
Resubmitted Defect
System Test Failed
*Responsible
Submitter
Submitter
Submitter
Submitter
Submitter
Submitter
Submitter
Submitter
Submitter
Submitter
Submitter
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Eng. Team Lead
Release Eng.
Submitter
Submitter
CCB
CCB
Process Change Plans
Technology Development Process
– How do you build new capacity and capability?
New process is always a project of this type.
The following, Technology Development
Process, type plan would have made it go
faster.
TD Process
Business Case
Development
Concept
SWOT Analysis
Initial Funding
Initial EMT list
Specification &
Planning
Requirements
Specification
Vendor Evaluation
SOW
SDP
Project Schedule
Test Strategy
Development
Pilot & Validation
Contract
Monitor Contract
Design
Construction
Prototype Build
Test Plan
Testing
Accept Contract
Pilot Monitoring
Post Mortem
Technology
Transfer
Develop Training
Training
Contract Closeout
ISO Update
TD Phase 1
Statement of Work (or Project Plan)
– Detailed scope and responsibility
Requirements & Specification
– Context diagram, quality goals, environment
Vendor & Tool Selection
– May make this phase take a while
TD Phase 2
Process Model & Documentation
Tool Implementation
Pilot training material
Pilot validation plan
TD Phase 3
Pilot
Validate and measure process
Assist new users
Fix problems
Develop standard migration procedure
Advertise
TD Phase 4
Publish process documents
Publish training
Migrate
Train and record
Advertise