1
Implementing Scrum on a Large
Scale
Dan LeFebvre
Agile Coach, CSP
Orlando Scrum
Gathering
3/16-18/2009
2
Orlando Scrum
Gathering
3/16-18/2009
Biography
• Over 20 years in software product development
as a developer, manager, director, and coach
• Last 5 years using agile development practices
• Last 2 years implementing Scrum in a 700
person engineering organization as an Agile
Coach
▫ Sites in MA, OR, TX, GA, IL
▫ Also in Canada – BC, Que
▫ Also in Belgium and Noida, India
3
Orlando Scrum
Gathering
3/16-18/2009
Intent of this Session
• To take you through the journey of that 700 person
engineering organization toward agility through
Scrum.
▫
▫
▫
▫
▫
▫
▫
How did they decide to go agile?
How did they decide to go Scrum?
How do they coordinate a large suite release?
How do they handle organizationally impediments?
What about progress reporting?
What were their biggest challenges?
What are their challenges yet to face?
4
Orlando Scrum
Gathering
3/16-18/2009
5
Orlando Scrum
Gathering
3/16-18/2009
Productivity was plummeting
Version
New Feature
Development
(months)
Stabilize
(months)
Total
(months)
5.0 actual
8
6
14
5.1 actual
6
7
13
5.2 planned
5
7
12
6
Orlando Scrum
Gathering
3/16-18/2009
Cutter Consortium Evaluation
• Michael Mah and Jim Highsmith
• Evaluation – both quantitative and qualitative
• Recommended and sold Agile to CTO
7
Orlando Scrum
Gathering
3/16-18/2009
“Agile Lite” Goal 11-1
•
•
•
•
•
Iterative/incremental development
Test Driven Development
Emergent/Evolutionary Design
Daily Standups
Retrospectives
• Phase gated governance model
▫ Commit to contents, dates, cost for a 12 month
release
▫ Account for entire organizational effort
8
Orlando Scrum
Gathering
3/16-18/2009
Weakest Link Theory
• Interdependent suite of products
• All must ship simultaneously
• If any of them not agile, suite is in trouble
• Roll out to entire organization at once
• 5 full time coaches from Cutter, 5 part time
internal coaches
• After 1 year, only used 5 part time coaches
9
Orlando Scrum
Gathering
Results
•
•
•
•
Delivered 7-5
Improved Quality
More automated tests
Agile in the culture
3/16-18/2009
10
Orlando Scrum
Gathering
3/16-18/2009
“Inevitable and steady deterioration of a system or society.”
The American Heritage® Dictionary of the English Language, Fourth Edition
11
Orlando Scrum
Gathering
3/16-18/2009
Slip back to old ways
• All part-time coaches back to their day jobs
• Committed to 3 suite-wide projects within the same
release (each considered all or nothing)
▫ Globalization, New UI and new Reporting platform
• Results
▫
▫
▫
▫
▫
Projects fell behind
All UI-based automation broke
2 projects stopped doing retrospectives
Team morale suffered
No or meaningless burncharts so no transparency
12
Orlando Scrum
Gathering
3/16-18/2009
The Agile Coach
• Without the coaches, teams had no help
• Decided to get a full-time Agile Coach
▫
▫
▫
▫
Teach agile to new employees
Observe and help teams that are struggling
Roll out agile to newly acquired companies
Become the agile conscience of the organization
13
Orlando Scrum
Gathering
3/16-18/2009
14
Orlando Scrum
Gathering
3/16-18/2009
Management still too “Command and
Control”
• Culture is very hierarchical
• People treated as “resources”
▫ Regularly moved from team (workgroup?) to team
to handle crises and delays
• Management makes most decisions
▫ Teams do not feel empowered
▫ Many teams just going through the motions of
agile development
• “Blame” is typical reaction
15
Orlando Scrum
Gathering
3/16-18/2009
Collaboration Explained
• 90 people trained in collaboration skills by Jean
Tabaka and Ronica Roth from Rally
▫ Learned tools
▫ Facilitation techniques
▫ Lots of hands-on exercises
• Results
▫ Meetings ran better
▫ Better agendas and capturing of group learning
▫ Better retrospectives
16
Orlando Scrum
Gathering
3/16-18/2009
Management not Agile
• Management not quite agile
▫ Very little training was given to managers
• Still division between Dev and QA
Manager
Community
Customer
Community
Developer
Community
17
Orlando Scrum
Gathering
3/16-18/2009
Scrum to the rescue
• Scrum is a management framework
• Training focused on the managers and product
managers
▫ 60 ScrumMaster receive CSM from Ken Schwaber
▫ 30 Product Owners receive CSPO from Ken Schwaber
• Create true cross-functional teams
▫ No more “communities”
▫ 1 ScrumMaster, 1Product Owner, Team of developers,
testers, writer, etc.
• Managers became ScrumMasters, Product Managers
are now the Product Owners
18
Orlando Scrum
Gathering
3/16-18/2009
19
Orlando Scrum
Gathering
3/16-18/2009
Large Interdependent Suite
CTO/Executive
VP
VP Applications
Director –
Flagship
Director – Addons
VP Customer
Engineering
VP Systems
Director –
Development
Director – QA
Director –
Development
• About 30 teams need to coordinate effort to release the suite
• Deemed impossible to synchronize sprints
Director – QA
20
Orlando Scrum
Gathering
3/16-18/2009
Parallel Project Organization
• Suite Management Team (SMT)
▫ Steering committee
▫ 1 Director and 1 Product Owner from each of the 3
organizations
• Suite Integration Team (SIT)
▫ Installs suite on a set of integration servers each week
▫ Creates and runs series of suite-wide tests
• Suite Release Team (SRT)
▫ ScrumMaster and Product Owner from each team (60
people)
▫ Responsible for coordinating dependencies and
resolving suite-wide issues
▫ Primary communication vehicle for the suite
21
Orlando Scrum
Gathering
3/16-18/2009
How do they synchronize and review
the suite?
• Created the concept of “Release Candidates”
▫ Every 6 to 8 weeks
▫ Entire system integrated and tested together
▫ Suite build and battery of suite tests run weekly
• Teams hold their own sprints but integrate into
last integrated system each week
▫ Each team has daily builds and battery of daily
tests (some team build more often)
22
Orlando Scrum
Gathering
3/16-18/2009
Heartbeats
Sprint 1
Team 3 heartbeat
Team 2 heartbeat
Team 1 heartbeat
Suite heartbeat
Sprint 1
Sprint
1
Sprint 2
Sprint 2
Sprint
2
RC1
Sprint
3
Sprint 3
Sprint 4
Sprint 3
Sprint
4
Sprint 5
Sprint 4
Sprint
5
Sprint
6
Sprint 6
Sprint 5
Sprint
7
RC2
Sprint 6
Sprint
8
RC3
• Beginning each RC
• Each team presents release plan focusing on
dependencies and impacts to SRT
• End of each RC
• SRT holds an RC Retrospective
• Product Owners hold an RC Review for
organization
Sprint 7
Sprint
9
23
Orlando Scrum
Gathering
3/16-18/2009
24
Orlando Scrum
Gathering
3/16-18/2009
Dealing with suboptimization
• Each team is run fairly independently
• Each identify and resolve impediments
▫ Experiencing the same impediments
▫ Different solutions
▫ Some solutions impacted other teams
25
Orlando Scrum
Gathering
3/16-18/2009
First Attempt – Etc.
• Senior Execs doing the work of prioritizing and
resolving organizational impediments
▫ Identified by the development teams
▫ Resolve by creating small teams to remove the
impediment
▫ Use Scrum to run the process
• Issues
▫ Execs had no time for this work
▫ Reluctant to assign people to impediment removal
teams
26
Orlando Scrum
Gathering
3/16-18/2009
Second Attempt – Scrum
Implementation Team
• Small group of 8 people from across the
organization
• Issues
▫ Focused more on process definition instead of
impediment removal
▫ Not all the skills represented
▫ Limited time to work on this, still had day jobs
27
Orlando Scrum
Gathering
3/16-18/2009
Third Attempt – ScrumMaster Meeting
• Hold a regular meeting of ScrumMasters to
identify, prioritize, and volunteer to resolve
impediments
• This group got some traction
• Issues
▫ Many impediments around Product Ownership
▫ Also many architectural impediments
28
Orlando Scrum
Gathering
3/16-18/2009
Final Attempt – Agile Leaders Meeting
• Regular meeting of ScrumMasters, Product
Owners, and Architects led by Agile Coach
• Agenda includes a brief coaching session on an
agile topic from one of the team
• Review of resolved impediments, identifying and
prioritizing new ones, and volunteering to
resolve highest priority
• Q & A session to solicit help from team and
coach on individual issues
• Resolved over 50 impediments in 1 year
29
Orlando Scrum
Gathering
3/16-18/2009
30
Orlando Scrum
Gathering
3/16-18/2009
Transparency is more translucent
• Each team uses their own tools for tracking data
▫ Flipcharts and post-its
▫ Excel – Various workbooks
▫ X-Planner
• Rollup data collected manually
▫ Sometimes late
▫ Sometimes not complete
▫ Apples to Oranges rollup
• “Blame” culture prevents full disclosure too soon
31
Orlando Scrum
Gathering
3/16-18/2009
Clearing it up
• SMT created a standard workbook that
normalizes data to facilitate a suite-wide rollup
• Includes a projected organizational velocity to
predict end date
“I have never been in an organization that knew
more about where it was throughout the lifecycle
of the project” – One Senior Staff Member
32
Orlando Scrum
Gathering
3/16-18/2009
Clearing it up more
• Implementing VersionOne
▫ Centralize the project information
▫ Give execs instant access to status of any project
▫ Standardize how information is collected and
managed throughout the organization
▫ Simplify suite rollup
▫ Provide unprecedented transparency
• Create standard reports
33
Orlando Scrum
Gathering
3/16-18/2009
34
Orlando Scrum
Gathering
3/16-18/2009
Suite planning still phase-gated
• Need still exists to commit to an annual plan
• Company expects large features to justify the
700 person engineering staff
• Outside engineering still driven by “waterfall”
model
▫ Cannot or will not take advantage of iterative
delivery
35
Orlando Scrum
Gathering
3/16-18/2009
Creating a Balance
• They created a multi-tiered content strategy
▫ Commit at the high level
▫ Establish budgets at the mid-level
▫ Stay flexible at the details
• Budgets give guidance to Product Owners on
what senior management is willing to invest to
get the benefit or return
36
Orlando Scrum
Gathering
3/16-18/2009
Requirements hierarchy
• Initiatives – Broad areas of focus tied to corporate strategy
• Headlines – Major Feature Sets/Capabilities within an
Initiative
▫ Engineering commits to these within a budget
• Shippable Units – The smallest feature that is worth
shipping to a customer
▫ Analogous to Minimal Marketable Feature from Software by
Numbers
▫ Engineering delivers as many of these within the budget of a
headline
▫ Business Value measured here
• Stories – User stories as we all know and love
37
Orlando Scrum
Gathering
Planning Onion
Strategy – Sets Initiatives
Portfolio – Headlines to Products
Product – Shippable Units to RCs
Release - Stories to Sprints
Sprint – Tasks to Stories
Daily - Tasks
3/16-18/2009
Planning and Outputs
Portfolio Plan
Rel 6.2
Headline
Headline
Headline
Headline
Headline
Headline
Headline
Headline
Rel 6.1.1
Headline
Headline
Headline
Headline
Product Plan
RC 1
RC 2
RC 3
RC 4
SU
SU
SU
SU
SU
SU
SU
SUSU
SU
SU
SU
SU
SU
SU
SU
SU
SU
SU
SU
Release Plan
Sprint 1
Sprint 2
Sprint 3
Rel 6.1.2
Headline
Headline
Headline
Sprint Plan
Task
Task
Task
Started
Done
Task
Task
Task
39
Orlando Scrum
Gathering
3/16-18/2009
40
Orlando Scrum
Gathering
3/16-18/2009
Unpredictable “tail”
• Planned 3 months for final stabilization
▫ Always goes longer
▫ Many unpredictable issues found
• The Definition of Done (DoD) was not the same
across teams
▫ Some due to the large amount of legacy code
▫ Some based on management interpretation –
“Spirit of the Criteria”
41
Orlando Scrum
Gathering
3/16-18/2009
Executive Bias
• Setting DoD too “weak” - guideline
▫ Leads to too much work during the tail
▫ Makes it hard to know where you are
▫ Causes integration issues due to impedance mismatch
• Setting DoD too “Strong” - rule
▫ No real progress can be made due to late issues
▫ Sets the bar too low on content
• Executives wants team to “stretch” by setting
aggressive DoD but knows he’ll get less (and accepts
it)
▫ This line is not apparent to the teams
42
Orlando Scrum
Gathering
3/16-18/2009
Created 2 lists of criteria
• Definition of Done
▫ This is the must be complete to declare victory
• Quality Goal
▫ Strive to complete and must justify shortfall
• Example
▫ DoD – Performance test must be completely run
▫ Quality Goal – Performance must not have
degraded by more than 5% with no S1 defects
43
Orlando Scrum
Gathering
3/16-18/2009
DoD is too much for each Sprint
• Doing all the work needed for shipping each
sprint is considered wasteful
• Some tests run almost as long as a sprint!
• Lots of legacy code with manual tests
44
Orlando Scrum
Gathering
3/16-18/2009
Modify DoD and QG by timebox
• Defined unambiguously by SMT
• For Ship – include everything that must be done
• For RC – remove what they are unable or
unwilling to do each RC
• For Sprint – remove what they are unable or
unwilling to do each Sprint
45
Orlando Scrum
Gathering
3/16-18/2009
46
Orlando Scrum
Gathering
3/16-18/2009
Increased Automation
60,000
50,000
40,000
30,000
20,000
10,000
0
RC1
RC2
RC3
RC4
RC5
RC6
Automated Tests Run
RC7
RC8
47
Orlando Scrum
Gathering
3/16-18/2009
What happened to quality
Open Defects
5000
4000
3000
V6.0
Pre-Scrum
Scrum
V6.1
2000
1000
0
0
10
20
30
40
Weeks
50
60
70
48
Orlando Scrum
Gathering
3/16-18/2009
49
Orlando Scrum
Gathering
3/16-18/2009
Enticing rest of Organization to be
more Agile
• There is still a “batch” approach to preparing
Service and Customer training
• Marketing not strongly tied in with Product
Management and the Product Owners
50
Orlando Scrum
Gathering
3/16-18/2009
Performance and Compensation still
very individualized
• Has a negative effect on teamwork
• Some teams view ScrumMaster/Manager as an
evaluator and aren’t as open with issues
• Individual goals trump team goals or
organizational goals
51
Orlando Scrum
Gathering
3/16-18/2009
Still treat people as “resources”
• Cannot break habit of moving people from team
to team
• Cannot break habit of needing “full utilization”
• Cannot break habit of accounting for hours
worked
52
Orlando Scrum
Gathering
3/16-18/2009
Recent Restructuring
• Increased fear
▫ Potential to reduce transparency
▫ Potential to reduce teamwork to “CYA”
• Lowered morale
▫ Lower energy
▫ Potential less commitment
53
Orlando Scrum
Gathering
3/16-18/2009
Loss of the Coach
• They eliminated the position of Agile Coach
• Entropy may set in again
▫ No focus for getting help to the teams
▫ Impediments mechanism may break down
▫ Training of new employees and organizations will
be affected
54
Orlando Scrum
Gathering
3/16-18/2009
55
Orlando Scrum
Gathering
3/16-18/2009
Where are they?
• Scrum is implemented throughout
• Mechanism for organizational impediments in
place
• Transparency is improving
• Improvements to engineering practices are ongoing
• Quality is improving
• Planning is becoming more flexible
56
Orlando Scrum
Gathering
Dan LeFebvre
Director, Software Engineering; Agile Coach
[email protected]
http://PracticalAgileByDan.blogspot.com
3/16-18/2009