The Continuous Delivery
Maturity Model
second edition
by: Andreas Rehn, Tobias Palmborg
and Patrik Boström
The principles and methods of Continuous Delivery are rapidly gaining recognition as
a successful strategy for true business agility. For many organizations the question is
no longer “why?”, but rather “how?” How do you start with Continuous Delivery, and
how do you transform your organization to ensure sustainable results. This Maturity
Model aims to give structure and understanding to some of the key aspects you
need to consider when adopting Continuous Delivery in your organization.
Why a maturity model?
Continuous Delivery is all about seeing the big picture, to consider all aspects that affect the ability to
develop and release your software. For any non-trivial business of reasonable size this will
unfortunately include quite a lot of steps and activities. The end-to-end process of developing and
releasing software is often long and cumbersome, it involves many people, departments and
obstacles which can make the effort needed to implement Continuous Delivery seem overwhelming.
Where do we start? Do we have to do everything, what parts can we leave out? Where do we get the
biggest and quickest effect? These are questions that inevitably will come up when you start looking
at implementing Continuous Delivery.
This is why we created the Continuous Delivery Maturity Model, to give structure and understanding
to the implementation of Continuous Delivery and its core components. Inspiration for this model
comes mainly from our combined experiences in several continuous delivery implementation projects,
and also from selected books, papers and blog-posts on the subject where Jez Humble & David
1
Farley’s groundbreaking book Continuous Delivery and Eric Minick & Jeffrey Fredricks terrific
2
whitepaper Enterprise Continuous Delivery Model are two that deserve special recognition. With this
model we aim to be broader, to extend the concept beyond automation and spotlight all the key
aspects you need to consider for a successful Continuous Delivery implementation across the entire
organization.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1
2
http://www.amazon.com/dp/0321601912?tag=contindelive-20
http://www.urbancode.com/html/resources/white-papers/Enterprise_Continuous_Delivery_Maturity_Model/
A structured approach
The journey that started with the Agile movement a decade ago is finally getting a strong foothold in
the industry. Business leaders now have begun to embrace the fact that there is a new way of
thinking about software development. A paradigm shift, if you like, that will inevitably segment the
industry into those who struggle to keep up with competition, and those who have the ability to stay
agile and ahead of competition by having an IT organization that responds quickly and efficiently, and
serves as a true business enabler. IT can once again start pushing innovation instead of restraining it
by expensive, slow, unpredictable and outdated processes. There are many ways to enter this new
era and here we will describe a structured approach to attaining the best results. While agile
methodologies often are described to best grow from inside the organization we have found that this
approach also has limitations. Some parts of the organization are not mature enough to adapt and
consequently inhibit development, creating organizational boundaries that can be very hard to break
down. The best way to include the whole organization in the change is to establish a solid platform
with some important prerequisites that will enable the organization to evolve in the right direction. This
platform includes adopting specific tools, principles, methods and practices that we have organized
into five key categories, Culture & Organization, Design & Architecture, Build & Deploy, Test &
Versification and Information & Reporting. Structuring Continuous Delivery implementation into these
categories that follows a natural maturity progression will give you a solid base for a fast
transformation with sustainable results.
The purpose of the maturity model is to highlight these five essential categories, and to give you an
understanding of how mature your company is. Your assessment will give you a good base when
planning the implementation of Continuous Delivery and help you identify initial actions that will give
you the best and quickest effect from your efforts. The model will indicate which practices are
essential, which should be considered advanced or expert and what is required to move from one
level to the next.
Five levels
The model defines five maturity levels: base, beginner, intermediate, advanced and expert. The
model is calibrated with a typical industry average around the base level where we see most
organizations are at today. Some organizations have beginner maturity in some categories and below
base (outside the model) in others but on average base-level is the typical starting point for many
organizations. The intermediate level is what we consider a mature level of continuous delivery
practices where you benefit from the larger effects. The advanced level includes practices that will
bring substantial value and effect to a higher effort. The last maturity level, expert, includes practices
that will be very valuable for some organizations that want or need to specialize in certain practices.
The levels are not strict and mandatory stages that needs to be passed in sequence, but rather
should serve as a base for evaluation and planning. It is however important to try to keep the overall
maturity level fairly even and to keep in mind that big changes may cause skepticism and reluctance
in the organization, so an incremental approach to moving through the levels is recommended.
Five categories
The model also defines five categories that represent the key aspects to consider when implementing
Continuous Delivery. Each category has it’s own maturity progression but typically an organization will
gradually mature over several categories rather than just one or two since they are connected and will
affect each other to a certain extent.
Culture & Organization
The organization and it’s culture are probably the most important aspects to consider when aiming to
create a sustainable Continuous Delivery environment that takes advantage of all the resulting effects.
Base
A typical organization will have, at base level, started to prioritize work in backlogs, have some
process defined which is rudimentarily documented and developers are practicing frequent commits
into version control.
Beginner
Moving to beginner level, teams stabilize over projects and the organization has typically begun to
remove boundaries by including test with development. Multiple backlogs are naturally consolidated
into one per team and basic agile methods are adopted which gives stronger teams that share the
pain when bad things happen.
Intermediate
At the intermediate level you will achieve more extended team collaboration when e.g. DBA, CM and
Operations are beginning to be a part of the team or at least frequently consulted by the team.
Multiple processes are consolidated and all changes, bugs, new features, emergency fixes, etc, follow
the same path to production. Decisions are decentralized to the team and component ownership is
defined which gives teams the ability to build in quality and to plan for sustainable product and
process improvements.
Advanced
At the advanced level, the team will have the competence and confidence it needs to be responsible
for changes all the way to production. Continuous improvement mechanisms are in place and e.g. a
dedicated tools team is set up to serve other teams by improving tools and automation. At this level,
releases of functionality can be disconnected from the actual deployment, which gives the projects a
somewhat different role. A project can focus on producing requirements for one or multiple teams and
when all or enough of those have been verified and deployed to production the project can plan and
organize the actual release to users separately. This separation of concerns will optimize the projects
flexibility of packaging and releasing functionality and at the same time give the development teams
better flow and speed since they don’t have to stack up changes and wait for coordinated project
releases.
Expert
At expert level some organizations choose to make a bigger effort and form complete cross functional
teams that can be completely autonomous. With extremely short cycle time and a mature delivery
pipeline, such organizations have the confidence to adopt a strict roll-forward only strategy to
production failures.
Design & Architecture
The design and architecture of your products and services will have an essential impact on your
ability to adopt continuous delivery. If a system is built with continuous delivery principles and a rapid
release mind set from the beginning, the journey will be much smoother. However, an upfront
complete redesign of the entire system is not an attractive option for most organizations, which is why
we have included this category in the maturity model.
Base
A typical organization will have one or more legacy systems of monolithic nature in terms of
development, build and release. Many organizations at the base maturity level will have a diversified
technology stack but have started to consolidate the choice of technology and platform, this is
important to get best value from the effort spent on automation.
Beginner
At beginner level, the monolithic structure of the system is addressed by splitting the system into
modules. Modules gives a better structure for development, build and deployment but are typically not
individually releasable like components. Doing this will also naturally drive an API managed approach
to describe internal dependencies and also influence applying a structured approach to manage 3rd
party libraries. At this level the importance of applying version control to database changes will also
reveal itself.
Intermediate
Moving to intermediate level will result in a solid architectural base for continuous delivery when
3
adopting branch by abstraction and other techniques for feature hiding for the purpose of minimizing
repository branching to enable true continuous integration. At this level the work with modularization
will evolve into identifying and breaking out modules into components that are self-contained and
separately deployed. At this stage it will also be natural to start migrating scattered and ad-hoc
managed application and runtime configuration into version control and treat it as part of the
application just like any other code.
Advanced
At the advanced level you will have split the entire system into self contained components and
adopted a strict api-based approach to inter-communication so that each component can be deployed
and released individually. With a mature component based architecture, where every component is a
self-contained releasable unit with business value, you can achieve small and frequent releases and
extremely short release cycles. At this level it is easy for the business to experiment with rapid release
of new features, monitor and verify expected business results; therefore techniques to push and
graph business metrics from the application becomes important and natural part of the overall design
and architecture.
Expert
At expert level, some organizations will evolve the component based architecture further and value
the perfection of reducing as much shared infrastructure as possible by also treating infrastructure as
code and tie it to application components. The result is a system that is totally reproducible from
source control, from the O/S and all the way up to application. Doing this enables you to reduce a lot
of complexity and cost in other tools and techniques for e.g. disaster recovery that serves to ensure
that the production environment is reproducible. Instead of having a separate process, disaster
recovery is simply done by pushing out the last release from the pipeline like any other release. This
together with virtualization gives extreme flexibility in setting up test and production environments with
minimum manual effort.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
3!
http://paulhammant.com/blog/branch_by_abstraction.html!
Build & Deploy
Build and deployment is of course core to Continuous Delivery and this is where a lot of tools and
automation come into the pipeline; this is what is most commonly perceived when Continuous
Delivery is discussed. At first glance a typical mature delivery pipeline can be very overwhelming;
depending on how mature the current build and deployment process is in the organization, the
delivery pipeline can be more or less complex. In this category we will describe a logical maturity
progression to give structure and understanding to the different parts and levels it includes.
Base
At a base level you will have a code base that is version controlled and scripted builds are run
regularly on a dedicated build server. The deployment process is manual or semi-manual with some
parts scripted and rudimentarily documented in some way.
Beginner
Beginner level introduces frequent polling builds for faster feedback and build artifacts are archived
for easier dependency management. Tagging and versioning of builds is structured but manual and
the deployment process is gradually beginning to be more standardized with documentation, scripts
and tools.
Intermediate
At intermediate level, builds are typically triggered from the source control system on each commit,
tying a specific commit to a specific build. Tagging and versioning of builds is automated and the
deployment process is standardized over all environments. Built artifacts or release packages are
built only once and are designed to be able to be deployed in any environment. The standardized
deployment process will also include a base for automated database deploys (migrations) of the bulk
of database changes, and scripted runtime configuration changes. A basic delivery pipeline is in place
covering all the stages from source control to production.
Advanced
At this stage it might also become necessary to scale out the build to multiple machines for parallel
processing and for specific target environments. Techniques for zero downtime deploys can be
important to include in the automated process to gain better flexibility and to reduce risk and cost
when releasing. At this level you might also explore techniques to automate the trailing part of more
complex database changes and database migrations to completely avoid manual routines for
database updates.
Expert
Expert practices will include zero touch continuous deployment to production where every commit can
potentially make it all the way to production automatically. Another expert practice is taking the
infrastructure as code and virtualization even further and making the build process produce not only
deployment artifacts but complete virtual machines which are promoted down the pipeline all the way
to production replacing the existing ones.
Test & Verification
Testing is without doubt very important for any software development operation and is an absolutely
crucial part of a successful implementation of Continuous Delivery. Similar to Build & Deploy, maturity
in this category will involve tools and automation. However, it is also important to constantly increase
the test-coverage of the application to build up the confidence in speed with frequent releases.
Usually test involves verifying expected functionality according to requirements in different ways but
we also want to emphasize the importance of verifying the expected business value of released
features.
Base
At the base stage in the maturity model a development team or organization will typically practice unittesting and have one or more dedicated test environments separate from local development
machines. This system and integration level testing is typically done by a separate department that
conducts long and cumbersome test periods after development "code freeze".
Beginner
When moving to beginner level you will naturally start to investigate ways of gradually automating the
existing manual integration testing for faster feedback and more comprehensive regression tests. For
accurate testing the component should be deployed and tested in a production like environment with
all necessary dependencies.
Intermediate
Beginner level lets you broaden the scope of your automatic tests to functional testing which includes
bigger parts of the component than unit tests but will have “mocked” implementations of it’s external
dependencies, e.g. database or other backend services. These tests are especially valuable when
working in a highly component based architecture or when good complete integration tests are difficult
to implement or too slow to run frequently. At this level you will most likely start to look at gradually
automating parts of the acceptance testing. While integration tests are component specific,
acceptance tests typically span over several components and across multiple systems.
Advanced
Advanced practices include fully automatic acceptance tests and maybe also generating structured
acceptance criteria directly from requirements with e.g. specification by example and domains specific
languages. This means no manual testing or verification is needed to pass acceptance but typically
the process will still include some exploratory testing that feeds back into automated tests to
constantly improve the test coverage and quality. If you correlate test coverage with change
traceability you can start practicing risk based testing for better value of manual exploratory testing. At
the advanced level some organizations might also start looking at automating performance tests and
security scans.
Expert
It might seem strange to state that verifying expected business result is an expert practice but this is
actually something that is very rarely done as a natural part of the development and release process
today. Verifying expected business value of changes becomes more natural when the organization,
culture and tooling has reached a certain maturity level and feedback of relevant business metrics is
fast and accessible. As an example the implementation of a new feature must also include a way to
verify the expected business result by making sure the relevant metrics can be pulled or pushed from
the application. The definition of done must also be extended from release to sometime later when
business has analyzed the effects of the released feature or change..
Information & Reporting
Any business process includes a specific information set which can be reported on; the process of
developing and releasing software is no exception, and the set of information that is handled in this
particular area would typically include concepts like component, requirement, version, developer,
release, environment etc. In this category we want to show the importance of handling this information
correctly when adopting Continuous Delivery. Information must e.g. be concise, relevant and
accessible at the right time to the right persons in order to obtain the full speed and flexibility possible
with Continuous Delivery. Apart from information directly used to fulfill business requirements by
developing and releasing features, it is also important to have access to information needed to
measure the process itself and continuously improve it.
Base
At the base level in this category it is important to establish some baseline metric for the current
process, so you can start to measure and track. At this level reporting is typically done manually and
on-demand by individuals. Interesting metrics can e.g. be cycle-time, delivery time, number of
releases, number of emergency fixes, number of incidents, number of features per release, bugs
found during integration test etc.
Beginner
At beginner level, you start to measure the process and track the metrics for a better understanding of
where improvement is needed and if the expected results from improvements are obtained. Reporting
at this stage would typically include static analysis of code and quality reports which might be
scheduled so that the latest reports are always accessible to facilitate decisions on quality and where
improvements are needed.
Intermediate
Moving to intermediate the level of automation requires you to establish a common information model
that standardizes the meaning of concepts and how they are connected. This model will typically give
answers to questions like; what is a component?; how are requirements related to changes and
components? When this model is established not only can you automate build, test and deployment
even further but you can also build in traceability and information transparency to the pipeline and e.g.
automatically generate information like release notes and test plans. Automatic reporting and
feedback on events is implemented and at this level it will also become natural to store historical
reports connected to e.g. builds or other events. This gives management crucial information to make
good decisions on how to adjust the process and optimize for e.g. flow and capacity.
Advanced
At advanced level when you have started to practice frequent releases and the tools, products and
services are mature enough to e.g. push business metrics, a natural progression is to set up a real
time reporting/graphing service where people can get specific real time information that is relevant to
their specific need. At this level real time graphs and other reports will typically also include trends
over time. As a complement to static code analysis and unit tests coverage reports you might also at
this level start looking at dynamic test coverage and profiling information from production like runtime
environments when e.g. running automatic integrations tests.
Expert
Moving to expert level in this category typically includes improving the real time information service to
provide dynamic self-service useful information and customized dashboards. As a result of this you
can also start cross referencing and correlating reports and metrics across different organizational
boundaries,. This information lets you broaden the perspective for continuous improvement and more
easy verify expected business results from changes.
Jump start the journey
Every company is unique and has its own specific challenges when it comes to changing things, like
implementing Continuous Delivery. This maturity model will give you a starting point and a base for
planning the transformation of the company towards Continuous Delivery. After evaluating your
organization according to the model you need to set the goals and identify which practices will give
your organization the best outcomes. If there are practices you do not want to adopt you need to
analyse the consequences of excluding them. It is also important to decide on an implementation
strategy, you can e.g. start small using slack in the existing process to improve one thing at a time.
However, from our experience you will have a better chance of a successful implementation if you
jump start the journey with a dedicated project with a clear mandate and aggressive goals on e.g.
reducing cycle time. A typical Continuous Delivery implementation project will need skills and
competence according to the categories and practices in the maturity model in order to implement a
solid platform of tools, methods and practices that the organization can continue to grow and improve
on.
About the authors
Andreas Rehn is an Enterprise Architect and a strong advocate for Continuous Delivery, DevOps, Agile
and Lean methods in systems development. With extensive experience in many disciplines of software
development and a solid understanding of process, information and management theories and practises;
he is dedicated to help customers implement Continuous Delivery and transform their business by
adopting new methods for efficient and modern software development.
Tobias Palmborg, Believes that Continuous Delivery describes the vision that scrum, XP and the agile
manifesto once set out to be. Continuous Delivery is not just about automating the release pipeline but
how to get your whole change flow, from grain to bread ,in a state of the art shape. Former Head of
Development at one of europes largest online gaming company. Tobias is currently implementing
Continuous Delivery projects at several customers.
Patrik Boström is one of the founders of Diabol. Senior developer and architect with experience in
operations of large system. Strong believer that Continuous Delivery and DevOps is the natural step in
the evolution of Agile and Lean movement. Wants to change the way we look at systems development
today, moving it to the next level where we focus more time on developing features than doing manually
repetitive tasks. Where we visualize and understand the path from idea to where it is released and brings
business value
About Diabol
Diabol is a Swedish company specialized in modern agile software development.
We make sure you get the best out of your IT department by maximizing the throughput and free
resources for customer value adding activities. We are specialists in creating a business oriented culture
with the help of Processes and Tools.
We believe effective software development is a business enabler that frees your vision. We value the
whole system development process and our team foundation not only makes sure you get the best of our
expertise it also spreads our wealth of experience to your development team.
With the help of our knowledge in agile methods, processes and tools we produce business value faster
with less waste, higher quality, and at a lower cost. We join forces with our clients, to make sure your
system development process is as efficient as possible, working through the impediments one by one as
they come up.
Our extensive experience within the system development domain and especially with a focus on Java
development puts us in a unique position that enable us to implement processes and cultural change on a
hands-on level opposed to traditional management consultants with their high flying PPT presentations.
Some of our specialities are: Java, DevOps, Continuous Delivery, Application Servers, Continuous
Integration, System Development Process, Lean and Agile System Development
http://www.diabol.se/