“Continuous delivery improves both delivery performance and quality, and also helps improve culture and reduce burnout and deployment pain.”
– Accelerate by Nicole Forsgren Ph.D., Jez Humble & Gene Kim
_
We, the undersigned, believe that a minimal definition of continuous delivery (CD) is required to improve the flow of delivery and achieve the outcomes above. While our contexts may be different, there are universal practices common in all. By defining them we can:
Introduce new practitioners in a consistent way
Discuss engineering practices that comprise CD
Help each other improve current capabilities
Only by implementing core practices do we begin to see the benefits of continuous delivery.
The practices below are the minimum, a starting point. Continuous improvement of the speed, quality, and safety of the delivery pipeline is the expected outcome.
Continuous Delivery
CD is the engineering discipline of delivering all changes in a standard way safely. It covers a broad spectrum of activities depending on what is being delivered. However, there are behaviors and abilities that must be met in every context to qualify as “continuous delivery”
CI is the activity of very frequently integrating work to the trunk of version control and verifying that the work is, to the best of our knowledge, releasable.
Work is tested with other work automatically on merge
All feature work stops when the build is red
New work does not break delivered work
Trunk-based Development
Trunk-based development is the branching pattern required to meet the definition
of CI. It prevents lost work, the risk of corruption that comes from merge conflict resolution, and also reduces movement
waste that increases batch size.
The minimum activities required for TBD are:
All changes integrate into the trunk
If branches from the trunk are used:
They originate from the trunk
They re-integrate to the trunk
They are short-lived and removed after the merge
Why did we build this?
For background on Minimum CD and answers to other common questions, please read the FAQs.
Starting the Journey
Questions on where to start? Check out some recommendations.
Contributing
Do you want to submit a translation, good practices, suggestions, or an experience report?
“Trunk-based development has been shown to be a predictor of high performance in software development and delivery. It is characterized by fewer than three active branches in a code repository; branches and forks having very short lifetimes (e.g., less than a day) before being merged; and application teams rarely or never having “code lock” periods when no one can check in code or do pull requests due to merging conflicts, code freezes, or stabilization phases.”
Excerpt from Accelerate by Nicole Forsgren Ph.D., Jez Humble & Gene Kim
Definition
TBD is a team workflow where changes are integrated into the trunk with no intermediate integration (Develop, Test, etc.) branch. The two common workflows are making changes directly to the trunk or using very short-lived branches that branch from the trunk and integrate back into the trunk.
It is important to note that release branches are an intermediate step that some chose on their path to continuous delivery while improving their quality processes in the pipeline. True CD releases from the trunk.
What is Improved
Smaller changes: TBD emphasizes small, frequent changes that are easier for the team to review and more resistant to impactful merge conflicts. Conflicts become rare and trivial.
We must test: TBD requires us to implement tests as part of the development process.
Better teamwork: We need to work more closely as a team. This has many positive impacts, not least we will be more focused on getting the team’s highest priority done. We will stop starting and start finishing work.
Better work definition: Small changes require us to decompose the work into a level of detail that helps uncover things that lack clarity or do not make sense. This provides much earlier feedback on potential quality issues.
Replaces process with engineering: Instead of creating a process where we control the release of features with branches, we can control the release of features with engineering techniques called evolutionary coding methods. These techniques have additional benefits related to stability that cannot be found when replaced by process.
Reduces risk: There are two risks with long-lived branches that happen frequently. First, the change will not integrate cleanly and the merge conflicts result in broken or lost features. Second, the branch will be abandoned. This is usually because of the first reason. Sometimes because all of the knowledge about what is in that branch resides in the mind of someone who decided to leave before it was integrated.
2 - Continuous Integration
Start here
Definition
While CI depends on tooling, the team workflow and working agreement are more important.
We will work as a team to define work with testable acceptance criteria. Those acceptance criteria will drive our testing efforts.
No work will be committed to version control unless accompanied by all required tests.
Work committed to version control may not be “feature complete”, but it must not break existing work.
All work begins from the trunk and integrates into the trunk at least daily.
If the CI server detects an error, the team stops feature work and collaborates to fix the build. We should not create more unverified changes without the ability to receive quality feedback from the CI server.
Recommended practices
Evolutionary coding methods:
Keystone Interfaces (A.K.A Dark Launching) lets you deploy some portion of the code to production without being visible or usable by end-users. It can also let you review metrics on how well the feature behaves performance-wise before making it accessible.
Branch by abstraction is a good process for replacing existing new behaviors or frameworks with something new while constantly delivering. Also, a good pattern to use for A/B testing.
Feature flags can be temporary tools for feature release management or permanent tools for enabling behaviors for different personas. They can also be controlled with application configuration or dynamically with logic.
What is Improved
Teamwork: CI requires a lot of teamwork to function correctly. If the team currently uses a “push” workflow where work is assigned instead of a “pull” workflow where the next most important work is picked up by the next available teammate, then CI will be very difficult. Teamwork suffers because everyone is focused on their individual “assignments” rather than team goals. Long delays in code review, large changesets, and excessive team process hurt outcomes. Find a cadence for code review to process them quickly and set some team norms on changeset size and collaboration. Pair programming is a good way to help address these problems quickly.
Work Breakdown: We need to break down work better. We should have a “Definition of Ready” that requires every story and task has a testable description of “done” before any work starts. A good rule of thumb is that if everyone agrees the team can complete that item in less than 2 days, it’s refined enough for CI.
Testing: This is a common struggle. It’s common that teams either do not test or know little about basic unit testing. Testing implementation instead of behavior is another common issue. Teams will need to improve the efficiency and effectiveness of their tests and build a suite of various types of tests to move detection as close to creation as possible. CI requires falling passionately in love with testing, but that should be true of software engineering anyway.
Health Metrics
Commits/Day/Developer: How frequently are we as a team integrating code to the trunk each day? “Good” is 1 or more per day per dev. Never compare individuals. this is a team average.
Development cycle time: The time from when work begins until it is delivered to the end-user. “Good” is less than 2 days on average.
Defect rate: Critical backpressure metric to ensure speed does not overtake quality and vice versa.
FAQ
“How do I complete a large feature in less than a day?”
You probably don’t. However, there are several strategies available for making evolutionary changes that build toward the complete feature. See Recommended practices.
“What tests should run during CI?”
Our goal is to detect issues as early as possible. Any functional tests that can be executed without deploying the application should be run. This includes, but is not limited to:
What code coverage level is needed before we can do CI?
You don’t need any tests in existing code to begin. You need to test new code without exception.
What code coverage percentage should we have?
“I’m confident”. Are you confident you’ve covered enough positive and negative cases to make you confident?
What code coverage percentage should we set as a standard for our team?
“We will not go lower than the current level of code coverage.” However, if the team is not committed to a disciplined quality process to the extent that delivery dates are never seen as an excuse, this could incentivize fake tests to meet the coverage minimum.
What code coverage percentage should we set as a standard for all teams?
We shouldn’t. Code coverage mandates incentivize meaningless tests that hide the fact that code is not tested. It is better to have no tests than to have tests you do not trust. See the Dojo Consortium’s info on this metric.
3 - Immutable Artifact
No human changes after commit.
Central to CD is that we are validating the artifact with the pipeline. It is built once and deployed to all environments. A common anti-pattern is building an artifact for each environment. The pipeline should generate immutable, versioned artifacts.
Definition
Immutable Pipeline: In the beginning, it may seem that the obvious way to address a failure in the pipeline is to go to the failure point, make some adjustments in the environment, test data, or whatever else failed, and then to re-start the pipeline from that point. However, that transforms a repeatable quality process into an untrustworthy custom build. Failures should be addressed by changes in version control so that two executions with the same configuration will always yield the same results.
Immutable Artifacts: Some package management systems will allow the creation of release candidate versions. For example, it is common to find -SNAPSHOT versions used for this in Java. However, this means we have an artifact where the behavior can be changed without modifying the version. Version numbers are cheap. If we are to have an immutable pipeline, it must produce an immutable artifact. We should never have dependencies that use -SNAPSHOT versions and we should never produce -SNAPSHOT versions.
Immutability provides us with the confidence to know that the results from the pipeline are real and repeatable.
What is Improved
Everything must be version controlled: source code, environment configurations, application configurations, and even test data. This reduces variability and improves the quality process.
4 - Prod-Like Test Environment
Leveraging production-like test environments in your CI/CD pipeline enables reliable testing.
Definition
It is crucial to leverage pre-production environments in your CI/CD to run all of your tests (Unit / Integration / UAT / Manual QA / E2E) early and often. Test environments increase interaction with new features and exposure to bugs – both of which are important prerequisites for reliable software.
Example Implementations
There are different types of pre-production test environments. Most organizations will employ both static and short-lived environments and utilize them for case-specific stages of the SDLC.
Staging environment: Ideally, this is the last environment that teams will run automated tests against prior to deployment, particularly for testing interaction between all new features after a merge. Its infrastructure will reflect production as closely as possible.
Ephemeral environments (collected from EphemeralEnvironments.io): These are full-stack, on-demand environments that are spun up on every code change. Each ephemeral environment should be leveraged in your pipeline, which will run E2E, unit, and integration tests against them on every code change. These environments are defined in version control and created and destroyed automatically on demand. They are short-lived by definition but should closely resemble production; they are intended to replace long-lived “static” environments and the maintenance required to keep those stable, i.e., “development,” “QA1”, “QA2”, “testing,” etc.
What is Improved
Infrastructure is kept consistent: Test environments deliver results that reflect real-world performance. Few unprecedented bugs sneak into production since using prod-like data and dependencies allows you to run your entire test suite earlier against multiple prod-like environments.
Test against latest changes: These environments will rebuild upon code changes with no manual intervention.
Test before merge: Attaching an ephemeral environment to every PR enables E2E testing in your CI before code changes get deployed to staging. New features get tested in parallel, avoiding the dreaded “waiting to run my tests” blocking your entire SDLC.
