Description on the Minimums

• 1: Trunk Based Development
• 2: Continuous Integration
• 3: Immutable Artifact
• 4: Prod-Like Test Environment

1 - Trunk Based Development

Excerpt from Accelerate by Nicole Forsgren Ph.D., Jez Humble & Gene Kim

Definition

TBD is a team workflow where changes are integrated into the trunk with no intermediate integration (Develop, Test, etc.) branch. The two common workflows are making changes directly to the trunk or using very short-lived branches that branch from the trunk and integrate back into the trunk.

It is important to note that release branches are an intermediate step that some chose on their path to continuous delivery while improving their quality processes in the pipeline. True CD releases from the trunk.

What is Improved

• Smaller changes: TBD emphasizes small, frequent changes that are easier for the team to review and more resistant to impactful merge conflicts. Conflicts become rare and trivial.
• We must test: TBD requires us to implement tests as part of the development process.
• Better teamwork: We need to work more closely as a team. This has many positive impacts, not least we will be more focused on getting the team’s highest priority done. We will stop starting and start finishing work.
• Better work definition: Small changes require us to decompose the work into a level of detail that helps uncover things that lack clarity or do not make sense. This provides much earlier feedback on potential quality issues.
• Replaces process with engineering: Instead of creating a process where we control the release of features with branches, we can control the release of features with engineering techniques called evolutionary coding methods. These techniques have additional benefits related to stability that cannot be found when replaced by process.
• Reduces risk: There are two risks with long-lived branches that happen frequently. First, the change will not integrate cleanly and the merge conflicts result in broken or lost features. Second, the branch will be abandoned. This is usually because of the first reason. Sometimes because all of the knowledge about what is in that branch resides in the mind of someone who decided to leave before it was integrated.

2 - Continuous Integration

Definition

While CI depends on tooling, the team workflow and working agreement are more important.

1. We will work as a team to define work with testable acceptance criteria. Those acceptance criteria will drive our testing efforts.
2. No work will be committed to version control unless accompanied by all required tests.
3. Work committed to version control may not be “feature complete”, but it must not break existing work.
4. All work begins from the trunk and integrates into the trunk at least daily.
5. If the CI server detects an error, the team stops feature work and collaborates to fix the build. We should not create more unverified changes without the ability to receive quality feedback from the CI server.

Recommended practices

Evolutionary coding methods:

• Keystone Interfaces (A.K.A Dark Launching) lets you deploy some portion of the code to production without being visible or usable by end-users. It can also let you review metrics on how well the feature behaves performance-wise before making it accessible.
• Branch by abstraction is a good process for replacing existing new behaviors or frameworks with something new while constantly delivering. Also, a good pattern to use for A/B testing.
• Feature flags can be temporary tools for feature release management or permanent tools for enabling behaviors for different personas. They can also be controlled with application configuration or dynamically with logic.

What is Improved

• Teamwork: CI requires a lot of teamwork to function correctly. If the team currently uses a “push” workflow where work is assigned instead of a “pull” workflow where the next most important work is picked up by the next available teammate, then CI will be very difficult. Teamwork suffers because everyone is focused on their individual “assignments” rather than team goals. Long delays in code review, large changesets, and excessive team process hurt outcomes. Find a cadence for code review to process them quickly and set some team norms on changeset size and collaboration. Pair programming is a good way to help address these problems quickly.
• Work Breakdown: We need to break down work better. We should have a “Definition of Ready” that requires every story and task has a testable description of “done” before any work starts. A good rule of thumb is that if everyone agrees the team can complete that item in less than 2 days, it’s refined enough for CI.
• Testing: This is a common struggle. It’s common that teams either do not test or know little about basic unit testing. Testing implementation instead of behavior is another common issue. Teams will need to improve the efficiency and effectiveness of their tests and build a suite of various types of tests to move detection as close to creation as possible. CI requires falling passionately in love with testing, but that should be true of software engineering anyway.

Health Metrics

• Commits/Day/Developer: How frequently are we as a team integrating code to the trunk each day? “Good” is 1 or more per day per dev. Never compare individuals. this is a team average.
• Development cycle time: The time from when work begins until it is delivered to the end-user. “Good” is less than 2 days on average.
• Defect rate: Critical backpressure metric to ensure speed does not overtake quality and vice versa.

FAQ

“How do I complete a large feature in less than a day?”

You probably don’t. However, there are several strategies available for making evolutionary changes that build toward the complete feature. See Recommended practices.

“What tests should run during CI?”

Our goal is to detect issues as early as possible. Any functional tests that can be executed without deploying the application should be run. This includes, but is not limited to:

• Static code quality
• Static security scans
• Functional tests that do not require external services.

What code coverage level is needed before we can do CI?

You don’t need any tests in existing code to begin. You need to test new code without exception.

What code coverage percentage should we have?

“I’m confident”. Are you confident you’ve covered enough positive and negative cases to make you confident?

What code coverage percentage should we set as a standard for our team?

“We will not go lower than the current level of code coverage.” However, if the team is not committed to a disciplined quality process to the extent that delivery dates are never seen as an excuse, this could incentivize fake tests to meet the coverage minimum.

What code coverage percentage should we set as a standard for all teams?

We shouldn’t. Code coverage mandates incentivize meaningless tests that hide the fact that code is not tested. It is better to have no tests than to have tests you do not trust. See the Dojo Consortium’s info on this metric.

3 - Immutable Artifact

Central to CD is that we are validating the artifact with the pipeline. It is built once and deployed to all environments. A common anti-pattern is building an artifact for each environment. The pipeline should generate immutable, versioned artifacts.

Definition

• Immutable Pipeline: In the beginning, it may seem that the obvious way to address a failure in the pipeline is to go to the failure point, make some adjustments in the environment, test data, or whatever else failed, and then to re-start the pipeline from that point. However, that transforms a repeatable quality process into an untrustworthy custom build. Failures should be addressed by changes in version control so that two executions with the same configuration will always yield the same results.
• Immutable Artifacts: Some package management systems will allow the creation of release candidate versions. For example, it is common to find -SNAPSHOT versions used for this in Java. However, this means we have an artifact where the behavior can be changed without modifying the version. Version numbers are cheap. If we are to have an immutable pipeline, it must produce an immutable artifact. We should never have dependencies that use -SNAPSHOT versions and we should never produce -SNAPSHOT versions.

Immutability provides us with the confidence to know that the results from the pipeline are real and repeatable.

What is Improved

• Everything must be version controlled: source code, environment configurations, application configurations, and even test data. This reduces variability and improves the quality process.

4 - Prod-Like Test Environment

Definition

It is crucial to leverage pre-production environments in your CI/CD to run all of your tests (Unit / Integration / UAT / Manual QA / E2E) early and often. Test environments increase interaction with new features and exposure to bugs – both of which are important prerequisites for reliable software.

Example Implementations

There are different types of pre-production test environments. Most organizations will employ both static and short-lived environments and utilize them for case-specific stages of the SDLC.

• Staging environment: Ideally, this is the last environment that teams will run automated tests against prior to deployment, particularly for testing interaction between all new features after a merge. Its infrastructure will reflect production as closely as possible.
• Ephemeral environments (collected from EphemeralEnvironments.io): These are full-stack, on-demand environments that are spun up on every code change. Each ephemeral environment should be leveraged in your pipeline, which will run E2E, unit, and integration tests against them on every code change. These environments are defined in version control and created and destroyed automatically on demand. They are short-lived by definition but should closely resemble production; they are intended to replace long-lived “static” environments and the maintenance required to keep those stable, i.e., “development,” “QA1”, “QA2”, “testing,” etc.

What is Improved

• Infrastructure is kept consistent: Test environments deliver results that reflect real-world performance. Few unprecedented bugs sneak into production since using prod-like data and dependencies allows you to run your entire test suite earlier against multiple prod-like environments.
• Test against latest changes: These environments will rebuild upon code changes with no manual intervention.
• Test before merge: Attaching an ephemeral environment to every PR enables E2E testing in your CI before code changes get deployed to staging. New features get tested in parallel, avoiding the dreaded “waiting to run my tests” blocking your entire SDLC.