DOCUMENTING THE DEPLOYMENT PIPELINE FOR TIGHTLY COUPLED ARCHITECTURES

Chapter 7 DOCUMENTING THE DEPLOYMENT PIPELINE FOR TIGHTLY COUPLED ARCHITECTURES

Documenting and optimizing the DP for tightly coupled architec- tures is an important concept and what all of this book’s chapters have been leading up to. We have built the foundation and put every- thing into context so you can determine how to handle the com- plexity of large systems and appreciate the differences from DevOps for small independent teams. For tightly coupled architectures that require hundreds or even thousands of engineers to coordinate the development, qualification, and deployment of code, the DP gets more complex. One way to try to work with this complexity is to have one big continuous integration process on the front end of the pipeline where everyone checks in and does multiple builds a day. A complex enterprise system can take a long time to deploy, however, and automated testing for everything on that complex of an environ- ment can take a while. It can result in builds with changes from too many developers, which makes it challenging to localize the issues during the triage process and hard to keep the builds green. So it is important when designing complex DPs to break the design down into more manageable pieces. This can be done by increasing the build frequency, reducing the test time by running a subset auto- mated testing defined as BAT, and using service virtualization to break the system into smaller more manageable pieces. Once you

CHAPTER 7 DOCUMENTING THE DEPLOYMENT PIPELINE FOR TIGHTLY COUPLED ARCHITECTURES 65

showing the couplings and interfaces. If you can build, deploy, and run a reasonable set of BAT tests on the system frequently enough to have a small enough number of changes that triage is efficient, then you should use one large continuous integration process for the entire system.

If you can’t build frequently enough to keep the number of commits small, which is typical of large complex enterprise systems, think about how to break this up into smaller, more manageable pieces. Look for clean interfaces where it would be easy to mock the inter- face with service virtualization. Ideally, you would not need this step at all because maintaining the service virtualization is going to take some effort. Therefore, you should avoid the step where possible by just using build frequency to localize the offending code, but when the build time of this complex system takes too long or there are just so many developers working on the system that you have over

20 commits per build, it probably makes sense to break it up into smaller subsystems that you can keep stable for integration. When

66 STARTING AND SCALING DEVOPS IN THE ENTERPRISE

Once the subsystems that can be optimized based on build frequency are defined, you need to define the DP for building each of them. This should start with continuous integration or stage 1 of the DP for each application, with each team owning their component with a quality gate for keeping major issues out of the subsystem build. This gate at each stage will be defined by a subset of BATs that will define the minimal level of stability that you will ever allow in the system.

Next, the DP should take the latest green builds out of each of the components and build those into a subsystem with automated sub- system BAT testing running against the service virtualization as many times a day as possible. Stage 2 of the DP is really continuous integration of the components of continuous integration that have to work together running the subsystem BAT. Stage 3 takes the latest green build each day out of stage 2 and runs the full set of auto- mated regression tests. This step is designed to catch issues that slip past the BAT. If you see major drops in the regression passing rates

CHAPTER 7 DOCUMENTING THE DEPLOYMENT PIPELINE FOR TIGHTLY COUPLED ARCHITECTURES 67

Keeping this code base up and running with green builds at this sub- system level should be the top priority for everyone in this part of the organization. Creating subsystems around natural organizational boundaries where possible helps by providing clarity in respon- sibilities for keeping these subsystems stable. Having BAT gates at each stage helps keep the large systems as stable as possible while localizing the ownership for fixing issues to the teams that created the problem. These subsystems are then built and tested in the full system without virtualization as frequently as possible, using BAT testing in stage 4 and then full regression in stage 5 before going into production. This is ideally how code should flow through your sys- tem. It is nice to start with the end point in mind.

68 STARTING AND SCALING DEVOPS IN THE ENTERPRISE

Next, understand how your current DP works. You may not call it