An Introduction To CI/CD
The adoption of CI/CD has changed how developers and testers ship software. This blog is the first in a series of posts about this transition that will provide insights into different tools and process changes which can help developers be more successful with CI/CD. Dev approaches are always changing: first it was Waterfall, then it was Agile, and now it's DevOps. This is how modern developers approach building great products. The new methods of Continuous Integration, Continuous Delivery, (CI/CD) and Continuous Deployment have come with the rise of DevOps. Conventional software development and delivery methods are rapidly becoming obsolete.
In the Agile age, most companies would deploy/ship software in monthly, quarterly, bi-annual, or even annual releases (remember those days?). In the DevOps era, weekly, daily, and even multiple times a day deployments are the norm. With SaaS taking over the world, you can easily update applications on the fly without forcing customers to download new components. Often times, if your CI pipeline is working right, they won’t even realize an update has happened.
Development teams have adapted to the shortened delivery cycles by embracing automation across their software delivery pipeline. Most teams have automated processes to check in code and deploy to new environments. These adaptations have birthed the CI/CD process and have been coupled with a focus on automating the testing process, which we’ll cover in a future article. Today, we’ll cover CI/CD, what it is, and how modern software companies are using CI/CD tools to automate the process of shipping new code.
Continuous integration (CI) focuses on blending the work products of individual developers together into a repository. This can be done several times each day, and the primary purpose is to enable early detection of integration bugs while also allowing for tighter cohesion and more development collaboration.
The aim of continuous delivery (CD) is to minimize the friction points that are inherent in the deployment or release processes. Typically, a team's implementation involves automating each of the steps for build deployments so that a safe code release can be done at any moment in time.
Continuous deployment is a higher degree of automation, in which a build/deployment occurs automatically whenever a major change is made to the code.
Each of these stages is part of a delivery pipeline. In their book, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, Humble and Farley explain how every change to your software “goes through a complex process on its way to being released. That process involves building the software, followed by the progress of these builds through multiple stages of testing and deployment. This, in turn, requires collaboration between many individuals and possibly several teams. The deployment pipeline is a model of this process, and its incarnation in a continuous integration and release management tool is what allows you to see and control the progress of each change as it moves from version control through various sets of tests and deployments to release to users.”
A basic deployment pipeline
Continuous Integration (CI)
With continuous integration, developers frequently integrate their code into a main branch of a common repository. Rather than building features in isolation and submitting each of them at the end of the cycle, a developer is able to contribute software work products to the repository several times on any given day.
The main idea with CI is to reduce integration costs by having developers do it sooner and more frequently. In practice, a developer will often discover boundary conflicts between new and existing code at the time of integration. If it’s done early and often, the expectation is that such conflict resolutions will be easier and less costly to perform.
Of course, there are trade offs. This process change does not provide any additional quality assurances. Indeed, many organizations find that such an integration becomes more costly since they rely on manual procedures to ensure that new code doesn’t introduce new bugs and doesn’t break existing code. To reduce friction during integration tasks, continuous integration relies on test suites and an automated test execution. It’s important, however, to realize that automated testing is quite different from continuous testing.
The goal of CI is to refine integration into a simple, easily-repeatable everyday development task that will serve to reduce overall build costs and reveal defects early in the cycle. Success with CI will depend on changes to the culture of the development team so that there is incentive for frequent and iterative builds, and an eagerness to deal with bugs when they are found frequently.
Continuous Delivery (CD)
Continuous delivery is actually an extension of CI, in which the software delivery process is automated further to enable easily and confident deployments into production at any time.
A mature continuous delivery process exhibits a codebase that is always deployable. With CD, software release becomes a routine and no frills event without anxiety or urgency. Teams proceed with daily development tasks with the confidence that they can build a production-grade release ready to go at any time without elaborate orchestration or special late-game testing.
CD depends centrally on a deployment pipeline in which the team automates the testing and deployment processes. This pipeline is an automated system that executes a progressive set of test suites against the build. CD is highly automatable, and, in some cloud-computing environments, easily configurable.
In each segment in the pipeline, the build may fail a critical test and alert the team. Otherwise, it continues on to the next test suite, with successive test passes resulting in automatic promotion to the next segment in the pipeline. The last segment in the pipeline will deploy the build to a production-equivalent environment. This is a comprehensive activity since the build, the deployment, and the environment are all exercised and tested together. The result is a build that is deployable and verifiable in an actual production environment.
A solid exhibit of a modern CI/CD pipeline can be seen with AWS. Amazon is one of the cloud-computing providers that offers an impressive CI/CD pipeline environment where you can choose from among its many development resources and link them together in a pipeline that is readily configurable and easily monitored.
Many consider continuous delivery to be attractive primarily because it automates all of the steps from submitting code into the repository to releasing fully-tested, properly-functional builds that are ready for production. CD is an elaborate automation of the build and testing processes, but decisions about when, how, and what should be released remain a manual process. Continuous deployment can improve and automate those activities.
Continuous deployment extends continuous delivery so that the software build will automatically deploy if it passes all tests. In such a process, there is no need for a person to decide when and what goes into production. The last step in a CI/CD system will automatically deploy whatever build components/packages successfully exit the delivery pipeline. Such automatic deployments can be configured to quickly distribute components, features, and fixes to customers, and provide clarity on precisely what has been pushed to production.
Organizations that employ continuous deployment will benefit immensely from the ability for users to give quick feedback on new deployments. Features are quickly delivered to users, so any defects that become evident should be handled promptly. Quick user response on unhelpful or misunderstood features will help the team refocus and avoid devoting more effort into functional areas that are unlikely to produce a good return on that investment.
With the move to DevOps, there has also been a rise of new automation tools to help with the CI/CD pipeline. These automation tools typically integrate with other various popular developer tools including code repository systems like GitHub and bug tracking systems like Jira. As SaaS has become a more popular delivery model, many of these tools are running in the cloud, the same place where many modern developers (including mabl!) are running their apps.
The most popular automation tool is Jenkins (formerly Hudson), which is an open source project supported by hundreds of contributors as well as a commercial company, Cloudbees. Cloudbees even offers several different Jenkins training programs and product add-ons.
Besides open source, there are also several modern commercial software products available including CircleCI, Codeship, and Shippable. These products have several different advantages and disadvantages when compared to one another. In order to really understand them, I’d encourage trying each of them specifically within your developer workflow to see how they work in your environment (how they work with your tools, your cloud platform, your container system, etc.).
At mabl, we’re building on Google Cloud Platform and so we were looking for a product that was compatible and preferably integrated with GCP. We took a look at CircleCI, Codeship, and Shippable to compare their strengths and weaknesses:
We ultimately landed on Codeship and couldn’t be happier with our decision and the support we get from the Codeship team.
Once you’ve deployed a modern CI/CD pipeline, you’ll likely realize that your existing tools and processes in your developer workflow will also need to be modernized. One area that you’ll likely very quickly realize will need a lot of attention is testing. If your deployment frequency is daily or several times a day, your current testing practices, whether they take hours to run or are set to run overnight, won’t keep up. This is a problem that mabl is aimed at solving. Learn more with a free trial.
Download the whitepaper on Testing in the Development Phase of a CI/CD Pipeline to get more in-depth information on different types of testing used in the development phase of a CI/CD pipeline including:
- Unit testing in the development phase of the CI/CD pipeline
- Using mocks for testing in the development phase of the CI/CD pipeline
- Local UI testing in the development phase of the CI/CD pipeline