How to move from CI to CD with Jenkins Workflow
I learned the hard way building PaaS and SaaS frameworks, and perhaps there is no easy way, but here are some tips on how you can move beyond CI to CD by taking advantage of Jenkins Workflow.
Continuous Integration (CI) is the practice of merging all developer working copies with a shared mainline several times a day. This helps developers to get faster feedback when their code broke something, and it would look something like this.
Meanwhile, Continuous Delivery (CD) helps all teams involved in software delivery keep producing valuable software in short cycles, and makes sure that the software can be reliably released at any time.
This gives developers feedback when their code breaks one of the integration tests. What makes it fundamentally different from CI is that rather than just validating code, it prepares the application for deployment in the real world – and that means addressing the operating environment, and even how user behavior might affect the application.
There’s some misconceptions out there – so it’s important to stress that the decision whether and when to deploy production is not automated, although a fully automated process could be triggered manually if desired.
Continuous Deployment is a DevOps practice where artifacts are deployed automatically to various stages of a build pipeline. When all stages are performing satisfactorily, the software is finally pushed into production. This will be right for some teams and not so right for others. If an automated test on a production-like environment passes, the automated approvals for generated artifacts would ultimately lead to the software being deployed to production without human intervention. It’s just like Continuous Delivery, but automated all the way.
So what’s the added value of using CD over CI?
CI provided a framework for dev teams to divide and conquer, but you still have “big bang” software releases. This means that, as a developer, the features you are working on may not be implemented for some time so they can’t deliver value to the user. Likewise, you don’t get timely user feedback because the cycle time is “drawn out”.
By extending CI into CD, new features can be introduced faster to market. This is achieved by making much smaller incremental changes to software, and these incremental changes with measured feedback – it also makes it easy to change or revert when things don’t go as planned!
All tests are repeated every time before something is propagated to the next stage, which results in higher overall quality. Those quality gates get inserted at various pipeline stages to ensure that non-functional (and business) requirements are also met. CD applies the same software principles as CI, and provides traceability for artifacts (who did what when?) from source code to the delivery of software.
Enough evangelising about it – how’s it done? Let’s start with a typical CI pipeline for Java using Maven. A typical pipeline would look something like this:
- maven compile – compile the source code
- maven test – do unit tests
- maven package – package compiled code into a jar file
- maven install – add artifacts to your local repository
- maven deploy – add artifacts to a remote repository
- (That’s probably all done in one build job)
- upload WAR file to test system
- execute tests
- tests are green – DONE!
Are we really done?
At this point, we have an application that can be validated and tested. In essence we do mvn deploy and some kind of upload of a war file to an application container for testing purposes. This usually will be taken care of by two Jenkins jobs.
However, before any software can be released we need to consider what it will take to deploy it, the environment, and how it will be used:
- What about variations of the runtime environment? (Linux/Windows, Tomcat/Jetty/WebSphere, MySQL/DB/2/ Microsoft SQL Server)
- What about load tests? “We do them manually before every release”
- What about performance regressions? “We click-test before every release”
- What about scalability? “It was fine for the last release”
If I define a CD pipeline, the main thing I’m trying to do is to increase confidence from left to right. Is your deployment target a clean install, an upgraded system, which patch versions etc.? The CD mantra is “automate everything” so we need to build in tests, and also consider strategies such as Docker containerization for dealing with changes to the target environment.
Databases can introduce complexity – what if you are deploying to several databases, rather than a single database. How do you avoid issues?
You need to build test cases that are relevant to your application. I won’t delve too deeply into automated testing, but needless to say that considering each of the points above will increase the confidence you have that the software is fit for the operating environment.
How can we improve and start doing CD?
The good news is that creating a CD pipeline with Jenkins DSL is not painful! With just a couple of lines you can start to automate the CD process:
Convert the two CI jobs into one Workflow job
show workflow definition #1 snippet
Move the tests to Docker-Containers, so we can have the basic integration tests in parallel
show workflow definition #2
When those tests are green, use additional load tests as a quality gate
show workflow definition #3
Stages in the workflow job
- integration tests
- quality gateway load tests with a typical environment
- quality gateway load tests with all required environments
- when all quality gates are passed, push changes into production
show workflow definition #4
What’s next? Why not take a clone of your CI pipeline and give it a try? If you need to talk to operations, then it’s all the better – the closer you work with the Ops team, the sooner you can start making the mistakes from which you build your CD pipeline!