“The codebase is a minefield —there will be casualties”
Continuous Delivery is the antidote for stressful, scary and fragile releases. A proper CD pipeline gives us the certainty that most of the mines got detected and disarmed and even if anything was overlooked, we have a way to quickly revert the damage. We invited Anton Weiss, Principal Consultant and CEO at Otomato and DevOps evangelist, to talk about the pitfalls of not putting continuous delivery into practice and the advantages of having a proper CD pipeline.
Continuous delivery is no longer a theoretical concept, but that doesn’t mean that all companies put it into practice. We talked to Anton Weiss, Principal Consultant and CEO at Otomato and DevOps evangelist, about the perks of continuous delivery and the pitfalls of ignoring it.
JAXenter: Is Continuous Delivery the best way to deliver software?
Anton Weiss: We still don’t know if it’s the best way of delivering software. The IT industry is changing rapidly and new, improved techniques and methodologies are being invented and applied as we speak. Still CD is certainly the best way we currently know because it:
- Enhances agility
- Provides visibility
- Improves quality
- Reduces uncertainty, rework – and effectively – human burnout.
The important thing is to keep CD definition simple. Interestingly, some organizations I’ve worked with (especially enterprises working on non-SaaS solutions) had some hard time trying to describe what CD is. While it is in fact quite trivial: It means being continuously able to release software to customers without compromising quality.
I do realize this is simple to define but not that simple to implement because for that we should have the tooling and processes to answer the following questions:
- Who is our customer?
- How does our customer use our software?
- What is the last good build of our software?
- What is that last build’s content (features, bug fixes, 3rd party integrations, configuration changes)?
- What tests did it run through? What’s the test coverage rate?
- Can we perform a one-click deployment of that build?
- How do we measure quality and performance?
- If something goes wrong, how do we roll back the change with certainty? (this can also mean rolling forward or feature toggling. The point is to minimize MTTR – Mean Time To Restore)
If any of these questions can’t be answered, we’re bound to fail. If we don’t understand your customer , we’ll be testing the wrong flows and may not even realize our software is malfunctioning. If we don’t have defined quality gates, we may be delivering but we can’t release with certainty. If we don’t measure test coverage, our continuously delivered changes won’t get properly tested. If we don’t manage content, we may be able to release but we will have a hard time trying to undo the damage. And there will be damage.
A proper CD pipeline gives us the certainty that most of the mines got detected and disarmed.
Here’s a story of a client of ours: a small web startup in the field of sharing economy. They were lean and had a lot of the pieces of the puzzle in place – a CI server, a staging environment, some testing routines, a promotion process. But they didn’t have delivery content management and test coverage metrics. They would continuously deliver when they felt they were ready (according to test results or intuition) or when a bug fix was urgently needed.
But then things started to break and they couldn’t say with any amount of clarity:
- what change introduced the bug
- which version to go back to
- if there were any schema changes between the good and the bad version
After a couple of months of endless firefighting, exhausting night shifts trying to restore service and a couple of key developers giving up and resigning they started looking for a solution. That’s where we came in and helped them revise their CD pipeline and actually provided them with the value they were expecting.
JAXenter: What potential problems are we avoiding when we choose to give CD proper attention?
Anton Weiss: The biggest potential problem that Continuous Delivery helps prevent is the following: stressful, scary, fragile releases. In most organizations which don’t practice CD, a release requires codebase freeze and people are expected to work extra hours. All because everyone realizes that the codebase is a minefield and we now have to disarm all the mines. And there will be casualties.
On the other hand, a proper CD pipeline gives us the certainty that most of the mines got detected and disarmed and even if anything was overlooked, we’ll have a way to quickly revert the damage. All because we are delivering in small chunks and verifying every change. Continuous delivery turns releases into a non-issue.
Good engineers love automation.
The other big business problem that Agile originally came to solve is delivering something the customer doesn’t need. CD helps us verify our hypotheses the moment we are done instead of waiting weeks or months before releasing. Practices like A/B testing, blue/green deployments, canary releases and feature toggling are all enabled by continuous delivery and allow us to achieve perfect alignment to what both our customer and the market need.
Automation and measuring —a must
JAXenter: From your experience, how do people respond to the idea of automation?
Anton Weiss: Good engineers love automation because it allows them to deal with the interesting stuff. Managers love automation because it lets them be more efficient. We all love automation because it tames the machines and makes them dance. Everyone who has worked with computers long enough knows they were created to make humans suffer. Automation is our way to fight back.
So if everyone loves it, why isn’t everyone doing it? Because automation requires investment. It always pays off in the long run, but may consume significant resources in the short run. And we, humans, are are driven by fear of loss much more than by our desire of gain. We are afraid to lose time, waste effort, blow release deadlines and maybe get fired. So many times we prefer to stick to our old manual and ineffective routines instead of investing in automation. Needless to say this only widens the technical debt and makes everything ever more error-prone.
Continuous delivery turns releases into a non-issue.
JAXenter: How important is measuring when putting CD into practice?
Anton Weiss: Measuring is vital. If you’re not measuring, you don’t know if you’re improving or degrading, you don’t know what and how much you’re testing. Continuous Delivery can be easily hindered by countless issues. Like infrastructure hiccups, lack of communication, uncontrolled or missing resources. Good measuring allows us to identify these bottlenecks early and cater to them before they break our CD cycle and throw us back into manual, fear-driven software delivery practices.
There are three categories of measurements that you have to continuously build into your pipeline: those related to velocity, quality and performance. We’re currently working on a white paper expanding on the main metrics in each of the categories.
JAXenter: Is CD still a theoretical concept or are we beginning to see its benefits?
Anton Weiss: Long gone are the days when CD was a theoretical concept. Nowadays this is the way things are done. This is how the so-called unicorns deliver their software and many enterprise IT shops (e.g: AT&T, HPE) have already followed suite. Building CD takes time, knowledge and effort. It’s not something you can buy. I even gave a talk a few years back called “Continuous Delivery is Not a Comodity”. That’s again why many companies get to it as an afterthought. But even those who still don’t do CD already realize they should be.
Thank you very much!