Interview series with DevOps influencers — Final part

Key DevOps metrics that matter: How well does your team sleep?

Gabriela Motroc

© Shutterstock /Kom_Pornnarong

How can you tell if a DevOps initiative is succeeding? Is it the number of tools that you’re using? How about lead time, deployment time, customer satisfaction, and performance? Is sleep part of the equation? In the last part of our interview series, nine DevOps influencers talk about the key metrics that matter and explain why you shouldn’t skip steps in the DevOps transformation cycle.

Metrics are a key tenant of any DevOps transformation

Successfully implementing DevOps practices can have a profound impact on a company. However, there are a lot of implications when going down this road, including tools, metrics and the amount of sleep your team is getting.

Are you measuring the right things? Are you measuring too little (or too late)? Are you using the wrong tools? Are you using too many tools? In the last part of this interview series, we talked with nine DevOps influencers about the importance of not skipping steps in the DevOps transformation cycle and the key metrics that matter. Our DevOps heroes also revealed whether the abundance of DevOps tools has helped or slowed down DevOps adoption.



9 answers: Some companies are still struggling with DevOps metrics. What are the key metrics that matter and how can they enhance DevOps success?

DevOps Influencers

Charity Majors is engineer/CEO at Honeycomb.

Mike D. Kail is CTO at Cybric.

John Arundel s the author of several technical books and has worked with hundreds of clients as a consultant.

Gregory S. Bledsoe is a consultant with Accenture, writer, speaker and thought leader.

Jérôme Petazzoni is an international speaker. He previously worked at Docker, Inc.

Thorsten Heller is CEO and Co-Founder at Greenbird.

Eric Vanderburg leads the cybersecurity consulting division at TCDI.

Quentin Adam is CEO at Clever Cloud.

Hans Boef is a Developer Advocate at IBM.

Charity Majors: How happy your users are and how much sleep your engineers get.

Mike D. Kail: Every key initiative has three basic components: People, Process, and Technology. The latter two are the easy part, so the people part is what you want to spend a decent amount of time measuring. A couple of standard metrics that can be applied are LTV (is the employee adding value over time by increased contribution and engagement) and Churn (are you able to retain key talent). Process metrics will continually evolve, but the initial key metrics are time to build and deploy, time to rollback/remediate, number of deployments per day/week, and incident response time.

How many times were your people woken up by faults in production?

John Arundel: The most important metric that people don’t track is the number of out-of-hours and weekend pages generated by their monitoring system. In other words, how many times were your people woken up by faults in production? If you optimize for this metric, you’ll have a happy team and a highly reliable service.

Gregory S. Bledsoe: There is no “one true metric.”  This depends on the organizational context and finding it is a result of exploring an organization’s DNA, history, and context.  Many people find that “change velocity” is the best indicator of performance, but that certainly isn’t always true, and adopting a metric without due consideration can pull your focus away from things that might actually be more important for your particular organization.  

Jérôme Petazzoni: There are many metrics that you can use as a proxy to figure out if you’re “doing it right.” How long does it take for a new developer in the team to stage up their local development environment? How long until they can get their first commit into production? How long does it take for a fix to be deployed? To roll back an older version? To scale up? In all these operations, how many people have to be involved and why? (Are they involved because nobody else knows how to do it, or because they are the ones to assume responsibility?)

And of course, we can tap into more classic metrics like uptime, response time, user satisfaction … At the end of the day, it doesn’t matter how stable your service is, or how fast you deploy. What matters is the value you deliver to your customers or users and (if you’re a for-profit company) your revenue.

SEE ALSO: How well-defined metrics enhance DevOps success

Thorsten Heller: There are several key DevOps metrics or KPIs such as availability, response time, error rates, number of succeeding deployments, deployment frequency, etc.  But most of the typical used KPIs are technical ones. We see our clients asking for more business-related DevOps metrics such as “Time to Value” where several of the “technical metrics” are aggregated to a business KPI focusing on the value generation.

Eric Vanderburg: Firstly, it is important to define metrics that you determine are crucial to success. I can provide some examples, but the best ones are those that you can directly tie to business objectives and reliably measure. Some companies create metrics but measure them ineffectively, and the metrics do not accurately reflect reality.

Some important metrics to consider include the number of errors detected per volume of code, deployment time, build frequency, error remediation time, requirement discrepancy percentage, change volume, customer satisfaction, and performance.

Too many teams focus on tools and not on optimization.

Quentin Adam: There are two hard measurements that I think we should talk about first. And these are “How well does your team sleep?” (“Clever Cloud, selling sleeping hours since 2010”) and “Can you ship ideas to production before that idea becomes irrelevant?”

The final goal of every good DevOps has to be NoOps: the ability of the automation to manage the entire stack at minimal human cost. Too many teams focus on tools and not on optimization, that’s why simple business metric s are so important. Some call it serverless (ok, often to describe proprietary lock-in solutions), but the major point is to remove server management and not the server itself (I wrote about this here), and the final goal is to make software evolutions as easy as possible, to help business evolve. Maybe the ultimate metrics in a big company will be the organic reduction of shadow IT because working with internal IT becomes more and more comfortable.

Hans Boef: DevOps is also about continuous delivery and deploying code as fast as possible. By tracking DevOps metrics, you can evaluate just how fast you can move before you start breaking things. The most important metrics are lead time, deployment time and customer tickets.

SEE ALSO: Measuring DevOps: The key metrics that matter


How important is it to not skip steps in the DevOps transformation cycle? What are the steps that companies and/or teams usually ignore or underestimate?

Mike D. Kail: My view of DevOps transformation is that it is a continuous evolution/journey, not a destination that has a predetermined route. There is no “Waze for DevOps transformation”. There are plenty of poster children for DevOps transformation (Netflix, Etsy, etc…) and going back to a previous answer, companies tend to underestimate the challenges around people and removing cultural inertia.

There is no “Waze for DevOps transformation”.

John Arundel: Not steps, but people. Some people just fundamentally don’t get it, or they do get it, but they don’t like it and won’t adopt it. There is no way around these people. They got used to doing things the traditional way, and resent the ground shifting under them.

The most effective way to stop these people holding up the DevOps transformation is to replace them. Most companies are not willing to do this, and that is a major reason why most companies fail to transition to DevOps successfully.

Gregory S. Bledsoe: This question is very difficult to answer without knowing what the transformation methodology you are referring to actually is.  In the methodology I subscribe to and have successfully applied many times, every stage is critical, but these focus primarily on the human system and drive change from there to the technical system.  

There is a reason we say in DevOps that we go through 1. People 2. Process 3. Tools. You can think of the process as the intersection of culture and technology — the guide rails around its use, and this is where reform is first needed.  In general, organizations want to implement technical solutions instead of human solutions because human solutions are much more difficult to implement, and we aren’t generally attuned to how to manage the human system in this new way. Trying to skip to the end of implementing examples instead of principles is a sure route to underachievement.  

Jérôme Petazzoni: I’ve seen too many teams and organization try to go straight from “works on my machine” to “works in production.” Instead of taking that huge leap of faith, I often suggest to go through a number of intermediary steps, including but not limited to continuous integration, continuous deployment to staging and QA, etc.; each step helps to build skills and confidence that will be critical when going to production.

Thorsten Heller: Not quite sure whether there is a “the DevOps transformation cycle”. We meet regularly clients have some kind of or some parts of DevOps in their organization without knowing it or calling it DevOps. We meet regularly clients telling us they have DevOps and the only things they did is placing them in the same office.

SEE ALSO: Open source software tools of DevOps

Eric Vanderburg: Most of us learn early on that skipping steps is not a good idea.  Skipping steps typically results in an error such as skipping a step in solving a math problem, research project, or a first date.  You are far more likely to end up in a dangerous situation if you skip steps in DevOps transformation too. The steps are there because each phase builds on the one before it, receiving inputs from that stage, and refining the application.  The two phases I see most often skipped are the planning and testing phases. Applications are still built, deployed, and maintained in some way, but teams, in a rush, to get a product out the door, skip the planning stage and jump right in or they skip the testing phase and go directly from development to operations.  Skipping the planning stage results in missed requirements, more errors, rework poor coordination, poor project management, cost overruns, and security problems. Similarly, skipping the testing phase results in poorly performing software, missed requirements, bugs and flaws, security holes, and a very poor customer experience.

The two phases I see most often skipped are the planning and testing phases.

We do see game development somewhat combining prototyping with deployment to gain further enthusiasm for the product.  Unlike a beta test, game development firms release a game to the community at a reduced price, but the game lacks features or contains bugs that the community evaluates and tries.  New versions of the software are released to the community as the game works its way towards completion. Some might say that this skips a step. Instead, I would say that they are simply pursuing CI/CD.  Each build is released to the community to use, and it meets their customers’ expectations. The game may never be complete as the community continues to offer new ideas for improvement which takes us back to digital transformation, not a skipped step.  Skipping steps will only provide you with more headaches.

Quentin Adam: Since DevOps is, first and foremost, a cultural change you can’t really skip anything. If you try to go too fast, changes won’t stick and you’ll be back to your old ways. Don’t try to transform everything suddenly. Changing culture takes time and has to be gradual.

You need to put the accent on learning because DevOps relies on serious skills around networking, Linux, system… Learning is the key to success. Use tools you understand better than cookbooks and dark wizardry, hoping it will do the job.

Hans Boef: You need the full process to take advantage of DevOps, some steps in the process do not take much time but are essential. 

SEE ALSO: 6 must-have tools for DevOps success


Do you think the abundance of DevOps tools has helped or slowed down DevOps adoption?

Charity Majors: Tools are essential.  Discernment in selecting them, even more so.

Mike D. Kail: Much like with the cybersecurity sector, the overabundance of tools and point solutions tend to create (more) Fear, Uncertainty, and Doubt (FUD). A good analogy is purchasing a large piece of furniture from IKEA and then attempting to assemble it. The sheer number of parts and pieces is overwhelming and serves to add to the aforementioned cultural inertia.

John Arundel: There are no DevOps tools. Is a gun a crime tool or a law enforcement tool? It depends who’s holding it. The difference between doing DevOps and not doing DevOps is in the people, not the tools.

A DevOps architect is an architect of principles more than toolchains, and once you understand that you’ve made a quantum leap towards DevOps effectiveness.

Gregory S. Bledsoe: Honestly I don’t think this matters to success or failure very much.  There are many tools that can achieve the same ends, and it is a poor craftsman that blames his tools.  DevOps adoption is a function of the absorption of the underlying principles into the organization overall.  A DevOps architect is an architect of principles more than toolchains, and once you understand that you’ve made a quantum leap towards DevOps effectiveness.

Jérôme Petazzoni: Helped! Of course, sometimes it can slow things down a bit on a local level because people are waiting to see which particular tool is going to “win” or be better suited for their needs. And that’s perfectly fine! But I’m glad that today we have Ansible, Chef, Puppet, and Salt; that we have Kubernetes, Mesos, Nomad, and Swarm; because they all have different strengths and weaknesses and this is not a “one size fits all” situation.

Thorsten Heller: DevOps is more mindset, organization or processes than tools.

Eric Vanderburg: An abundance of tools adds complexity to the tool selection process.  However, with this abundance comes diversity and the opportunity to address niche needs of DevOps consumers.  It can be challenging to work through the marketing information and truly understand what differentiates DevOps tools so it is important for independent parties to evaluate and compare these tools so that DevOps consumers can make an informed, unbiased decision as to which tool is the best fit for their company, team, product, and customers.

Quentin Adam: Tools are accelerators. Which can be good or bad depending on the tool and the moment you are using it. If you are not ready culturally, it can lead to misunderstanding or failure. And people usually don’t like to try the same thing again after a failure.

Globally I would say it is a great thing but should not be seen as the only answer. Culture first, tooling second. They will, however, become more and more important as the DevOps culture matures, resulting in a bigger user base and so hopefully more enhancements, features, and empowerment for DevOps practitioners.

Hans Boef: It has definitely helped, but it can be hard to find the right tools for the organization.



Are your calendars marked for JAX DevOps 2018? If you’d like to know more about the latest trends in DevOps and meet the top movers and shakers in the global DevOps scene, join us in London between April 9-12, 2018.


Gabriela Motroc
Gabriela Motroc was editor of and JAX Magazine. Before working at Software & Support Media Group, she studied International Communication Management at the Hague University of Applied Sciences.

Inline Feedbacks
View all comments