“We conflate productivity with value too often”: Finding the right metrics for DevOps
How do you choose the most important metrics? Co-founder and Principal Consultant at Lagom Solutions, Julia Wester spoke at DevOpsCon 2018 about “Finding metrics that matter and using them safely”. In this interview, we discuss with her the importance of proper metrics when utilizing DevOps.
JAXenter: Which metrics are the most important ones for DevOps?
Julia Wester: There are definitely some useful baseline metrics; you have to determine the best metrics for your context. With that disclaimer, below are some hot metrics for organizations adopting DevOps principles. Just remember to choose one or more from each section (or choose another you prefer) and look at them on a single dashboard to catch when you’re over optimizing one metric and its causing pain somewhere else.
- Productivity metrics:
- Deployment Frequency (for extra credit, calculate value per deploy – that’s the important thing after all)
- Responsiveness metrics:
- Cycle Time
- Quality metrics:
- Incidents per service or application (for extra credit, calculate cost per incident – not all problems are equal in their impact)
- Customer Satisfaction/Engagement (NPS, post touchpoint surveys, usage metrics, etc.)
- Sustainability metrics:
- Employee Satisfaction (I like giving an NPS question here… would they recommend your organization as a place to work for their family and friends?)
- PTO usage (are people taking enough vacation time, are sick days trending up due to burnout – be careful with this one – it can send the wrong signals very easily if used incorrectly)
If you want to improve your systems, which is the 3rd way of DevOps, then you will want to measure Flow Efficiency. Flow Efficiency is the ratio of active working time to total duration of a piece of work. Most of the life of a work item is spent waiting. Any improvement in that will have great effect on how fast you can finish work.
JAXenter: What is the first approach for using metrics for business decisions?
Julia Wester: Choosing the right metrics with ODIM (the acronym was coined by Larry Maccherone.)
- You start by defining your desired Outcomes.
- Then determine which Decisions you need to make to achieve those outcomes.
- Next, ask yourself what Insights you need to make that decision.
- Finally, determine what Metrics can help get you those insights.
When you do that for each decision, you have the metrics inventory that’s right for your context.
JAXenter: How do you start taking a look behind the metrics? Is one starting point, say a specific dataset, better than another?
Julia Wester: It seems pretty clear to me, from listening to talks and conversations at conferences, that DevOps practitioners realize that we have to expand the understanding, and thus improvement, of the value stream beyond just the dev/ops handoff but the seminal DevOps books still only talk about deployment metrics. So, you can start with your data on “how long does it take to deploy a line of code.” But, move quickly to wider metrics that can help you optimize a wider piece of the system. At some point you’ll end up with a very optimized dev/ops handoff but the first half of the pipeline is still completely broken. If work can’t get to dev/ops, our improvements don’t matter.
JAXenter: Which metrics should definitely not be included in business decisions?
Julia Wester: I don’t have a problem with individual metrics, but I do have a problem with how we use many of them. There are metrics that tell us how much of something we do: lines of code, # of issues closed, # of deployments per day. We get very excited because we think “gosh, we’re so productive!” But we conflate productivity with value too often. They aren’t the same thing. We could deploy lots of bad code all day long – the metric will look awesome, but the outcome will be negative. This is why we work from outcomes to metrics to help us avoid this issue. In essence, I’d say that the biggest pitfall to watch out for is to misunderstand what your metric really tells you.
JAXenter: What do you hope people took from your session “Finding metrics that matter and using them safely” when you spoke at DevOpsCon 2018?
Julia Wester: Start by ensuring you can map your metrics back to outcomes if you already have metrics in place and discard any that don’t.
Measure competing metrics, such as productivity vs quality, in order to be aware of the impacts of over-optimizing for any one factor.
Look to see if the metrics you are using have generated unintended consequences. Thinking about consequences of actions makes you a systems thinker!
Know what you’re measuring – we can be easily fooled that being more prolific with our deployments creates more value, but that can easily be wrong. We also think that measuring individuals can create good team behaviors. Another myth. If we do nothing else, we should earnestly discuss what our metrics really tell us.
Thank you for the interview!