Billy Beane, general manager of the Oakland A’s in 2002, lost his three best baseball players to free agency and had little cash to spend on other superstars. Instead, he focused on using a specific set of metrics — slugging percentage and on-base percentage — to build a team of undervalued, unheralded players, much to the bafflement of the baseball world.
The result: Beane led the A’s to the fifth-longest winning streak in baseball history and three American League West titles in five years.
For software development teams, metrics can similarly be used not only to understand how productive and efficient a team is at a variety of tasks, but also to predict future changes in that efficiency and address any roadblocks along the way. Rather than early-stage development, DevOps metrics usually focus on deployment, operations and support. To that end, the metrics themselves are frequently broken down into three specific categories: people (such as response time); process (lead time); and technology (uptime).
While some of the more common measures are known to many DevOps teams, other lesser-known metrics can also have a major influence. Much like Billy Beane focused on specific metrics to unlock new understandings of baseball productivity, we sought to find unique data points that do the same for software engineers. We sat down with Jason Vertrees of Overhaul to discuss which lesser-known metrics he uses to measure the productivity of his engineering team.
Jason Vertrees is senior vice president of technology at Overhaul, a supply chain integrity solutions company. While his engineering team uses a variety of metrics to measure productivity, he has zeroed in on two useful measures in particular that have unlocked greater models of velocity and prediction at work.
What metrics do you use to measure the productivity of your engineering team, and why?
There are the standard measures many companies focus on, including us, like uptime, mean time to recovery, lead time, change failure rate and deployment frequency. Most know about those, so I’ll talk about something else.
Two that I’ve found useful outside the above are “team happiness” and “predictability rate.”
What actions came out of these insights?
“Team happiness,” while subjective, can be tossed onto a Likert-type scale giving you the opportunity to quantify it. In addition, happiness is a leading (not lagging) indicator: if your happiness goes down, your upcoming velocity is going to decrease as a result.
Predictability helps us measure the percentage of large projects delivered on time. It helps us hone our estimations, which are critical for external teams’ planning.
Happiness is a leading (not lagging) indicator: if your happiness goes down, your upcoming velocity is going to decrease as a result.”
What impact has tracking these metrics (and acting upon the data) had on your team’s productivity?
Intentional focus on happiness helped us uncover the sources of lost productivity.
I used this on one project and at one point we noticed a dip in the team’s happiness. Once the decreased scores were spotted, that spawned a line of questioning in the retrospective where two non-technical reasons surfaced: lack of clear product preparation/requirements and developers’ sense that management wasn’t actively listening. After knowing the cause, we were able to address the issues, restoring that team’s great output, happiness and trust in management.
