“If you can’t measure it, you can’t improve it” – Peter Drucker
Organizations that are adopting agile ways of working are finding the old ways of measuring don’t provide the information they once did. So what are some metrics that actually matter and will provide a sense of how you are executing toward goals?
The first word of caution is you will get the things you measure. If you are measuring velocity and talking about it within teams, they’re going to push to continuously increase it, even at the expense of the product quality. I look at these two measures as the gas (velocity) and the brakes (quality). Both measures should be considered as the process is improved.
Based on the most recent State of Agile Report, the following are the most common methods for measuring success of agile projects:
The very first principle in agile is: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” This means evaluating the value being delivered, which may not mean financial value. For example, it could be valuable for the installation of servers to make the product perform better, thus creating a better customer experience. Each organization has to determine how it defines value.
As you’re gauging the success of your agile efforts, there’s no clear-cut way to define value. What are the key measures that organizations can consider, and most importantly what do agile leaders need to know about them?
- Quality is of the utmost importance in an agile organization, so it cannot be stressed enough that it needs to be visible and discussed regularly. The importance of quality is higher in an agile environment because every defect created is future work that is spawned, while trying to deliver production-ready code for every iteration. You can measure quality by the defect injection rate, defects per thousand lines of code (KLoC), or some other mechanism.
- Predictability is a great measure of the quality in planning processes. An easy trap to fall into however, is to simply say, “we planned 40 points, we delivered 40 points”, but this only indicates a known velocity. It is important to confirm the 40 points delivered are what were identified in the planning session, in order to have a high predictability. Of course, there are many legitimate reasons (change in business needs, technical impossibility, changing assumptions, learning in the sprint, etc.) not to deliver what was discussed in planning, which is why this is a measure that can be used to evaluate the planning process.
- Depending on the agile management method being used, Scrum or Kanban, there are different ways to measure to productivity. Scrum uses Velocity, or the number of points that can be delivered in a given sprint reliably and with quality. While Kanban, a flow-based system, uses two primary measures: (1) Lead Time and (2) Cycle Time. Lead Time is the measure from when a customer asks for a change until it is delivered back to them. Cycle Time is the measure from when the team starts working on something until they have delivered it to the customer. The objective in both cases is to keep the durations short to increase the value delivered to the customer.
- An additional metric that is recommended is the Release Burn-up. This measures the progress to delivery of a given release based on the points assigned. This can be shown in a graph format or as a % complete and helps measure progress to the next delivery.
It is very easy to get caught up in the tracking of metrics and measures and lose sight of the original goal, to deliver something of value to your customers. These indicators are not telling the whole story, but they go a long way to helping you understand where to dig deeper and start asking more questions. They can be used to help create a workable roadmap, evaluate progress to it, and deliver that message to sponsors, but the real measure of progress and success should always be the measurable outcome or progress to it.
These traditional measures and metrics are valuable as you begin your journey, but as you grow, you’ll find that they don’t necessarily demonstrate the outcome you are working towards. Be open to changing the metrics that make sense to your organization, don’t make it a ‘one size fits all’ solution. As you grow in your agile journey, it is possible that metrics like ‘Experiments Attempted’, ‘Experiments Completed’, ‘Experiments Failed’, become more important to reflect innovation and the willingness to fail and learn.