What are DevOps Key Performance Indicators

DevOps is about culture, enabled by the implementation of best practices and appropriate tools. This approach aims to improve the design of software products and services, using constant feedback and continuous monitoring. However, to ensure the efficiency of this approach, it is better to know which metrics to evaluate.

MTT-R Mean Time to R-

This one is a bit tricky because the MTTR can represent 4 different measures since the R can mean “repair”, “recovery”, “respond” or “resolve” and even though these measures may seem similar, each one has its own meaning.
Consequently, it’ s important to have fully integrated the communication principle promoted by DevOps and to clarify which MTTR we are talking about. Before starting to measure your MTTR, your teams must agree on what is going to be tracked and make sure that everyone is talking about the same thing.

Mean Time to Repair

MTTR (mean time to repair) is the average time required to repair a system. MTTR is a critical measure of the maintainability of systems, whether they are applications or infrastructures. MTTR is the value used to assess the efficiency of the recovery of these systems when an IT incident occurs. The MTTR is particularly valuable since nearly 90% of the repair MTTR is dedicated to problem identification.
Calculating the Mean Time To Repair:
The MTTR is calculated by adding the total time spent on repairs during a given period of time, then dividing that time by the number of repairs.

Mean time to recovery

In DevOps metrics, MTTR (mean time to recovery) is the average time to recover from a product or system failure. It encompasses the whole length of the failure; from the moment the system or product fails to the precise moment when it is fully operational again.

Calculation of Mean Time to Recovery
Mean Time to Recovery is calculated by adding up all the downtime during a specific period and dividing it by the number of incidents. It is usually measured in hours and may refer to business hours, not clock hours.

Mean time to Resolve

MTTR (mean time to resolve) is the average time between the detection of an incident and the moment the affected system or component is available to users again.  This includes the time it takes to detect the failure, to diagnose the problem and to repair it, as well as the time spent to ensure that the failure does not reoccur. This indicator is closely tied to customer satisfaction.

Calculating the Mean Time To Resolution:
To calculate this MTTR, you add up the total time to resolution during a given period and divide it by the number of incidents.

Mean time to respond

MTTR (mean time to respond) is the average time it takes to recover from a product or system failure starting when you are first alerted to the failure. This time does not include the latency of your alert system. Mean Time to Respond is useful in cybersecurity, combined with MTTD (Mean Time to Detect).

Calculation of the average response time :
To calculate this mean response time, add up the entire response time from the alert to the time the service or product is fully functional again. Then divide by the number of incidents.

MTTD – Mean Time to Detect

MTTD is the average time it takes for a team to identify an issue. This is a very critical metric to monitor in cybersecurity.

Calculating Mean Time to Detect:
To find the MTTD, just add up all the incident detection times for the given team member or period and divide by the number of incidents.

Defect escape rate

Clearly, in a perfect world, all bugs are found during the testing phase on the development server. However, in real life, and because DevOps is about delivering code quickly, it is important to track this metric to minimize the number and severity of defects that make it to production.

The percentage of critical and escaped defects is an important KPI, it ensures that the team and their testing efforts are focused on rectifying critical product issues and defects, which helps them ensure the quality of the overall testing process as well as the product.

Reliability and maintenance metrics

MTBF – Mean time between failures

MTBF (mean time between failures) is the mean time between failures of a system. This measure is used to monitor both the availability and reliability of a product. MTBF is a key maintenance metric for assessing the performance and security of critical or complex assets. It is one of the few KPIs that must be increased: the higher the time between failures, the more reliable the system is.

Calculation of the mean time between failures

This is the sum of the hours of Uptime in a period divided by the number of failures that occurred in that same period. MTBF is usually measured in hours, but if you are lucky enough to be able to measure it in days: Congratulations

MTTF – Mean Time to Failure

This metric is often mistaken and misused interchangeably with mean time between failures (MTBF). The main difference between MTBF and MTTF is that MTTF refers to non-repairable equipment, while MTBF applies to repairable systems. Software can be repaired and will experience multiple failures over its lifetime, and thus will have periods of time between failures, whereas non-repairable items, such as SSDs, will function properly for a period of time before permanently failing, and thus will only have one failure over its lifetime.

How to calculate MTTF:
To calculate MTTF, divide the total number of hours of operation by the total number of resources used. The MTTF represents the mean time to failure.

Automated test (in percentage)

DevOps relies heavily on automation, so tracking the performance of your automated tests is extremely important. This metric counts the number of automated tests performed for a given build. However, this metric does not consider redundancy, efficiency or variability of tests. This metric goes beyond simply counting the number of automated tests and helps clearly define business risks. With this metric, they can focus their test automation efforts on the tests that matter most to the business.

Deployment metrics

Lead Time for changes

This is an important metric in the DevOps model as it helps assess the efficiency of processes. The DevOps philosophy implies deploying “small batches” of code frequently, which is why it is so important to evaluate the time needed to implement, test and deliver the code. To do this, two important pieces of data are needed: when the commit occurred and when the code was deployed.

Fréquence de déploiement

Il s’agit du nombre de déploiements de logiciels sur une période donnée. Attention, cette mesure concerne les performances techniques du pipeline de déploiement, et non la fréquence de livraison, en effet tous les déploiements ne sont pas poussés en production. Il peut être mesuré de différentes manières, notamment par des pipelines de déploiement automatisés, des appels API et des scripts manuels.

Deployment stability

Any deployment that may cause problems or failures for your users is considered a failure. Deployment stability is the percentage of time that the most recent repository for a given repository has been successful. Of course, DevOps teams are supposed to build quality into the product from the beginning of the project, but failures and mistakes can happen. Not all broken builds are due to code errors; it may be an infrastructure issue that needs to be resolved. It becomes a problem when these errors persist for a long period of time and start to affect developer productivity.

This number should be as low as possible: 0 being the magic number, less than 5% of deployment failures is considered acceptable.

Service metrics

Tickets volume

Ticket volume or total number of tickets tracks all tickets in your support queue over a period. Bugs and errors can sometimes bypass the testing phase and be detected by the end user, resulting in a new ticket being opened. The number of customer tickets reported as “problems” or bugs is a major indicator of the reliability of the application. A large number of tickets indicates quality problems, while a small number indicates the robustness of the application. In other words, this number is a measure of end-user satisfaction.
A variation of this metric is the total number of conversations, which counts all exchanges with customers, whether through an official support ticket, social networks, or any other channel.

Application and traffic usage

After a deployment, it’ s a good idea to check the level of application usage or whether the number of transactions or users accessing the system seems normal. If all of a sudden there is no traffic or a traffic spike (if you are using microservices), there is a problem.

The Devops origins – A revolutionary Road

DevOps is a set of practices that brings together Development (Dev) and IT Operations (Ops). It is essentially a culture that comes from the fusion of management principles, traditional software development, infrastructure management processes and tools that increase an organization’s ability to deliver applications and services at an intense pace.
Getting all IT departments to work together is a concept that may seem obvious, like isn’t it what they should do? But the reality is more complex for a very simple reason: traditionally, Devs and Ops each operate in silos and have diametrically opposed roles and objectives. DevOps was created to reconcile these two opposing camps. To fully understand it, we must deconstruct this tragedy in 3 acts, leading to the Dev/Ops rupture.

Systemic issues

To understand the very notion of DevOps, we need to travel back in time. Traditionally, IT departments were organized on a functional basis, organizing teams according to their areas of expertise – database administrators, network administrators, server administrators, developers, and so on. Each in different groups, and to add to the difficulty, each with diametrically opposed goals…

Devs

Developers focus on meeting business needs for new features and functionality and getting them into production as quickly as possible.

Ops

IT operations have the mission to provide customers with a stable, reliable and secure IT service, and therefore strive to avoid implementing changes that could jeopardize production.

And here comes the drama

Here begins the classic drama, a 3-act structure.

The Set-Up

The scene started in Ops, busy maintaining applications and infrastructure to ensure that the organization continues to deliver value to their customers. The grind is to deal with issues due to application complexity, technical debt and daily hacks. Of course, the pledge is made to fix the problem later, when the workload eases…

The confrontation

Obviously, the most vulnerable artifacts, being the most subject to heavy and ever-urgent modifications, are also the ones that generally support the most critical projects or the most revenue-producing systems. And therein lies the tragedy. A bug, a system down, delays, a security problem. And while the Ops teams are busy solving the problem, a Product Manager or a CEO or a Business Manager comes up with a brilliant idea for a new feature that will surely solve all the pre-existing problems (no matter how difficult or technically challenging). Once again, the Devs enter the scene and have to come up with a quick solution to create this new feature, meet the technical challenges and sometimes take shortcuts to meet the deadlines. And the cycle goes on…

Resolution

No happy ending here, the cycle ends up leading to extended delivery cycles, less reliable deployment results, failures, greater resistance from Ops to any change, internal and external feedback processing slows down… Finally the whole company loses its competitivity and the IT teams who feel stuck in a system that predisposes to failure in which they feel powerless and unrecognized Ask any system administrator what they think about the joys of Friday night deployment, which forces all teams to work all weekend.

The Follow Up – Calms down

Over the past decade, technological progress and digital transformation have impacted all businesses. Digital transformation is driven by customer, partner and supplier interactions, which are increasingly moving towards digital. Nowadays, « every company is a tech company » and applications and infrastructure have become key strategic differentiators for companies. This means that the stakes and pressure facing IT teams are huge.
And here comes DevOps! The idea appeared for the first time in 2008 from the meeting of Andrew Shafer (software developer) and Patrick Debois (sysadmin) at a session that became famous at the Toronto Agile conference: Birds of Feathers. Gone are the days of “siloed” development and operations teams, often merged into a single team where engineers work together on the life cycle (build – test – release). Over the years, frameworks have also appeared that bring together the basic principles of DevOps: CALMS, The Three Ways, Accelerate.

To Sum Up

DevOps is in itself a culture shift: small teams working on small chunks of work from beginning to end while focusing on shared goals, which always take priority over individual or group goals. Technically speaking, the DevOps model aims to normalize and standardize development environments, automating processes (testing and automation) and eventually providing a wide range of self-service functionalities. From a human perspective, DevOps allows for frequent and rapid feedback between operations and development, identifying operational issues earlier, thus allowing for more efficient work without wasting time. Most importantly, DevOps provides teams with psychological safety, enabling innovation. DevOps requires a shift from a hero culture to a learning culture, and a culture of collaboration and transparency. The results of DevOps adoption are indisputable. Organizations that practice DevOps deploy their code thirty times more often, the time it takes to go from “validated code” to “successful production release” is eight thousand times faster.
The highest performing organizations have timelines measured in minutes or hours, while the lowest performers have timelines measured in weeks, months, or even quarters. Not only do organizations get more work done, they get much better results: when top-performing organizations deploy changes and code, they are twice as likely to complete them successfully (i.e., without causing production downtime or service degradation), and when the change fails and results in an incident, the time to resolve the incident is twelve times faster.
The biggest companies have understood this, such as Facebook, Google or Netflix…

Rework

In software development, Rework is a well-known source of struggle and waste caused by quality issues that lead to defects and code rework.
The DevOps approach focuses on collaboration and quality. Implementing fast and accurate feedback through effective CI (Continuous Improvement) with automated build-acceptance testing – combining different types of tests, unit, functional, integration and end-to-end – will reduce the number of escaped defects. Code quality and service uptime are metrics that the entire organization should be concerned about, and no team should be rewarded for finishing early if the quality is poor.

Motion

When it comes to wasteful handoffs, the problem comes from too many dependencies between individuals-for example, to approve a task or accept changes. To solve this problem, DevOps teams should eliminate silos as much as possible.
Teams can use cross-functional team grouping to solve this problem. For example, developers can pair up with testers to approve items at runtime rather than at runtime. Continuous integration (CI) is another way to solve the problem of wasteful handoffs: implementing a single CI process to cover the various aspects of validation should serve as a vehicle for delivering software across different team members.

Overengineering

This is probably the easiest type of waste to solve and the most underestimated. This type of waste is spending too much effort on things that are not really essential. Examples include investing time and resources in things that are not critical to the functionality or system, or planning as a team onboarding procedure to relearn legacy and undocumented code. To avoid this, focus on the key features that add the necessary value through clear user stories.
A closer link between product management and development allows teams to know what is needed and when to stop. Instead of product managers or business analysts writing 150-page specifications and throwing them over the wall to developers, it makes more sense for these teams to draw on a deep understanding of the business domain to develop initial user stories that developers can start working with instead of waiting for a perfect specification to be completed

Inventory

In a factory, an inventory consists of all work-in-progress (WIP) and finished goods waiting for customer orders on the shelves [Push]. In DevOps, inventory consists of all partially completed work. An example of WIP inventory is the backlog of open tickets. This work in progress may be pending, waiting for information, or simply in the queue. The waste associated with inventory is the restart costs associated with reworking a partially completed job, the time it takes to manage it, and the delays in discovering quality issues.

Waiting

This one is easy to spot: the wait for an answer to a specific question about a product from customers, or for ready and up-to-date environments. The DevOps model helps with this, because it’s all about efficiency. Teams can share the same ticket system so that a development team can immediately start working on a defect as soon as it has been logged by the Technical Support team. This approach also focuses on empowering and empowering teams to take action on tasks that need it instead of waiting for someone. For example, environment upgrades or setting up infrastructure as code (IaC) to support the creation of DevOps environments. Unlike setting up a traditional environment that includes configuring the network, virtual machines, etc., IaC automatically generates the required environment on demand.

Task Switching

This is the contextual switching of tasks that slows down the entire software development and delivery process and results in mental exhaustion from the (repetitive) actions inflicted by the process. This includes changing tasks or responsibilities or managing remote or inaccessible resources. The answer to better processes lies in managing handoff and inventory waste, as well as providing an efficient environment and tools, which will help teams better collaborate and communicate…

Overproduction

Overproduction is when more product gets made than can be sold, such as when a company exceeds customer orders or continues to make a product even though sales are declining. In software development, the waste associated with overproduction is the result of developing features and solutions that will never be used.
To prevent this, software development should be focused exclusively on the right user stories to develop, clearly linked to epics, initiatives, themes, etc. Appropriate and frequent reviews within the feature team identify variances from the defined scope and help team members stay on track for delivery.
In Lean and DevOps, the ultimate goal is to deliver what the customer needs, when they need it. If the test team is busy writing useless scripts simply because they are bored waiting for code to test, that’s a sign you have a problem. There is probably a bottleneck or constraint that is blocking the flow and forcing you to overproduce in one area to compensate. In another example, consider provisioning application environments and deliberately requesting more capacity than needed, just to avoid going back and requesting more later. In a cloud environment, this type of over-provisioning is not necessary. Instead of producing more capacity than you need, you can acquire and pay for exactly what you need.

Summary

DevOps is more than a technology, it is about a culture and a mindset. There is no one size fits all solution. It’ s up to the organization to identify its own slowdowns. Understanding and naming the different wastes within a process can help you apply the right kind of (continuous) improvement. Improvement can be in people, process, technology or a combination of all three. The cloud itself will not streamline an organization if corresponding process (and culture) changes do not accompany it. You need to learn to “see the system” and recognize where the constraints and waste are and address them.