DORA Metrics: Delivery vs. Security

When it comes to software development and delivery, developers can never be fast enough. Because of that, measuring performance is key to addressing the constant pressure to deliver at high speeds. 

Enter DORA metrics. The DORA core metrics consist of four key traits for evaluating DevOps teams’ performance and helping them achieve higher performance rates.

This article will look into the following:

• What DORA is
• Overview of four DORA traits
• Benefits and challenges of DORA metrics
• DORA metrics' impact on delivery vs. security

What Is DORA?

DORA, the DevOps Research and Assessment, was originally a startup that provided assessments and reports on companies’ DevOps efforts. DORA was acquired by Google in 2008.

After six years of research and 31,000 data points, DORA is one of the longest-running, academically rigorous research programs. Using behavioral science, DORA's studies were able to highlight ways to make software development and delivery more efficient. 

Through DORA metrics, DORA strives to help DevOps teams achieve high performance and improve capabilities.

» Find out how to define DORA-like metrics for security engineering

Overview of DORA Metrics

DORA metrics' core objectives are assessing and improving the performance of DevOps teams. The four key metrics are: 

1. Deployment frequency
2. Mean lead time for changes
3. Change failure rate
4. Time to recovery

The first two metrics, deployment frequency and mean lead time for changes, measure the speed aspect. The change failure rate and time to recovery evaluate the stability in DevOps.

Using the performance benchmarks for each metric and analyzing how teams score in these four variables, the teams can be categorized into:

• Elite
• High-performing
• Medium
• Low-performing

Let’s explore each of these metrics in more detail.

Jit

The only open DevSecOps orchestration platform

Get Started

Coverage

Full app and cloud security in minutes

---

Integrations

Seamless integration with tech stack workflows

---

Security monitoring

Measure security performance metrics per team

---

Orchestration framework

Open framework ensures simple migration to any app or cloud security tool

Brief overview

Jit is a continuous security platform that helps you seamlessly embed security tools and control into your workflows. You can manage your entire DevSecOps toolchain across your IDE, code, pipeline, cloud, and runtime.

Easily plug any tool into Jit’s extensible orchestration framework to unify the execution and interface of any security tool.

Key features

• Multiple security plans for customized coverage
• Unified execution and UX for all security tools
• Fast and automated scanning within GitHub
• Jit’s Context Engine determines whether a vulnerability is actually exploitable in production to prevent alert fatigue and long backlogs of irrelevant vulnerabilities
• Measured security performance of different teams
• Jit’s open framework ensures a simple migration to any app or cloud security tool

Pros and cons

Dev-friendly

Open source

Orchestrates and unifies all tools

Easy to find problematic code

Provides training in the form of documentation and live online

Platforms supported: SaaS, Windows, Mac, and Linux

The number of tool integrations could be overwhelming

Relies on additional open-source tools that must be maintained



4 Key Dora Metrics Explained

1. Deployment Frequency

Deployment frequency measures how long it takes for your organization to get a change in production, i.e., how often the code is successfully released to the end user or deployed to production.

It is an essential DORA metric that evaluates the average throughput and displays the cadence of delivering value to the client.

A higher deployment frequency also means shipping values to the customers more quickly. The benefit of this is in consistent feedback and higher customer retention.

DevOps elite performers strive for continuous delivery every day, whereas low-performing teams often score a deployment frequency of about once per month.



2. Mean Lead Time for Changes

The mean lead time for changes is the time it takes the DevOps team to get a committed code to a successfully running state within the production state.

This DORA metric helps DevOps teams determine how healthy their cycle time is and if the team can handle a higher volume of requests. For instance, average-performing teams take about one week to complete this process, whereas elite performers often do it in less than a day.

To identify and narrow the obstacles to improving your DevOps teams' lead time, analyze the different metrics in the development pipeline: time to merge, time to open, and time to first review.

3. Change Failure Rate

Bugs and failures are inevitable in a production environment with frequent changes. The change failure rate is a percentage value derived from the total number of deployments to the total number of failures ratio.

As a metric, the change failure rate shows the number of changes that count toward failures in production. The failures can be failed deployments, rollbacks, or incidents that require fixing.

The elite, high, and average performers score around 0 to 15% change failure rates. Low performers often have a change failure rate of about 40% to 65%.



Teams with more automation tools tend to have lower change failure rates since their development process is more established and consistent. The lack of automation, on the other hand, leads to a higher change failure rate and loads of small changes to implement.

4. Time to Recovery

Time to recovery is a critical metric that directly impacts DevOps stability. It is expressed as the time needed to restore the service when a defect, accidental outage, or service impairment occurs.

The mean time to recovery (MTTR) for elite teams is typically less than an hour, whereas low-performing teams can take from one week to one month.

By improving observability and monitoring in DevOps teams, the failures are identified and recovered faster.

How Dora Metrics May Help Your DevOps Team

Any company requires tangible goals and evidence to achieve stability and progress.

DORA is a starting point in determining your team's current performance.

The metrics provide insights and evidence of the team's performance measured against the industry standards. These, in turn, help the DevOps leads identify bottlenecks to achieve high performance and stability and ultimately improve decision-making.

Challenges of DORA Metrics

Each enterprise is different, and its metric goals, problems, and delivery environments vary. Despite their many benefits, each DORA metric is a mixed blessing and doesn't tell the DevOps story from a broader perspective.



To avoid being blindsided by using DORA for DevOps, it's good to consider the following:

• Contextualization: DORA scores need to be put into the context of the company's needs and objectives.
• Data collection and analysis: Collecting the raw data and transforming it into calculable units for DORA scores can be a painstaking process due to the sheer volume of sources.
• DevOps non-linear nature: To avoid false conclusions, turn DORA metrics into KPIs and shift focus to achieving high DORA metric scores.
• Comparing DORA scores for market performance: Measuring market success only through DORA numbers is often pointless—after all, for market performance, the customers and clients will have a final say.

Plus, there are other factors to be considered in DevOps. One such primary concern is security.

The Role of DORA Metrics: Delivery vs. Security 

DORA metrics work best for DevOps performance speed and stability. But if you focus only on them to evaluate the software development and delivery process, you may be compromising security.

The security world is ever-changing, and your security measures must adapt, too. Forget about year-long plans and instead focus on agility, software protection, getting the right tools at the right time, and adapting to the ever-shifting market.

DORA metrics don’t adapt to the times, and relying only on them will lead to more bugs and vulnerabilities in the application at later stages. These will, in turn, burden your team with unnecessary workloads, worsen the user experience, and potentially lead to developer burnout.

A big part of the solution is continuous security: ensuring that your code is protected from development to production.

Following DevSecOps practices, such as implementing shift-left security, will protect against a vicious cycle that harms an organization's productivity and puts software at risk.

» See our solutions for DevOps continuous security

DORA Metrics: Delivering Wins, Securing Doubts

DORA is a stepping stone to high performance and stability in DevOps.

Although DORA metrics improve software delivery, their impact on security can be neutral at best and damaging at worst. A robust DevSecOps practice is a must if you want to compensate for DORA metrics’ missing elements.

How about we simplify continuous security for you?

Jit orchestrates and unifies security tools and controls into all stages of your CI/CD pipeline. When a pull request is created, Jit runs the relevant security tools so your developers can act upon the detected issues using our guidelines or automated remediation.

‍

‍