Beyond DORA and SPACE: Exploring Diverse Engineering Metrics Frameworks

Challa Pradyumna

Challa Pradyumna

— 3 min read
Beyond DORA and SPACE: Exploring Diverse Engineering Metrics Frameworks

Recently, engineering metrics, developer productivity, and developer experience have gained significant attention in leadership circles. But large organizations have started to measure the effectiveness of their engineering organizations and systems for quite a while.

DORA:

Although a lot of leaders in the industry had their own signals and metrics to understand how their organizations were working. The first commonly accepted framework is the DORA (DevOps Research and Assessment) metrics framework. This was formulated by the DORA group. They wanted to answer one simple question:

What makes the best-performing software engineering teams work?

They have conducted extensive research and published their first State of DevOps report in 2014. Which has now become a tradition for the last 10 years.

The framework focuses on 4 metrics:

  • Deployment frequency
  • Change lead time
  • Change failure rate
  • Mean time to recovery

Although these metrics gave a good understanding of how fast an organization is building systems and how stable these systems are. It fails to consider how well the people building the software are doing. Also, any individual contributor's job is more than just writing code; it's about writing documentation, designing architecture, participating in on-call, etc. Enter the SPACE framework.

( Google is a strong advocate of the DORA framework. )

SPACE:

Fast forward a few years, researchers from Github, Microsoft, and the University of Victoria came out with a new framework—SPACE. This framework fills in the gaps left by DORA. Instead of prescribing metrics, SPACE focuses on themes/areas that need to be monitored and gives organizations the opportunity to define their own metrics that would fall into each of the categories.

  • Satisfaction and well-being
  • Performance (of ICs)
  • Activity
  • Communication and Collaboration
  • Efficiency and Flow

The paper goes in-depth into how to decide metrics for each category.

Most people would have heard or read about DORA and SPACE, but there are more frameworks that are formulated.

DPH:

Similar to the SPACE framework, LinkedIn has the Developer Productivity and Happiness Framework. As the name suggests, the framework focuses on striking a good balance between developer productivity and happiness. The document goes in depth into how to think about each aspect and formulate your own version of the DPH framework, including the kind of metrics that have to be selected as potential candidates for your version.

GSM:

Again from the house of Google, this is a different framework from the book "Software Engineering at Google."

It introduces a very simple system of Goals/Signals/Metrics. Goals are the desired end result we need; signals tell us if we have reached that end goal; metrics are a measurable proxy for the signals. Although this framework can be used for an organization of any type, the book also talks about the QUANTS framework to select the goals, which is specific to software organizations.

  • Quality of code (Qu)
  • Attention from engineers (A)
  • Intellectual complexity (n)
  • Tempo and velocity (T)
  • Satisfaction (S)

DevEx:

DevEx is a relatively new entry into the field. Taking into account all of the research done till now, they have divided productivity into 3 core dimensions.

  • Flow State
  • Feedback Loops
  • Cognitive Load

Flow state is where an IC is able to work without distractions. Feedback loops help ICs to quickly understand if the work they have done is good to be pushed to production. Finally, cognitive load is the amount of things ICs need to remember to get their day-to-day tasks done. Similar to the SPACE framework, DevEx gives the opportunity to organizations to decide on which metrics would make sense for the core dimensions.

For most organizations, sticking to one framework might not be possible because of the scale of the org or the kind of systems they work on or a myriad of different reasons. Understanding these frameworks will provide you as a leader with a solid foundation for designing a metrics program for your organization.

Overwhelmed by the amount of work required to operationalize these frameworks for your organization? Fear not; that's why we are building EvolveDev to get you up and running in minutes instead of months.

Read more