The 2024 State of Testing in DevOps Report

Presented by mabl

Welcome to mabl's 5th State of Testing in DevOps Report, where we explore the full impact of software testing and test automation on DevOps maturity, organizational performance, and the user experience.

The 2024 State of Testing in DevOps Report is our most successful effort at this goal to date, surveying over 500 development and quality professionals in the United States. Roughly 40% of these respondents held a leadership role in software development and QA, while 60% were practitioners on the frontlines of development. 

The diversity of roles reflected in the latest Testing in DevOps Report allows us to understand how everyone in the software development organization - including end users - are impacted by testing. Though some trends were clearly positive - 89% of teams are prioritizing DevOps transformation, for example - others indicated plenty of room for improvement:

  • 80% of companies aren't delivering high-velocity software, defined as weekly or daily deployments
  • The biggest testing headache was test maintenance, with a massive 138% increase in teams ranking it as their top pain point
  • Testing tech stacks are becoming more complex, with 44% of fully DevOps teams using 5+ testing tools 
  • 60% of teams are using AI in some way, but the top use case (test summarization) doesn’t align with the most labor intensive testing tasks 

We hope that the trends uncovered by the 2024 Testing in DevOps Report help your teams benchmark your DevOps and testing practices, discover new ways to improve software quality, and gain a deeper appreciation for the importance of software testing . While many companies just look at the immediate effects of changing test automation tools and tactics on quality teams, mabl's State of Testing in DevOps Report looks at the entire organization to understand how testing can transform an entire company, ultimately becoming a key competitive advantage.

The DevOps Journey and Common Roadblocks


Understanding how companies are progressing in their journeys to DevOps transformation demands a roadmap. Though DevOps adoption and maturity will look different at every organization, we broke the road to DevOps into four stages: aspiring DevOps teams, striding towards DevOps teams, mostly DevOps teams, and fully DevOps teams. Illustration: the four phases of DevOps adoption: aspiring to DevOps, striding towards DevOps, mostly DevOps, and fully DevOps



Once an organization has reached full DevOps maturity, they have adopted CI/CD and automated most (or all) workflows in their development pipelines, created seamless processes for collaboration, learned to measure their performance with more effective metrics, and have established the foundation for continuous improvement.

Pie chart showing the proportion of companies in each stage of DevOps adoption

Despite emerging in 2009, DevOps is still a work in progress for most organizations: just 1-in-3 teams (33%) consider themselves mostly or fully DevOps, with an equal number making early strides on their DevOps journey.

The slow journey to full DevOps maturity, however, isn’t due to lack of desire from companies. When asked about their biggest DevOps obstacles, just 11% of respondents said that their primary challenge is a lack of prioritization from their organizations.

89% of respondents shared that DevOps is a priority for their organization

Instead, respondents shared that their DevOps transformations are more likely to be slowed by technology limitations and budget constraints. 2024 marked the first time since mabl began running the Testing in DevOps Report that technology limitations were the most common DevOps obstacle, narrowly beating budget concerns. This is a notable shift from mabl’s previous report, which found that 82% of teams were slowed by non-technological challenges. 

Bar chart showing the most common challenges to DevOps transformation

The rise in technological challenges can largely be attributed to mature DevOps teams in the “mostly DevOps” or “fully DevOps” stages of transformation. 40% of fully DevOps teams say that technology limits are hindering their DevOps practices, compared to just 20% of fully DevOps teams who say that budget is their primary challenge. In contrast, early stage DevOps companies are much more likely to be held back by budgets, with approximately 1-in-3 aspiring DevOps teams reporting budgets as their primary concern. 

DevOps challenges for aspiring DevOps teams versus challenges for fully DevOps teams

The shift between budget constraints and technological limitations as teams reach DevOps maturity underscores the importance of early investments in DevOps tools. If they choose the less effective or useful solutions, an aspiring DevOps company will see a worse return on investment and set themselves up for ongoing challenges on the road to DevOps. If they identify tools and capabilities that support long-term success, however, the odds increase that they’ll join the elite 16% who have succeeded in their DevOps transformation. 


Deployments Accelerate, But Automation Lags


One significant trend identified in the State of Testing in DevOps Report: 83% of teams have accelerated deployments in the last 12 months. This trend was consistent across mobile app development teams and web app development teams, signaling a growing emphasis on delivering new innovations to market at an increased pace.

Bar chart showing changes in deployment frequency YoY

That said, the survey also revealed that deployment frequency improvements are still needed by a vast majority of software development teams, as 80% of respondents shared that their organizations have release cycles of a week or longer. This includes the two-thirds of both mobile app development teams and web app development teams shipping on similar cadences. 

80% of teams aren't delivering high-velocity software

The fact that almost two-thirds of mobile app and web app development teams are shipping software once per month or less is striking, but particularly notable for mobile applications, which can face penalties from app stores if their applications aren’t updated frequently enough or fail to meet certain performance, accessibility, and compatibility standards. More frequent updates also improve an app’s standing in search rankings in app stores, building a competitive edge for agile companies who are able to ship software more often than their competition. Being able to consistently deliver new features, important security updates, and performance improvements are essential for acquiring and retaining customers, yet the majority of development organizations are currently unable to do so. 

Bar chart showing deployment frequency for web and mobile apps

The lack of high-velocity teams mirrors the large proportion of companies still struggling to automate their development pipelines: 87% of teams have yet to fully adopt CI/CD, with another 4% of respondents unsure of their automation capabilities. 

Pie chart showing proportion of teams with different levels of pipeline automation

Low pipeline automation impacts teams on multiple fronts. For one, teams with less automated pipelines face an uphill battle as they accelerate deployment frequency since their team’s workload is compounded by repetitive tasks, which increases burnout and turnover. The costs of low automation then multiply across the organization as workloads increase, burnout builds, and employees quit. Unless these organizations hire more employees or find tools to reduce the burden on their current workforce, there’s a human limit and financial cost to how fast their teams can deliver new innovations to market.

DevOps success, though a priority for 89% of companies, is still an ongoing challenge for two-thirds of companies. With technology limitations rising 44% since 2022, it’s clear that early investments in automation and tooling are critical for teams in the early stages of their DevOps transformation. Slow progress in pipeline automation indicates an opportunity to surpass competitors through smart investments in technology that further streamlines development and supports continued gains in deployment frequency. 

Benchmarking Quality: DevOps Maturity, Collaborative Testing, and Testing Tools


Software testing teams, despite being the connective tissue between how apps are built and how apps are used, are managing a rising tide of manual testing and legacy test automation challenges.

Bar chart showing types of testing tools used

Manual testing dominates the current testing landscape, with 74% of teams using it in their software testing strategies. A majority (60%) also reported using open source test automation tools like Selenium or Playwright, with slightly less than half of companies relying on commercial test automation tools to support quality.  

The reliance on manual testing extends to mobile application teams. Almost two-thirds of these teams report using manual testing in their quality strategy, followed by 56% of teams leveraging automated testing. Slightly less than one-third of mobile app development teams say they’re using third party device farms. 

The widespread reliance on manual testing is unsurprising, as manual testing is essential for exploratory testing for new features and edge cases. But if manual testing makes up the bulk - or even the entirety - of an organization’s software testing strategy, there’s a greater risk of bottlenecks as deployments accelerate. 

When it comes to automated testing tools, teams are more likely to leverage open source tooling until they reach the final stage of DevOps maturity. As organizations progress on their DevOps journey, the likelihood that they utilize commercial test automation tools increases significantly. 

Bar chart showing commercial test automation usage by DevOps stage

The impact of open source test automation can be seen in how development teams spend testing time. When asked about the most time-consuming testing tasks, a plurality of respondents (21%) ranked test maintenance as their top choice, followed closely by test execution (19%).  

Test maintenance is the most time-consuming testing task


Test maintenance rose from a distant second place in 2022 to earn the dubious distinction of being the most time-consuming testing task in 2024, with 1-in-5 teams reporting it as the largest time commitment needed for testing. 

pie chart showing the time spent on different testing tasks

The most time-consuming testing tasks closely corresponded with the most significant testing headaches, with one-third of respondents saying that test maintenance was their biggest pain point. This percentage is almost double the amount of teams who said that test maintenance was their biggest testing challenge in 2022, a massive shift that signals the challenges of flaky, unreliable automated tests as deployment frequencies accelerate. 

Bar chart showing the top testing pain points

The rise of test maintenance as the #1 testing challenge corresponds to mature DevOps teams empowering more people to contribute to testing, investing in more testing tools (below), and identifying technology limits as their main DevOps obstacle. Software testing and test automation are clearly important to DevOps transformation, as illustrated by a growing investment in automated testing tools, but unless test maintenance is sustainably managed, DevOps teams will struggle to reach their full potential as flaky tests cause delays in delivery cycles.

Collectively, these changes signal that DevOps teams embrace a culture of quality that engages everyone in testing, which is further supported by the fact that fully DevOps teams are the most likely to report that everyone on the team has a role in quality. Approximately 1-in-3 fully DevOps teams responded that testing is a shared activity across the organization, versus roughly 1-in-5 mostly DevOps teams or striding DevOps teams. 

Bar chart showing how QA Pros and Manual Testers Contribute more to testing in mature DevOps teams Fully DevOps teams reported a rise in testing contributions from QA professionals and manual testers, automation engineers, and software engineers, signaling that testing needs and testing collaboration increase with DevOps maturity. 

unnamed (12)

The testing needs and concerns of these different groups vary significantly, as do their areas of expertise. Meeting these varying needs, therefore, requires a diverse set of testing tools. 

unnamed (10)

The increase in roles contributing to testing coincides with an increased investment in software testing tools as organizations mature their DevOps practices. Virtually zero respondents (5% total) said their organizations were using no testing tools, while a similarly small percentage were using one testing tool (8% of aspiring and mostly DevOps teams, 7% of striding DevOps teams, and 6% of fully DevOps teams). A majority of companies (58%) are using 3+ testing tools, despite the fact that many teams are struggling to maintain sufficient test coverage and the fact that budget issues and technology limitations are the two most common DevOps obstacles. Software testing is clearly important for DevOps maturity, yet complex testing technology stacks aren’t solving the core challenges of testing in DevOps.

Bar chart showing the number of teams at each DevOps stage using 5+ testing tools

Once teams reach three or more testing tools, a pattern emerges: fully DevOps are more likely to be using a greater number of testing tools. A full quarter (26%) of fully DevOps teams are using at least five tools, with another 12% using four tools, and 31% using three. 

The growing testing tech stack illustrates the growing complexity of managing testing in DevOps. Not only do DevOps teams need automated testing tools that can be used by a variety of roles and skills, as seen above, but they also need to validate functional and non-functional quality, multiple devices and browsers, plus APIs.

Testing across each of these aspects is essential, but a growing testing tech stack means increased costs, increased silos, and increased onboarding time. Testing is an important priority for DevOps maturity, but teams need to consider what automated testing tools can help them avoid duplicative work, unnecessary costs, and siloed workflows. 

Automated Testing Tools Prove Key Investment for DevOps Progress


Given the impact of testing tools, it’s no surprise that a plurality of survey respondents reported that their organizations plan to invest in new automated testing tools to improve their approach to quality.

2024 Investments to Improve Testing

Underscoring the need to further improve testing tools on the road to DevOps is that new test automation tools are the #1 way companies plan to invest in testing, even as budgets are being limited. Aspiring DevOps teams are even more likely to buy new software testing tools, with 48% planning to make new purchases in the near future. This trend indicates that these organizations are prioritizing bringing their testing tech stack up to industry standards. 

The right testing tools can help address other quality priorities ranking high on companies’ to-do lists, which likely explains why new testing tools took the lead. 

New testing tools are also essential to support the wide remit of testing types being handled by in-house QA and development teams. The 2024 Testing in DevOps survey found that the vast majority of tests are being planned and executed by employees, not third-party consultants. 

Bar chart showing 2024 Extent of Outsourced Testing

Across web applications and mobile applications, functional testing and non-functional testing, teams are being asked to handle a greater share of testing strategies. This is a marked shift from mabl’s previous Testing in DevOps Report, which found that 61% of QA teams relied on some level of third-party support. Even specialized test types like accessibility testing and performance testing are largely performed in-house, indicating that companies are focused on building internal capabilities and talents while on the road to DevOps. 

AI Adoption and DevOps Maturity


Perhaps no technology has become essential as quickly as AI, which has sprouted from futuristic curiosity to must-have capability.

Pie chart showing AI adoption in software development

4-in-5 teams have started incorporating AI into their development processes, with 60% reporting that their development and QA teams have somewhat or fully embraced AI. Interestingly, that proportion rises to 76% for fully DevOps teams, indicating that DevOps maturity sets the stage for ongoing technological evolution and adaptation.

76% of fully DevOps teams who have somewhat or fully adopted AI

How AI is being applied to software testing varies by DevOps maturity and pipeline automation levels. When accounting for pipeline automation levels, the popularity of AI use cases shifts drastically. Organizations with fully automated pipelines are more likely to use AI to summarize test results. In contrast, fully DevOps teams are more inclined to harness AI to generate test cases.

Bar chart showing the top uses of AI in software testingunnamed (6)


The popularity of using AI to summarize test results is an interesting trend, considering that test analysis was ranked as the least time-consuming testing task.


unnamed (5)


In the same vein, fully DevOps teams using AI to generate test cases also aren’t using new technologies to tackle the most burdensome testing activities (test maintenance and test execution). This suggests that AI tools are still evolving to meet the most acute bottlenecks impacting testing in DevOps.

Benchmarking Quality: The State of Testing in Development Pipelines


With a growing number of testing tools and quickly evolving set of AI capabilities available, a final question remains: when are DevOps teams testing in their development pipelines?

Diagram showing when testing is being executed in DevOps pipelines

Both functional and end-to-end tests are carried out throughout development, though in notably different concentrations. Most teams (52% code stage and 41% pull request stage) are doing the bulk of their functional testing, most likely unit and integration testing, in the early stages of development pipelines. In contrast, end-to-end testing is more likely to be performed in the deployment stage of development. 

Bar chart showing when product issues are identified in development pipelines

The number of issues discovered late in the development process is likely connected to the number of teams with low test coverage across their web applications and mobile applications. 67% of both web and mobile app development teams report having less than 60% test coverage, while 1-in-5 teams have less than 20% test coverage.

Bar chart showing levels of functional test coverage across development teams

Test coverage challenges proved even more acute for non-functional testing: 73% of mobile app development teams have less than 60% non-functional test coverage, and approximately 1-in-4 mobile teams have less than 20%. On the web app development side, 71% of teams have less than 60% non-functional test coverage and 25% have less than 20% test coverage.

Bar chart showing levels of non-functional test coverage across development teams

A lack in non-functional test coverage can introduce serious headaches - and therefore churn - to customers. A 1-second delay in response time can reduce web conversions by 7%, and 53% of consumers will abandon a mobile site if it takes more than 3 seconds to load. Likewise, 45% of people will leave a slow application with a bad impression of a business. A lack of accessibility testing can result in lawsuits and penalties, particularly for industries like government, consumer retail, and healthcare. 

Effective non-functional testing is particularly important for mobile applications since Google and Apple have accessibility and performance criteria for their respective app stores. Failing to meet these requirements can result in delays in app updates, downgrades in search results, and even removal from app stores. 

Identifying defects that late in the development process can cause headaches for deployment frequency and DevOps maturity, particularly most development teams (53%) are considered mid-tier performers who take at least one full work day to deploy fixes, with a median of 11.12 hours (1.5 days) necessary to address issues. Though this aligns with medium performance according to DORA, 11 hours can cause delays when deploying on a weekly or daily basis. 

A key part of being able to address bugs efficiently and mature DevOps processes is an organization’s ability to collaborate cross-functionally, which is reflected in the handoff process between QA and developers. According to the 2024 Testing in DevOps survey respondents, 43% of development teams need to improve the handoff process. 

45% of mobile teams who say their defect handoff processes need improvement

Despite the fact that mobile teams are more likely to catch defects earlier in the development process (though 48% of bugs are caught in the deployment stage), issues collaborating between QA and developers is likely to undermine any efficiency gains made with early detection. 

Key Takeaways

key takeaways