Top 5 Lessons Learned in 2024 State of Testing in DevOps Report

Now in its 5th edition, the State of Testing in DevOps Report is the industry’s most in-depth look at the impact of software testing and test automation on company success. While many research reports focus on the immediate effects of changing test automation tools and tactics on QA, mabl's State of Testing in DevOps Report looks at the entire organization to understand how testing can transform an entire company, ultimately becoming a key competitive advantage.

To understand this impact, we surveyed over 500 development and quality professionals in the United States. Roughly 40% of these respondents held a leadership role in software development and QA, while 60% were practitioners on the frontlines. 

These respondents revealed that DevOps transformation is still being prioritized by 89% of companies, even though most are still navigating the road to DevOps maturity. It’s clear that DevOps is more important than ever, but changing technologies are rapidly shifting the challenges to maturity. 

Wherever you are in your DevOps journey, I hope the 2024 Testing in DevOps Report helps you benchmark your progress, inspires new opportunities for success, and shines a light on the importance of quality in our digital-first world. Explore 5 key trends from the report in this article, and download the full report here. 

#1 - Mobile App and Web App Development Teams Face Similar Challenges 

Mobile app development teams face similar challenges as their web app development counterparts, with a few notable exceptions. 67% of both types of teams have 60% or less test coverage on their applications, while 1-in-5 teams have less than 20% test coverage.

Lower test coverage means a higher likelihood of defects slipping into production, and for mobile applications, a higher likelihood of being rejected by app stores, who have their own quality and customer experience criteria that must be met in order to be approved. 

Lower test coverage is likely connected to the fact that roughly half (48%) of mobile app defects are found in the deployment stage of development, right before new features are shipped. The challenge is even more acute with non-functional test coverage, where 73% of mobile app development teams have less than 60% non-functional test coverage, and approximately 1-in-4 mobile teams have less than 20%. This indicates a serious need to continue shifting testing to the left in order to identify defects earlier in development pipelines, when they’re faster and cheaper to fix.

Achieving good test coverage is a complex process, especially for mobile app development teams since there are an exponential number of device and OS combinations to test. There are over 24,000 distinct types of Android mobile devices, which could be operating on any of the 14 major Android OS versions, as well as over 30 types of iPhones with 17 possible operating systems. Achieving effective test coverage at this scale at a reasonable cost while deploying weekly or daily updates is a truly monumental challenge.

Mabl’s new mobile app testing capabilities empower teams to seamlessly scale across devices without the cost and maintenance of infrastructure management. Register for a 14-day free trial to start creating comprehensive end-to-end tests for mobile apps in minutes.

#2 - Technology Limitations and Budget Constraints are the Biggest DevOps Challenges

One major shift identified in the 2024 Testing in DevOps Report: technology limitations rose from a distant second place to become the most common DevOps obstacle.

This is the first time in Testing in DevOps’ history that technology limitations have earned the dubious honor of being the most DevOps headache, particularly for mature DevOps teams. When looking at fully DevOps teams, 40% say they’re being held back by technology. In contrast, 1-in-3 aspiring DevOps teams reported budget as their most pressing obstacle.

#3 - Fully DevOps Teams Are Harnessing Modern Test Automation Tools

Despite technology limitations and budget issues, mature DevOps teams are investing in test automation tools and building cultures of quality. Survey respondents indicated that two-thirds of fully DevOps organizations are leveraging commercial test automation tools for QA, while just 40% of aspiring DevOps teams do the same. 

The growing investment in the testing technology stack - 26% of fully DevOps teams are using 5+ tools and 41% of all teams plan to purchase new testing tools - indicates that quality is a significant concern for maturing DevOps teams.

However, an increasingly complex testing tech stack also risks compounding budget constraints and siloed workflows, as an array of specialized tools comes with multiple licensing agreements, differing workflows and integrations, as well as increased onboarding needs. 

#4 - Test Maintenance is the Biggest - and Fastest Growing - Challenge for Aspiring DevOps Teams

Early stage DevOps teams are more likely to rely on manual testing and open source test automation tools in their QA strategies, the effects of which can be seen in how development teams spend testing time. When asked about the most time-consuming testing tasks, a plurality of respondents (21%) ranked test maintenance as their top choice, followed closely by test execution (19%).  

Building a sustainable automated testing strategy requires balancing each of these tasks to maximize the time and talents of everyone engaged in testing. Approximately 1-in-3 fully DevOps teams reported that “everyone” in their organization plays a role in quality, including QA professionals, developers, and product owners. Using their time effectively means that the entire company can deliver new innovations to market faster without sacrificing quality or inducing burnout. 

Unsurprisingly, the top testing pain point corresponds with the most time-consuming testing task. Over one-third of respondents said that test maintenance was their biggest pain point, a 138% increase compared to 2022.  A shift of that significance that signals that the challenges of flaky, unreliable automated tests are growing quickly as deployment frequencies accelerate.

#5 - AI Plays A Critical (And Growing) Role in Software Testing and Test Automation

The need to optimize human effort in software testing is a likely contributor to the rise of AI in test automation. 4-in-5 teams have started incorporating AI into their development processes, with 60% reporting that their development and QA teams have somewhat or fully embraced AI. Interestingly, that proportion rises to 76% for fully DevOps teams, indicating that DevOps maturity sets the stage for ongoing technological evolution and adaptation.

The popularity of using AI to summarize test results is an interesting trend, considering that test analysis was ranked as the least time-consuming testing task. In the same vein, fully DevOps teams using AI to generate test cases also aren’t using new technologies to tackle the most burdensome testing activities (test maintenance and test execution). This suggests that AI tools are still evolving to meet the most acute bottlenecks impacting testing in DevOps.

Let mabl Be Your Partner in Delivering Modern Software Quality 

The journey to DevOps has never been more important, or as complex, than in today’s digital-first world. The 2024 Testing in DevOps Report dives deep into the impact of software testing and test automation on DevOps, AI, and innovation, helping you understand how to maximize the value of your QA strategy. Download the full report here and join us on June 4th to hear a panel of quality leaders share their thoughts on testing in DevOps trends.