Roles Represented by Testing in DevOps Survey Respondents
Respondents were nearly evenly distributed across small businesses and startups, mid-sized companies, medium enterprises, and large enterprises, giving us insight into how quality strategies and DevOps adoption are transforming organizations of all sizes. Large enterprises with over 2000 employees had a slight plurality with 28% of respondents, along with small businesses/startups. Medium enterprises with 501 - 2000 employees had the lowest share of respondents at 21 percent.
Almost half (43%) of respondents work in software/hardware/advanced technology, with industries undergoing significant digital transformations like financial services/banking and manufacturing also seeing a notable number of respondents. Insurance and media/entertainment represented 4% of respondents, with retail, transportation, government, travel/hospitality and energy following at 3 percent. Respondents also reported working in nonprofits, communications, real estate, construction, and agriculture.
With a wide range of roles, company sizes, and industries represented by survey respondents, it’s possible to take measure of the current state of DevOps of adoption.
State of DevOps Adoption
DevOps Transformation by Company Size
Though organizations of all sizes are moving towards full DevOps adoption with automated pipelines, progress towards that goal was highly segmented by company size. Large enterprises lead the pack in terms of DevOps adoption and maturity, with 40% reporting that they are fully DevOps with automated pipelines. Another sizable contingent of large enterprises consider themselves mostly DevOps or striding towards DevOps transformation. In contrast, just 11% of medium enterprises report being fully DevOps and 22% of mid-sized businesses are fully DevOps teams.
Interestingly, more than a quarter of small businesses (27%) have fully embraced DevOps and fully automated their development pipelines, but 51% of respondents in this category were unsure of their DevOps progress. In comparison, just 18% of large enterprises, 20% of medium enterprises, and 10% of medium businesses felt the same. Roughly a third of large enterprises (31%), medium enterprises (27%), and medium businesses (29%) consider themselves striding towards DevOps, indicating a strong interest in all categories to move towards DevOps transformation.
DevOps Transformation by Year
Comparing 2021 to 2022, there were very few changes in the proportion of companies at each stage of DevOps transformation. Though there was a slight increase in the number of organizations who consider themselves fully DevOps with automated pipelines, there was a corresponding decrease in mostly DevOps organizations and aspiring DevOps teams. The only category to remain steady across both years was ‘striding’ towards DevOps - those partially through their DevOps transformation.
Software Development Pipeline Automation
A core part of DevOps adoption: pipeline automation. Unsurprisingly, automation implementation across survey respondents’ organizations closely followed DevOps adoption rates.
Automation in DevOps Transformation
Obstacles to DevOps Adoption
So what’s holding companies back from achieving their full DevOps potential? Cultural factors, such as the slow process of change (26%), a lack of prioritization (15%), and a lack of leadership (14%) continue to frustrate would-be DevOps practitioners. Technological limitations (18%) and budgetary constraints (17%) play a notable role in stymieing DevOps adoption, but respondents overwhelmingly indicated that people and process factors play a far greater role, with 65% of respondents citing a cultural challenge as their primary obstacle. Note that respondents were able to select multiple options as part of the Testing in DevOps survey to better reflect the full range of interconnected DevOps obstacles.
Biggest Obstacles in the Way of DevOps Transformation
Though the distribution of obstacles varied somewhat across different company sizes, a few commonalities were apparent.
DevOps Obstacles by Company Size
85% of respondents indicated that DevOps is sufficiently prioritized by their company, suggesting that sustaining change is the biggest challenge to adoption. The slow process of change was the top obstacle for three of the four categories, outpacing budget concerns and a lack of internal expertise. The one exception was small businesses and startups with less than 100 people, which are slightly more likely to say a lack of prioritization is a bigger DevOps obstacle.
Regardless of company size, leadership and change management continue to be the main sticking points in DevOps transformation.
DevOps Obstacles: 2021 vs 2022
Looking at how obstacles to DevOps adoption have changed over the past year, there are several optimistic signs. First, concern over budget constraints dropped from 21%, the second most common obstacle in 2021, to 17% in 2022. Respondents also reported fewer issues with internal politics and the slow pace of change, granted the latter is still the most common concern in 2022.
In 2022, we added a lack of leadership/internal expertise as an option for respondents, which tied with a lack of prioritization as the fourth most common DevOps obstacle. A continuing trend from 2021 - 2022: the human element is the most critical piece of the DevOps puzzle, not technology. 72% of respondents indicated that technology limitations aren’t a factor in their DevOps transformation, the same proportion as last year. The largest - and most difficult - DevOps obstacles are about enabling people through better processes.
Speed of Software Deployment
Changes in Deployment Frequency
75% of respondents reported improvements in their overall deployment velocity, but how significant that increase was strongly correlated with DevOps transformation and automation adoption. Those with heavily automated pipelines with mostly DevOps development practices were far more likely to report doubling deployment frequency in the past year than those with few automated workflows, suggesting that automated pipelines support faster deployments.
Deployment Frequency by DevOps Adoption Stage
According to our respondents, teams who have mostly or fully adopted DevOps tend to deploy new code to production either weekly or bi-weekly. Nearly half (43%) of those who are have fully embraced DevOps are on a weekly / bi-weekly schedule, and 18% are deploying updates daily or multiple times per day - faster than any of their counterparts. On the flip side, teams who are still early in the DevOps maturity are more likely to deploy new code on a monthly basis or even quarterly basis. Considering that most organizations successfully increased deployment frequency in 2022, it’s likely that more teams will move to weekly or daily deployments as DevOps practices mature and more of their pipelines are automated.
Deployment Frequency Changes Over Time
Comparing four years of data, it’s clear how the onset of the pandemic impacted deployment frequency. Weekly and daily deployments spiked in 2020, indicating that many teams adjusted their deployment cycles as digital-first experiences became the norm. As norms once again shifted with the resumption of in-person interactions, deployment frequency has shifted accordingly. In 2022, one-in-four teams are deploying at least once per week, with another 18% deploying on a biweekly basis. A sizable contingent, however, are still deploying on a monthly or quarterly basis.
Being able to accelerate deployment frequency is a key indicator of DevOps success by being able to continue to delight customers with digital-first experiences.
Deployment Frequency Changes and Customer Satisfaction
The agility enabled by more frequent deployments is strongly correlated with higher customer satisfaction. Among all the respondents who shared their pipelines sped up last year, in each of the categories the majority of them also indicated customer satisfaction was amazing / good. For teams that increased deployments by 50-100% they were 2.5x more likely to report positive customer satisfaction than terrible user sentiments.
Deployment Frequency by Level of Pipeline Automation
The ability to accelerate deployment frequency proved to be closely connected to pipeline automation. For respondents who deployed up to 25% faster, most of them (64%) have some or few of their workflows automated, while teams that reported deploying 50-100% faster, the gap was far more narrow. It was nearly split down the middle of teams who have few/some versus most/all of their workflows automated. But, for teams who reported they have more than doubled deployment speed last year, they are 1.5x more likely to have to all or most of their workflows automated. Automated pipelines are clearly a key factor in helping drive faster deployments.
Significance of Software Testing
Roles Involved in Software Testing
Deploying with quality, of course, is a make-or-break factor for companies that are accelerating delivery cycles. If new deployments pose a high risk of introducing bugs, both employees and customers will struggle. A positive sign: the diversity of roles engaged in software testing. QA professionals and manual testers lead the pack at 65%, with automation engineers, developers, and software engineers following. 1-in-4 respondents reported that everyone in their organization has a role in testing, indicating that a significant number of teams are embedding quality in their software development culture.
Time Spent on Testing Activities
Every quality professional knows when the rest of the world says ‘testing,’ they mean a wide range of tasks that encompass planning, management, maintenance, reporting, analysis, and actual test execution. To understand where testers are spending their time, we asked survey respondents to rank what tasks are the most time consuming. Test planning/test case management was the most commonly ranked as the most time consuming task, with 56% of respondents ranking it first or second. Test maintenance came in second, with 39% of respondents ranking it first or second.
Test execution (29%) and adding test coverage (28%) were closely tied for third, narrowly beating test analysis (26%) and defect reporting (22%). Respondents were the most likely to rank defect reporting and resolution as their least time consuming task, with 35% doing so. Despite the importance of evolving test coverage, test maintenance proved to be more resource-intensive.
Level of Reliance on Third Parties to Support QA
QA is largely handled by internal resources, according to survey respondents. A slim plurality (40%) said that some aspects of QA are supported externally, but over a third (35%) reported that all QA work is handled by internal employees. Very few organizations are fully outsourcing QA, with just 7% doing so, and 14% say that most of their QA effort is executed by third parties. Overall, most development organizations favor having their own employees perform and lead all or most of their QA efforts.
Types of Tests Outsourced to Third Parties
Security testing and UI/functional testing are the most common activities performed by third parties for organizations outsourcing their QA activities, though no test type held an outright majority. API testing, performance/load testing, regression testing, and integration testing were commonly performed by third parties, indicating that teams who do rely on third party testing are using external QA resources to handle a wide range of tests.
The range of tests being performed by a wide range of roles shows how quality - and testing strategies - are growing as deployment frequency is accelerating.
Importance of Test Coverage
Test coverage is a core measure of testing success for most software development organizations since it tracks how much of an application or website is covered by their software testing strategy. Though most organizations aim for high test coverage, the reality is that reaching 100% isn’t feasible, or even desirable, since maintaining that level of test coverage is likely an inefficient use of testing time and resources.
But as deployment speeds accelerate, maintaining an optimal level of test coverage becomes a challenge. Every time a new feature is introduced, quality teams need to adapt and expand their testing efforts to ensure all likely customer journeys are functional. Efficient testing is key to maintaining quality as deployments happen more frequently.
Deployment Frequency Changes and Test Coverage
Given this connection between the pace of testing and accelerating deployments, it’s unsurprising that most organizations that reported increasing deployments also have better test coverage. Teams that accelerated deployments by 50-100% were more than 3x as likely to have good or excellent test coverage. Despite the fact that these organizations deployed more frequently - potentially increasing the risk of bugs slipping into production - they were able to exceed customer expectations for quality.
Pipeline Automation and Test Coverage
It's clear that test coverage improves as overall pipeline automation is improved. Respondents who shared their organization has all automated workflows were over 7x more likely to have good / excellent test coverage indicating a strong connection between better test coverage and pipeline automation. Teams with very few automated workflows were 4x more likely to have almost no test coverage. Pipeline automation and test coverage go hand-in-hand when it comes to successful DevOps adoption and ultimately lead to higher customer satisfaction, as you’ll see later in the report.
Types of Automated Tests to Add in 2023
When asked how they plan to expand automated testing in 2023, survey respondents prioritized API testing, with 43% saying they planned to implement automated API testing. Considering that API adoption has surged in software development and that manual API testing is nearly impossible, it’s not surprising that quality professionals are very interested in implementing automated API testing.
Closely following were regression testing (40%), end-to-end UI testing (37%), and UI/functional testing (32%), indicating a common interest in automating routine testing that also reflects the customer experience. Non-functional tests like automated performance/load testing and accessibility also had notable interest in adoption in 2023, with 29% indicating interest in adopting automated performance testing and 21% looking to implement accessibility testing.
Stages of Development where Functional Tests are Executed
Stages where End-to-end Tests are Executed
Manual and automated end-to-end tests were similarly distributed across the development pipeline, according to survey respondents. 65% reported performing manual end-to-end tests in the deployment stage, slightly outpacing the 61% who executed automated end-to-end tests at the same point. End-to-end testing in the pull request stage was a distant second, 32% running these tests manually and 33% running automated end-to-end tests. Like functional testing, end-to-end testing in production has the greatest discrepancy between manual and automated testing: 36% of teams are performing manual end-to-end tests in production versus 29% of teams executing automated end-to-end tests.
Performance Testing Effectiveness
Performance testing is a type of non-functional testing that measures the stability and speed of an application. In other words, a crucial measure of how users actually experience an application. Considering that almost a third of respondents were interested in automating performance testing, it’s not surprising that 73% of respondents rate their current performance testing as merely good or needs improvement. Just 6% of respondents describe their performance testing as best-in-class.
Accessibility Testing Effectiveness
Similarly, accessibility testing is also still in the early stages of successful adoption. The vast majority (73%) rated their current accessibility testing practices as good or needs improvement, a serious gap when an estimated billion people around the world have different access needs.
Biggest Testing Pain Points
As shown by respondents’ testing goals for 2023, there’s still significant opportunity to evolve quality and testing strategies to meet the demands of DevOps. When asked about their biggest testing obstacles, respondents had mixed responses. A lack of test automation was the leading obstacle by far, causing concern for 1-in-5 development and testing teams, followed by test maintenance (13%) and collaborating with other teams (11%).
Testing obstacles are more technology-related. 53% of respondents indicated that technology limitations like a lack of test automation, test maintenance, and test case management, compared to process or people challenges like cross-functional collaboration, visibility of impact, small regression test windows, defect resolution, and a lack of testers, which are more people and process-related issues. In contrast, 72% of respondents said that technology limitations weren’t a major challenge to DevOps adoption.
Perceived Customer Satisfaction for Organizations without Test Automation
The need for technological evolution in testing becomes clearer when customer satisfaction is broken down by test automation adoption. For organizations who indicated the lack of test automation was their biggest testing challenge, they are likely to describe their customer satisfaction as good or needs improvement, with just 18% ranking user happiness as amazing.
Metrics Used to Improve Quality Results
What can’t be measured, can’t be improved. So how are testing professionals evaluating quality? Almost half use the effectiveness of tests as a barometer of testing success, closely followed by test coverage at 48% of respondents. Peer review also proved to be a common metric for quality teams, with 47% reporting that their organization uses this practice. Other common tactics for improving quality included evaluating testing redundancies or opportunities for reuse (38%), optimizing testing execution/timing (38%), and evaluating pass rates (37%). Worth noting is that most of these practices fall under test analysis or test execution, two of the tasks on which respondents were able to spend the least amount of time.
Biggest Pain Point when Utilizing Test Results
When it comes to maximizing the value of testing, respondents were mixed on obstacles. 1-in-4 reported that a lack of information around the root cause was their most significant challenge, while 22% said that slow processes around defect resolution was holding them back. Unreliable tests were problematic for 26% of testers, and a negative cultural perception of test failures caused issues for one-tenth of respondents. Like the overall obstacles to testing, the more common challenges to using test results skewed more towards technological limitations, contrasting with overall DevOps issues.
Perceived Value of Testing
Perceived Value of Quality Assurance at Organization
To address the cultural, human, and technological challenges limiting the potential of testing, it’s critical to explore how testing (and overall quality efforts) are viewed across the software development industry. 1-in-5 teams consider QA as a strategic effort in their organization, while almost half of respondents reported that QA is seen as very important to their overall organization. 25% described QA as somewhat important, and just 7% say QA is considered insignificant - an important measure of how organizations are elevating testing and quality.
Value of QA and Test Engineers Compared to Software Engineers
In the same vein, survey respondents were asked how their companies value QA and test engineers in comparison to developers. Their answers mostly reflected the overall perceived value of QA, with a quarter reporting that QA plays a strategic role at their company. Another 40% of respondents responded that their quality teams have an influential role in strategy and decisions, while 26% said that QA has little influence in their organization’s strategy or decision-making process. Like the general perception of QA and test engineer value, less than ten percent of respondents said that QA has no influence.
Leaders of Application Quality Investments
Quality leadership can vary vastly depending on the size and structure of a company and development organization. Though no leadership took an outright majority among survey respondents, a plurality reported that the Head of Technology/CIO/CTO leads application quality investments. Roughly a fifth of teams reported that these investments are helmed by a Head of Quality, and 15% said their Head of Product/CPO/Product Owner directs quality investments. A small number - 8% - have a Quality Center of Excellence Leader guiding testing and quality, while a very small percentage have executives outside the engineering/product organization taking a leadership role in quality.
Leaders of App Quality Investment by Company Size
The findings for quality leadership, of course, are heavily influenced by organization size. Looking at application quality leadership by company size, it’s most common for the CTO/CIO/Head of Technology to lead application quality at all types of organizations, from startups and small businesses to large enterprises. With the exception of small businesses, the Head of Quality was the second most common response, while the Head of Product and Head of Engineering battle for the third place ranking.
Inhibitors to Investing in Quality Efforts
Despite the impact and importance of application quality, many organizations are inhibited from further investing in quality efforts. Unsurprisingly, the most common obstacle was budget, which was named as the main limitation by 27% of respondents. Like DevOps obstacles, the slow process of change was the biggest challenge for 19% of teams, tied with a lack of skilled talent. Other inhibiting factors include: lack of prioritization by the company (14%), a lack of leadership (8%), and internal politics (8%). Technology limitations were ranked last.
Impact on Product Release Cycles
Typical Product Release Sentiment
Sentiment of Bug Handoff to Engineers
Managing release stress - and potential disruptions to users - begins with streamlining processes across the software development lifecycle. Considering that most DevOps and quality obstacles are process and culture related, it’s fair to say that improving collaboration means improving DevOps maturity. According to respondents, the crucial handoff of product issues between engineers and QA still has room for improvement. 46% said that their organization’s handoff process needs improvement, with another 9% saying it’s downright painful. In contrast, 37% said that their handoff process was pretty good and just 7% described it as totally seamless.
Collaboration Between QA and Engineers
With that snapshot of QA/developer collaboration, it’s unsurprising to see mixed views on overall cross-functional work. The process to identify and resolve bugs proved to be the greatest point of frustration for respondents, with 13% describing it as painful or in need of improvement and 9% saying it was pretty good or seamless. A similar collaboration point, finding information to perform root cause analysis, caused a similar amount of frustration, but 14% of respondents ranked their information sharing process as solid.
Communication and unreliable tests causing false results frustrate 7% of respondents, narrowly beating out positive responses (6%). Slightly more respondents (6%) thought their organization’s perception of test results was positive, compared to 5% who thought it was negative. Again, we see process issues causing the largest headaches, with information sharing and communication leading negative responses. Effective communication and collaboration are core tenets of any development organization adopting DevOps, so identifying and resolving some of these pain points could help teams be more successful.
Stage Where Most Bugs are Usually Identified
Almost half reported identifying the bulk of bugs in the deployment stage of the software development life cycle, right before deployment. Another 36% reported finding most bugs in the pull request stage, before code is merged with the main branch. Less than one-tenth of teams are consistently finding bugs in the earliest stage of development, and 11% of teams are catching bugs via….their customers.
Test Coverage and Stage Bugs are Identified
Though teams who find bugs in the pull request stage or deployment stage dominate the responses, nuances emerge when responses are segmented by test coverage. Teams with good / excellent test coverage were 2.5x more likely to find bugs in the code stage, and 2.3x as likely to identify bugs in the pull request stage. This connection is likely the impact of shift-left and continuous testing. When testing happens earlier in the development cycle, bugs can be uncovered earlier in the development cycle. If teams want to uncover bugs faster, they need to improve test coverage.
Time to Bug Resolution
The process of finding and resolving bugs ranked as a top challenge for respondents. So it’s no surprise to see that most teams take hours, if not days, to fix defects. One-in-three teams are able to resolve issues within a single business day, but another third need 1-3 days to fix a bug. 22% need 24-48 hours to implement a solution, and 8% need more than two days to resolve a defect. Once teams need more than a day to fix a bug, they begin risking serious delays to deployments and impact on customer satisfaction. According to respondents, 61% of teams are routinely navigating this risk.
Test Coverage and Issue Handoff to Engineers
Better test coverage is correlated with better handoffs, which is clearly needed by most software development organizations. Teams with good / excellent test coverage are almost 3x as likely to say their handoff process is good or seamless, indicating that adequate testing has an impact on collaboration.
Test Coverage and Time to Resolve Bugs
Improving test coverage also likely has a positive impact on the bug resolution process. Teams with good / excellent test coverage are roughly 3x as likely to resolve bugs in 24 hours or less compared to those with poor test coverage, while 24% of those with high test coverage are able to fix defects in 8 hours or less. In contrast, only 15% of teams with minimal or non-existent test coverage are able to resolve bugs in a business day.
Business Impact of Testing
Deployment Speed Change and Test Coverage
Among the software development teams who accelerated deployment frequency in the past year, better test coverage appeared to be a common connection. Those who improved deployment frequency by 50-100% were 2.5x more likely to describe their test coverage as good or excellent, compared to those who had no change to deployment frequency.
Deployment Speed and Customer Satisfaction
Pipeline Automation and Customer Satisfaction
Earlier in this report, respondents revealed a connection between better test coverage and broader pipeline automation. Taking that correlation a step further, it becomes clear that teams with more automated pipelines are highly likely to have higher test coverage and happier customers. Teams with fully automated pipelines are 3x more likely to rate their customer happiness as good or amazing, while teams with most automated pipelines are 3.2x more likely to say their customers are very happy.
QA Perception and Customer Satisfaction
Given the connection between test coverage and customer happiness, it’s unsurprising that organizations who value QA are also more likely to have satisfied users. 64% of teams that value QA say they have very happy customers, whereas teams who value QA less report their customer satisfaction is lower. When QA has the same level of influence and strategic importance as developers, customers and the business benefit.
Perceived Value of QA and Test Coverage
There’s also a strong connection between the value QA as a strategic asset and high test coverage. Teams who see QA as very important or strategic were twice as likely to report good or excellent test coverage compared to teams that perceive QA as insignificant or only somewhat important.
Test Coverage and Customer Satisfaction
Higher test coverage translates to higher customer satisfaction: teams with high test coverage were 1.6x as likely to report high customer happiness as those with minimal or non-existent test coverage. When QA is supported as a strategic asset within the software development organization, test coverage is high, and customers see the impact.
Release Sentiment and Test Coverage
Smooth releases also has a strong correlation with high test coverage. For respondents who shared that their code releases are manageable or low stress, they are 3.6x more likely to have good or excellent test coverage. This showcases how test coverage improves collaboration and working conditions across the organization.
Large Companies Made Greater DevOps Gains
- Enterprises with 2000+ employees were almost 4x more likely to be fully DevOps than medium enterprises with 501-2000 employees
- Large organizations were almost twice as likely to consider themselves fully DevOps than medium businesses or small businesses
Culture, Not Technology, Hinders DevOps
- 72% of respondents said technological limitations weren’t a major DevOps barrier
- The slow process of change was the top DevOps obstacle for the second year in a row
- A lack of leadership/expertise and lack of prioritization tied as the third-most common challenges
Most Organizations Accelerated Deployment Frequency
- 75% of organizations increased their deployment frequency in the past year
- One-in-four teams are deploying at least once per week, with another 18% deploying on a biweekly basis
- Teams who increased deployments by 50-100% were more than 2.5x as likely to report happy customers
Testing-Focused Teams are Collaborative Teams
- Teams who reported smooth handoffs between QA and developers were 3.6x more likely to have high test coverage
- Teams with good/excellent test coverage were ~2x as likely to report smooth/stress-free deployments
Testing is Growing, to the Delight of Consumers
- A quarter of respondents said that everyone in their organization plays a role in testing
- Teams who see QA as very important were twice as likely to report good or excellent test coverage
- Teams with high test coverage were 1.6x as likely to report high customer happiness
- 48% of teams say testing and QA are very important, while just 7% say testing is insignificant
Testing Improves Development Processes
- Teams with good/excellent test coverage were 2.5x more likely to find bugs in the earlier stages of development
- Teams with high test coverage are roughly 3x as likely to resolve bugs in 24 hours or less
- Teams with good/excellent test coverage were ~2x as likely to report smooth/stress-free deployments
Non-Functional, API Testing Testing Poised for Widespread Adoption
- 43% of teams plans to add automated API testing in 2023; 40% plan to adopt automated regression testing
- 73% of respondents rate their current performance testing as merely good or needs improvement; 28% of teams plan to start automated performance testing in 2023
- 73% rated their current accessibility testing practices as good or needs improvement; 21% plan to start automated accessibility testing
- 53% of respondents indicated that technology limitations were holding back their testing strategies, compared to process or people challenges that hinder DevOps.
Successful DevOps Teams Automate More, Test More, Deploy More
- Fully DevOps teams were more than 5x as likely to deploy multiple times per day
- Teams with fully automated pipelines were 7x more likely to rate test coverage as good/excellent
- Teams who accelerated deployment frequency by 50-100% were 3x as likely to have good or excellent test coverage
- Teams with fully automated pipelines are 3x more likely to rate their customer happiness as good or amazing
DevOps transformation for organizations of all sizes continues to be limited by the slow process of change, which is further hindered by a lack of leadership and prioritization. But companies that are finding DevOps success are doing so by also evolving their testing, pipeline automation, and development culture simultaneously. Survey respondents who described their teams as fully or mostly DevOps were the most likely to also report highly automated development pipelines, collaborative testing practices, and most importantly, high customer satisfaction. The old “move fast and break things” ethos is clearly outdated. Successful DevOps teams are those who have learned how to move fast but remain grounded in quality and customer satisfaction.