DevTestOps

Landscape Survey 2019

Is DevOps Actually Paying Off?

INTRODUCTION

Welcome to the first ever DevTestOps Landscape report! This report is based on a survey of the software testing community designed to understand their adoption of modern development practices and their impacts on testing. You might be wondering: “But how does this survey differ from other DevOps Landscape surveys out there?” Good question. Unlike other DevOps surveys, this one is focused on the evolving role of the Tester. As more organizations make the shift to DevOps and adopt release processes like CI/CD, testing is impacted in so many different ways. We’re mapping out the landscape of testing in DevOps to answer questions like:

• How has the rise of DevOps and Continuous Delivery affected testing?

• How much testing is being automated today? Does it matter?

• Can the least and most effective testing practices in DevOps be identified, to benefit teams moving to DevOps?

One of our questions the survey answered was whether or not DevOps is paying off. Here’s a hint: It’s not! Read on to find out why.

THE RESPONDENTS

We set out to better understand the landscape of testing in DevOps. Development is always paving the way in terms of leading software delivery practices, and testing has no choice but to follow through and put together the pieces as they go, often having to learn on the fly an influx of jargon like CI/CD, configuration management, MTTR, and more. That’s why we want this report to see where the state of testing in DevOps has landed. How are teams operating differently since waterfall to now? In what major ways do teams today succeed? In what way do they fail? What cultural approaches make them succeed? What tactical approaches do they take? Can we sift through the buzzwords and fads to find leading practices that actually contribute to success?

In order to get the data, we sent a public survey out to the testing community via our social networks and at testing conferences and received about 500 responses. We know we can’t claim the results represent the whole community. However, the consistency in the responses, as well as correspondence between the responses and our early market research lead us to believe the results we have are significant, and accurately represent the testing community.

An important aspect of our analysis was identifying if there was a variance in the type of tester responding to the survey - are they part of a “traditional” testing team in the sense that they ship code infrequently and have siloed testing roles? Or are they part of a modern development team that ships new code every day and takes a whole team approach to testing?

 As agile and DevOps are cultures, and implementation varies wildly across different teams, we began the survey with questions to help us understand our respondents in terms of software development practices, rather than flatly asking how they identified themselves. These questions assessed a variety of aspects such as cloud adoption, amount of test and pipeline automation, release frequency, and team dynamics.

Each response was weighted more heavily if it trended towards an agile and DevOps culture, and scores were tallied up and assigned to each respondent. The respondent’s category was finally identified based on their final score.

Traditional respondents were identified as having siloed testing roles, little to no automation, and long release cycles. Transitioning respondents had some automation and a higher release frequency. Modern respondents had many processes automated, short release cycles, and testing tasks were done by many different roles in the organization.

Based on our weighted scoring system, the highest score you could get was 11.2. We saw a full range of scores, from 0 to 11.2, with an average of 6.1. As a result, we found that there was a fairly even split between traditional and modern teams, with a large number of teams that were in transition.

Knowing this, let’s dive into the data!

DISTRIBUTION OF RESPONSES

BY SOFTWARE DEVELOPMENT CULTURE

CLOUD ADOPTION

Before we dig into the juicy stuff, keep in mind that for the sake of clarity, graphs in this report were created without depicting the outliers, these being answers with less than 1% of respondents.

We first asked respondents to tell us about their software delivery pipeline. How much of their pipeline is automated? How much testing is automated? How often do they ship? Do they use cloud infrastructure technologies like AWS, Azure, and GCP?

As DevOps doubles down on minimizing infrastructure operations with such innovations as infrastructure as code and cloud services, modern development teams would no doubt have cloud infrastructure. It makes it easier to be more agile not only when it comes to shipping software, but in scaffolding environments in each stage of development. With the availability of the private cloud, adoption is even accessible to companies that once was wary of cloud deployments because of high security standards.

48% of respondents reported to freely use cloud services, while an additional 23% are using cloud, but only for limited, specific purposes. An additional 18% were not yet using cloud infrastructure yet, but were looking into moving to the cloud. That’s 71% of respondents on the cloud, leaving only about 12% of respondents sticking to on-premise deployments.

71% OF ORGANIZATIONS

ARE EITHER ADOPTING CLOUDSOLUTIONS OR ALREADY ON THE CLOUD

THE MOST POPULAR APPS

TESTED TODAY ARE WEB-ENABLED

The high adoption of cloud aligns well with the most popular type of applications being tested, which are all web-enabled: web applications, websites, and browser-based mobile apps.

CI/CD ADOPTION

The top reasons to adopt a continuous integration and delivery pipeline is to deliver new features to users faster, have a stable operating environment, and faster resolution time for production problems. Therefore, CI/CD is core to the concept of DevOps, as it involves both the dev and ops (and test) teams to agree on how to implement the pipelines that tie development and production environments together.

However, how much of your pipeline do you need to automate? You can choose to automate only segments of your software delivery pipeline depending on the needs and preferences of your team. So we asked our respondents about continuous integration, continuous delivery, and continuous deployment adoption separately, each question including the options “Yes”, “No”, and “No, but transitioning to adopt it soon", to take into account all those in between.

At least 25% of all respondents are transitioning to these modern processes, so it’s reasonable to expect the number of modern teams to increase next year.*

*If you want more information about continuous integration, continuous delivery, and continuous deployment, there’s a helpful blog that goes into the 3 in depth: mabl.com/blog/what-is-cicd

MANY TEAMS HAVE CONTINUOUS INTEGRATION

BUT ARE LESS LIKELY TO HAVE CONTINUOUS DEPLOYMENT OR DELIVERY

MOST TEAMS SHIP

ON A WEEKLY TO MONTHLY BASIS

Release Frequency

Of course, a core trait of modern software development is the frequency of releases. The more frequently you ship, the shorter the feedback loop is between development and customers.

The most popular deployment frequencies were monthly and weekly, with about a fifth of respondents in each category.

When correlated with customer happiness, the majority of teams (40%) that reported a terrible customer happiness level (categorized by “Our customers only stay because they have nowhere else to go“) reported to have a yearly deployment frequency. On the other hand, respondents who had amazing customer happiness (categorized by “Our customers love our product and all open issues are nice-to-haves“) shipped on a monthly basis.

SHIPPING INFREQUENTLY

NEGATIVELY IMPACTS CUSTOMER HAPPINESS

A part of that may be due to how reactive these teams are to bugs found in production. When comparing customer happiness levels to MTTR (mean time to resolution), those with amazing customer happiness levels fixed critical bugs within 1-8 hours of being discovered in production, and those with terrible customer happiness levels fixed critical bugs after more than 48 hours.

MEAN TIME TO REPAIR:

ALL RESPONSES

LOWERING MEAN TIME TO REPAIR

POSITIVELY IMPACTS CUSTOMER HAPPINESS

TOOLING CULTURE

We all know that tooling doesn’t magically make you DevOps, but at the same time, it’s important to use them to be more effective in the transition. Thus, this question was less to identify specific tools that were adopted (that comes later), and more to identify how much culture around tooling influences success in modern software teams.

We wanted to see what motivated teams to experiment with new tools. Are the teams discouraged to look for new tooling because of lack of budget or support at an organizational level? Or has the team decided within itself that they’re satisfied with their productivity levels?

We included the questions “Who decides on new tooling?” and “Does the organization readily invest in training and tools?” to help us figure this out.

MOST ORGS READILY INVEST

IN TRAINING AND TOOLS

Q: Your organization readily invests in training and tools to improve testing and quality.

About 59% of the respondents feel that their organization readily invested in new tools and training for improving testing and quality.

87% OF TEAMS 

MAKE TOOLING DECISIONS COLLABORATIVELY

Q: What best fits how decisions are made around adopting new tools and processes?

About 52% of the respondents make decisions about tooling as a team.

ONLY HALF OF RESPONDENTS

ARE SATISFIED WITH THEIR TESTING TOOLING

Q: How satisfied are you with your testing toolset?

About 53% of respondents are satisfied with their toolset.

71% OF RESPONDENTS SEARCH FOR NEW TOOLS

TO IMPROVE QUALITY, SEVERAL TIMES A YEAR

Q: What best fits how often your team experiments with new tools, practices, strategies to improve product quality?

The majority of repondents experiment with new tools and technologies regularly to improve quality.

When digging into this further, most of the respondents regularly experimenting with new tools and tech turned out to be a mix of those who were neither satisfied nor dissatisfied, or totally unsatisfied with their toolset - as expected. Naturally, if you don’t like what you have, you’re going to look for something else.

What’s unexpected is the strong correlation between low tooling dissatisfaction and high customer happiness. We found that people with the happiest customers are 3x less satisfied with their tooling than those with unhappy customers. This leads us to believe the highest performing teams are never satisfied with status quo and therefore always looking for new tools.

It also makes sense when you factor in the workflow of those underwater with customer issues. It can be hard to find the time to fix under-delivering tooling when you’re spending the majority of your time just trying to keep up. These teams don’t have time to even think about what could be done better, because they’re in a hole of the immediacy of solving urgent issues, rather than forming a sustainable strategy. Teams with happy customers and consequently under less pressure have more time to optimize. But this still begs to question: Why such low customer happiness? Are teams with unhappy customers less customer centric? Are they relying too much on tooling?

RESPONDENTS HAPPY WITH THEIR TOOLING

ARE MORE LIKELY TO HAVE UNHAPPY CUSTOMERS

QA ACTIVITIES

We now arrive at the part of the report where we review how testers test. First we asked about the QA tasks conducted within the organization. Exploratory and automation coding skills were in no surprise the leading answers, with test design and test maintenance following closely behind. Despite the clear lean towards certain QA activities, when correlated against customer happiness, % increase of bugs found after release, and deployment schedule slip, there was no indication that any type of QA activity had more impact on improving those metrics than another.

        QA ACTIVITIES

        BY POPULARITY

TYPES OF TESTS

RUN MANUALLY AND AUTOMATICALLY

We also asked respondents to indicate which tests they conducted, and separately asked if they conducted said tests manually (not triggered automatically or scheduled) or automatically in their CI/CD pipeline. This let us compare general preferences of how these tests are to be run.

The most popular tests conducted manually were functional and end-to-end tests, while the most popular tests to be automated were unit and functional tests.

15% of respondents didn’t automate any tests in their delivery pipeline.

We next split up the respondents by the amount of tests they automate, and correlated them against the increase of bugs discovered during the days following a release.

We found teams that automate little to none of their tests are 13% more likely to find more bugs the days following a release than teams that automate most or all of their tests. This indicates that teams who automate more have more predictable builds.

Q: “AROUND THE DAYS FOLLOWING A RELEASE,

HOW MANY MORE BUGS ARE FOUND AND REPORTEDTHAN BEFORE RELEASE?”

We wanted to see how much having more predictable, less buggy builds affected customer happiness. So this time we correlated the same respondents with customer happiness levels. This revealed that teams who automated most of their tests vs. teams that had little to no automation were about 18% more likely to have pretty good or amazing customer happiness levels.

TEAMS THAT AUTOMATE ARE 25% MORE LIKELY

TO HAVE PRETTY GOOD CUSTOMER HAPPINESS LEVELS

Interestingly, correlating the amount of automation around satisfaction levels revealed that testers are more dissatisfied with their testing process the more they automate. A big contributor to that could be the maintenance burden of automated functional and end-to-end tests, the most popular types of tests to be automated as we saw earlier.

Tools for effective, robust functional UI testing have remained unavailable to noncoders for a long time, and even then required a highly specialized skillset that’s difficult to hire and train for.

As mabl’s mission is to bridge the gap between tests that scale in DevOps,* and tests that are easy to create and maintain, we hope to see this satisfaction level

*mabl offers scriptless cross-browser testing, auto-healing tests, visual testing, and diagnostics in one simple service. Visit www.mabl.com for more information.

TEAMS THAT AUTOMATE

MOST OR ALL OF THEIR TESTS ARE MORE DISSATISFIED WITH THEIR TESTING PROCESS

TESTING CULTURE

In this part of our analysis, we wanted to understand team dynamics around QA tasks, stress levels, satisfaction levels, and other traits that could paint a picture of our respondents’ team culture.

We started off by asking about which roles contributed to testing, since a big part of testing in DevOps is having a whole team approach to testing. We asked which roles on the team created or defined end-to-end tests. We kept the scope of this question to end-to-end tests because unlike component or unit tests, end-to-end tests can be conceptualized by all types of roles in an organization, from developers, to testers, to product owners to analysts.

Q: “WHAT ROLES CREATE OR DEFINE

END-TO-END TESTS IN YOUR ORGANIZATION?”

74% OF RESPONDENTS

FEEL MODERATE TO HIGH LEVELS OF STRESS AROUND RELEASE DAY

Then we looked at stress levels around release days, when your code is finally hitting the users for the first time. The majority of respondents, at 31%, felt moderate amounts of stress, and the majority of respondents felt higher levels of stress than not.

We expected that the more often teams shipped, the less stress they would feel. It’s reasonable to assume that if releases are incremental, there should be less surprises in production, hence, less stress. We ran a crosstabulation against stress levels and the different types of teams that we derived at the start of this report. Interestingly, this analysis told a different story.

In fact, 21% more modern teams felt higher levels of stress than traditional teams, and 23% more than transitioning teams.

RESPONDENTS ON DEVOPS TEAMS

FEEL THE MOST STRESSED

CUSTOMER HAPPINESS LEVELS

BY TEAM TYPE

We then dug into this further, we found that the customer happiness levels for all types of teams was about the same.

Though about 10% more modern teams had amazing customer happiness levels than traditional teams, they fell 9% behind in the pretty good happiness level range.

We can suspect modern teams are doing some things better than the rest, but what is it? We already established that shipping more frequently contributed to customer happiness, but what’s hindering them from pulling ahead on a broader scale?

When we looked at how much manual and automated tests were being done by types of teams, that gave us a clue.

This chart shows only respondents who conduct 75% or more their tests either manually or via automation. The ratio of automation to manual testing in traditional and modern teams nearly mirror one other. Traditional teams do 2x more manual testing than modern teams, and vice versa.

This makes sense - the more you ship, the less time you’ll have for manual testing. But the increase of automation doesn’t seem to offset that gap of as much as you’d think.

What seems to be more important that just simply automating is having a testing strategy that balances automation and manual testing.

27% OF MODERN RESPONDENTS

AUTOMATE 75% OR MORE OF THEIR TESTS

TOOLING

Lastly, tooling. Everyone wants to be able to validate their choices, or at least sneak a peek at what other teams are using. We asked about CI/CD platforms, testing tools, test case management tools, and issue tracking tools. Read on to see what tools are the most popular right now.

Q: "WHAT CI TOOL

DOES YOUR TEAM INTEGRATE YOUR TEST AUTOMATION WITH THE MOST?”

The most popular CI/CD tool is Jenkins. Notable write-in options included Azure DevOps, TFS, and Gitlab.

Another non-shocker is the most popular issue tracker tool being Jira. Noteable writein options included TFS, GitHub, Bugzilla, and Azure DevOps.

Q: "WHAT ISSUE TRACKING TOOL

DO YOU USE?"

Q: "WHAT TEST CASE MANAGEMENT

TOOL DO YOU USE?"

There were many respondents who weren’t using any test case management tools at all. The most popular tool was TestRail. Notable write-in options included Microsoft Test Manager, TFS, TestLink, qTest, HipTest, and of course, our love/hate Excel Spreadsheets.

Q: "WHAT TESTING TOOL DO YOU USE?"

Lastly, we asked for the testing tools they used. Selenium WebDriver was the clear winner in this category, with Cucumber coming in a far second.

CONCLUSION & TAKEAWAYS

We asked about the satisfaction levels of their testing process, tooling, and testing effectiveness to see where the landscape might be headed. The impetus for innovation is a problem, after all.

We asked questions about process and effectiveness separately because we wanted to differentiate between the testing activities and any issues outside of those activities that can affect the desired results, such as handoffs between testing and development, or misalignments between company goals and implementation.

Q: “HOW SATISFIED ARE YOU

WITH YOUR TESTING TOOLSET?"

The respondents responded similarly to each question about testing effectiveness, process, and tooling satisfaction. The majority response for each question was “Satisfied”, so we analyzed the responses with an upsidedown view on the data.

47% of respondents feel that there is room for improvement in their tooling.

Testing process covers testing activities and automation, and 44% of respondents feel there is room for improvement.

Q: “HOW SATISFIED ARE YOU

WITH YOUR TESTING PROCESS?"

Q: “HOW SATISFIED ARE YOU

WITH YOUR TESTING EFFECTIVENESS?"

Testing effectiveness covers anything outside of testing activities, which can include handoffs to other operations in the organization, empowerment, team dynamics, etc. 39% of respondents feel there is room for improvement.

IS DEVOPS ACTUALLY PAYING OFF?

The responses for these three questions about tooling, effectiveness, and process satisfaction align well with the response trends for questions centered around testing activities and culture. We found that the density of responses would taper off towards the higher-achieving ends of the scale. There are some that are reaching the potential of DevOps (the modern teams), but very little is reaching the potential of testing in DevOps - what we call DevTestOps.

This leads us to conclude that DevOps, which has many clear advantages like giving you a competitive edge in fast moving markets, is not the holy grail we think it is - at least, from a testing perspective. This isn’t the best news, as 79% of our respondents categorize as transitioning. Are they putting in all this work of implementing CI/CD and test automation just to get high stress levels and marginal improvements in customer happiness levels?

No. There is a way to truly improve, and our respondents know it too, as we can see from their satisfaction levels. The real emphasis in

achieving more than a competitive edge, but happy customers and happy teams is not to focus on just automating “all the things” and shipping frequently. It’s about building a solid testing strategy behind it so you know what to automate, and what to test manually. You need to build a quality strategy for production as well. Testing can’t be seen as a stop gate, but more of a checkpoint in a never-ending race.

Production isn’t the end here. That’s when your product finally comes alive to your customers - it’s merely the beginning. This doesn’t apply only to transitioning teams, but it applies to all teams - traditional and modern teams alike.

But how to do this? Granted, with higher release frequencies, we can’t expect the ensure this level of quality before release. But as we clearly saw from our data, there’s a direct correlation between Customer Happiness and Mean Time to Repair. The best thing teams can do is put a higher focus on lowering their MTTR in production to reduce the impact issues have on their customers. This is going to require an improvement on all three of these categories: testing process, effectiveness, and tooling.

THE KEY TO BEST IN CLASS CUSTOMER HAPPINESS ISN’T JUST SHIPPING FASTER -

IT’S FASTER MTTR

 

1) PROCESS

You need to find the right balance between automation and manual testing. You can’t go all in on one and not the other. This doesn’t mean you need to have two different testing teams. There are plenty of tools, like mabl, that combine the robust scalability of coded test automation with the ease of record and play tools. Manual testers can automate repetitive tasks so they can focus on quality checking that requires human interaction such as exploratory testing.

2) EFFECTIVENESS

Testers need to work more closely with dev and ops teams (hence DevTestOps) in order to collaborate on the entire delivery pipeline, especially since quality still matters far into production. They can be incredibly valuable when focusing on finding issues faster. By speeding up issue detection, they help lower your MTTR (mean time to repair).

3) TOOLS

With the evolution of the testing role and software development as a whole, testing tools need to evolve, too. So many teams are still using Selenium, but we all know how flakey and unreliable Selenium-based test scripts can be. We also saw earlier that flakey automation is likely a contributor to the high stress levels of respondents who have more automation than others.

New age tools for modern testing teams, like mabl, work reliably with auto-healing capabilities, and can easily run in production environments to help teams find issues quicker. They can also do more than give you a red or green status light. These tools help teams find the root cause of issues faster by providing logs, screenshots, and machine intelligence backed insights to provide deeper and quicker understanding of the application under test.

FINAL TAKEAWAYS

TRADITIONAL TEAMS:

You can’t keep a yearly deployment frequency. You need to evolve. You’re just not going to keep up with competitors who are innovating faster. But learn from existing DevOps teams and make sure you’re equipped with a thorough testing strategy before you embark on your transition.

MODERN TEAMS:

You need to put some emphasis back on testing. You ship fast, so you need to improve your MTTR. Don’t rely on automation alone, and let your testers test. And don’t forget about your team’s wellbeing. Figure out the reasons why your teams are stressed out and enable them to overcome their obstacles.

TRANSITIONING TEAMS:

You’re in the best position to learn from others’ mistakes, and pivot now. Empower your testers in the right ways. Don’t implement tools that are going to silo them off - try to find collaborative tools, and avoid tooling that requires a special skill set which the rest of the team won’t be able to understand or glean information from themselves.

SIGNOFF

Wooh! You’re still here? Glad you stuck it out to the end. We hope this report was as insightful to you as it was to us. Stay in touch via the mabl blog (mabl.com/blog) if you want to get a hold of the raw survey data yourself to do some digging on your own. Feel free to share your findings in the blog comments, or on Twitter (@mablhq).

Don’t forget to visit www.testingindevops.org for great community-contributed content from awesome testers like you!

Stay tuned for next year when we do one of these reports again so we can start seeing these trends move and transform over the years to come!

Until next time,
Chou Yang

Chou Yang had black hair and is wearing glasses and a gray sweater.

About the author:
Chou is an art student, turned engineer, turned marketer. She enjoys her days at mabl engaging with the testing community via the Ministry of Testing meetup group Boston, Twitter, and of course, the mabl blog. You can find her napping in random nooks in the mabl office, and at chou@mabl.com