Top 5 Challenges of Testing Automation

Picture of Chou Yang
byChou Yang

When done properly, testing automation can be a highly effective productivity booster, and a quality enhancer for your product pipeline or system development projects. But it can be difficult to apply best practices when a team is just starting this transformation.

 

We'll present the most common challenges that face teams as they pursue automation efforts—and also how to overcome these challenges.

 

Screen Shot 2018-04-09 at 10.05.37 AM

 

Challenge #1 — Effective Team Collaboration

Collaboration is a challenge for any product development group—developers, test automation engineers, and conventional testers alike. The problem is more acute with test automation because it requires considerably more planning, foresight, communication, and collaboration, because automation fully depends on historical testing data, tester experience, and at least one proof-of-concept. Testers need to work well with each other, developers, and customers to get all these pieces together.

How to overcome it:

Since testers will be central in communicating the value of testing automation to senior management, each automation tester should be in close contact with manual testers, business stakeholders, developers, and system architects. To be successful, it’s vital that most of the team commits to active and continuous involvement because of all the moving pieces coming in from each department involved. 

You'll also need management buy-in prior to automating your tests. This can only be done sensibly in a collaborative environment in which its obvious to management that team members communicate fluidly, deliver timely test results, with minimal risk. So make sure to document as you go so you can extrapolate metrics, and provide updates when management calls for it.

 

Challenge #2 — Deciding on the best testing approach

When possible, it’s actually best to prioritize meeting this challenge before buying new automation tools (which we discuss in the next section). Think carefully on this one. You might acquire good tools—but which tests should you automate first, which ones should you leave as manual tests, and how will you go about doing it all? The tools will not give you insight into what your team should automate.

 

Some next-generation AI tools, such as mabl, can help you automate many UI test cases and many other cases that could only be tested manually. Be aware that you won’t be able to automate all of your test cases, particularly in areas such as compatibility or recovery.

 

How to overcome it:

Which test cases are important to you? Before you begin to think about automation, it’s important to define a set of criteria that each test case must satisfy to be a candidate for automatization. It’s very important to focus on areas that contain unknowns—which means higher risk to product success. It’s also important to give serious attention to complex functional areas that are especially useful or interesting to customers.

 

Risk-focused testing ascribes a higher priority to testing the elements that are most likely to fail—and hold the potential for the most serious negative consequences in the event of a failure in the wild. As you assess the level of risk, it’s critical to keep the following in mind:

 

• Financial impact of a potential failure.
• Probability of failure for each vulnerability (get developer input on these).
• Service-level agreements with clients and customers.
• Human and equipment safety issues.

 

Agile development methodologies have become pervasive throughout the industry, and this means that many software applications contain significant changes with each new product iteration—which translates directly into testing volatility. This is a challenge in and of itself: how to design and implement automation into test suites that will readily and correctly identify these app changes and then quickly update the tests with modest levels of maintenance effort? Ideally, a test automation solution should automatically detect such issues, perform the updates, and revalidate tests with little or no human intervention.

 

 

AutomateTestsForTheDevOpsWorld


Challenge #3 — Evaluating and choosing good tools

Many record and play testing tools are available, all trying to alleviate the same problem of automating manual tests, but have proven to be harmful to team's productivity levels. This makes it difficult to sift through the chaff and find tools that will actually improve software testing and meet real needs. It’s important to steer clear of hype, do sufficient research, and carefully evaluate fact-based proposals of reputable tool vendors.

 

Tool evaluation can seem unachievable for various reasons:

• Lack of expertise to exploit the capabilities of a specific tool.
• Indefinite knowledge on whether a tool exists to meet specific needs.
• Misconceptions lead to desire for a tool that isn’t available.
• A favorite tool doesn’t seem to provide full test coverage.
• Tool costs exceed the team budget.

 

The challenge of choosing good automation tools is a serious problem since, according to the Cap Gemini World Quality Report, over 50% of IT leaders report that their organizations lack suitable automation tools for their  testing teams.

 

How to overcome it:

Though it requires effort from the most of the team, a good assessment begins with clearly outlining a set of tool requirements criteria. These definitions should correspond well with user acceptance criteria for each of your products, and also reflect the input from professionals who have considerable experience in using the tools. You may be able to find these experts at conferences or smaller-scale networking events, so it may worth investing in sending your teams to events as needed. Free webinars are often hosted by product vendors to demo their offerings without the pressure of being in a sales pipeline. Encourage your testing team to register to attend these webinars live, as the live Q&A portion can be the most valuable piece of a webinar.

 

If the tool that your team wants is beyond the reach of your budget, it may be worth the effort to prepare a cost-benefit analysis and make your case to management. Demonstrate how implementing this tool will minimize risk and improve release quality, using a pilot scenario that scales.

 

Challenge #4 — Training or acquiring skilled testers

It is a misconception that testing automation can be easily done by any tester. Likewise, many falsely believe that any developer can automate tests. Focusing, designing, configuring, and implementing test automation requires specific skills and time to build experience.

 

How to overcome it:

When filling this role, you need to look for a specific skill set beyond technical chops. Testers sit at the intersection between your developers, managers, and customers, so having the ability to articulate ideas from one group to another is a must. Being in the tech field, testers must be willing to stay up to date with the latest technologies and development methodologies to stay in touch with what the development team may be doing or moving towards. At the same time, they must also understand how to keep a holistic view of what the software in development is trying to achieve from a business perspective, and keep test cases relevant to those goals. Automation engineers, are admittedly, one of the toughest tech roles to hire for, but the effort to find the right hire will be significantly rewarded.

 

Challenge #5 — Significant capital investment 

In nearly all cases, the initial phase of test automation comes with some costs, since it requires analysis, design, and planning for the implementation. You’ll need to account for all licensing and operating costs—including hardware and software. Even if the choice is made to go with free or low-cost open-source automation tools, it will be important to devote some effort to training and maintenance.

 

How to overcome it:

Bite the bullet. Automating is not a movement that should be approached haphazardly. When done properly, the effort and cost can pay steady dividends well into the future. Test automation can deliver significant gains in quality, accuracy, and productivity. The business case can be built solidly when the team comes to realize that avoiding all the negative impact and cost of a single major defect can directly compensate for the all of the tangible and intangible costs of pursuing automation.

 

Be informed, not afraid.

These challenges are not the only ones for tackling test automation, but they are the most common major challenges. If a team embarks on the test automation journey without planning ahead, the automation project is likely to end in a catastrophic failure. 

 

Still, automation is necessary to keep up with the modern end-user and achieving customer success. The software world is now customer-driven, and the end-user experience directly affects revenue, growth, and trickles down to even internal company dynamics, such as talent acquisition and employee satisfaction.

 

So be plan ahead for the challenges, and keep an eye on the silver lining in the horizon, knowing it will all be worth it.

 

AutomateTestsForTheDevOpsWorld