Deciding What Tests to Automate

Quality engineering is a set of practices that allows software development teams to produce high quality software in faster, more iterative sprints, including planning, testing, analysis, and monitoring throughout the DevOps pipeline. A key component of this is an effective software testing strategy that uses both manual and automated testing to ensure quality throughout the software development lifecycle. But how can QA teams, especially those with limited experience in test automation, decide what tests to automate? 

Understanding the Strengths of Manual Testing and Test Automation

Choosing between test automation and manual testing is often depicted as an all-or-nothing decision, a misconception that often leaves quality assurance teams gun-shy about attempting automated testing. But for most organizations, a combination of the two is often the best approach for faster, more quality-driven development. The challenge is building a software testing strategy that harnesses the strengths of each technique for maximum impact. Too much manual testing and quality teams risk spending too many people hours on repetitive tasks, potentially slowing down development. Conversely, if they focus too much on automated testing, they risk overlooking errors outside the capabilities of test automation. 

Manual Testing: Boldly Going Where No Test Has Gone Before 

Manual testing is the tried-and-true software quality strategy for a reason: manual testing is crucial to building product knowledge, identifying bugs that only occur in specific circumstances, and understanding the true user experience. QA specialists that focus primarily or exclusively on manual testing provide valuable feedback on how the product looks and feels to users that may be navigating it for the first time. 

As quality engineering teams build out a hybrid manual and automated testing strategy, manual testing serves as the foundation for expanding testing and understanding the customer perspective. Software testers are able to use their product expertise to build test plans for new features and collaborate with developers to solve tricky software bugs. 

Test Automation: Confidence, Reliability, and Coverage 

Automated testing, meanwhile, provides consistent feedback on product quality for continuous improvement and faster development. In the early stages of quality engineering, QA teams are able to automate critical end-to-end and regression tests that take up much of software testers’ time, allowing them to shift focus to expanding their testing strategy and shifting testing to the left. As an organization’s quality engineering practice matures, test automation ensures that the product continues to work as expected, monitors application performance, and provides rapid feedback on code early in the software development lifecycle. 

Determining What Tests to Automate 

Like most quality quests, the process of deciding what tests to automate starts with a series of questions. Rather than plan their test automation strategy around the capabilities of their testing tool, these questions help guide software development organizations develop automated tests that fit their workflows, product, and customers. By focusing on the end goal -  better software quality and a delightful user experience - QA teams can ensure that they’re making the most of their test automation solution. 

What Tests Take the Most Time?

DataRobot Director of Engineering Meghan Elledge said it best when she described her approach to introducing automation: “If I have to do something twice, I’m already thinking about how to automate the third time.” Many quality assurance teams can start automating tests with the same mindset. If a software development organization is dedicating a large portion of testing hours to repetitive tests like end-to-end testing or regression testing, those tests are prime opportunities for automation. 

Regression testing or end-to-end tests for the most popular user journeys are ideal for test automation because they primarily serve to assert that existing features are working. Assuming that the basic application architecture doesn’t change, those tests are likely to remain relatively stable, minimizing possible maintenance. This allows QA teams to save the maximum amount of time and show ROI fairly quickly.

Where Can Test Coverage Be Improved?

Test coverage is a tried-and-true measure of success for any software testing strategy, and increasingly, for an organization’s entire quality engineering practice. Accurate test coverage is essential for ensuring that a team’s testing strategy reflects the true customer experience and matches how the application under test is built. 

QA teams evaluating where to implement test automation can start by understanding test coverage over their most popular user journeys. In the case of mabl customer Sensormatic, automating email and PDF testing played a large role in allowing them to increase automated test coverage from 40% to 80%. Considering that 78% of marketing departments have seen an increase in email engagement in the last 12 months, and 99% of consumers check their email every day, expanding test coverage to include popular customer pathways like email is an excellent starting point for test automation.  

How Can Software Testing Better Reflect The Product?

There are approximately 200 million APIs currently in use today, a number that’s growing rapidly as more developers turn to third party software to build and manage their applications. Yet there are few convenient ways to manually test APIs. At best, QA specialists can navigate through popular customer journeys and hope that they catch broken APIs in the process. But considering how unpredictable many APIs are - Salesforce, for example, is notorious for issuing updates without warning - relying solely on manual testing isn’t enough to manage risk in an API-dependent world. Taking an inventory of how an organization builds their product, such as APIs, other integrations, and open-source components, gives software testing teams useful insights in where their software testing strategy can evolve with test automation.

Helping Quality Engineering Teams Optimize Testing

Finding a healthy balance between manual testing and automated testing enables quality engineering teams to expand their software testing strategy, closely monitor the customer experience, and support faster software development. Though many QA teams are just starting their test automation efforts, understanding the complementary roles of manual testing and test automation provides a solid foundation for auditing an existing software testing strategy and identifying opportunities for automation to cover more of the customer journey or help QA teams better reflect the evolution of the product. 

Start building a high-impact software testing strategy with mabl's 14 day free trial and see what your team can achieve with low-code test automation.