A Tester Walks Into A Bar...

A tester walks into a bar. They order 1 beer, 0 beers, 9999 beers, and an orange - or so the joke is told. But how do testers handle scenarios with thousands of permutations? What’s the most efficient use of time and energy when faced with highly variable scenarios, and what do they look like in mabl?

When looking at a scenario with several thousand permutations, it’s key to define your goals before even developing a testing strategy: do we need full test coverage, or are we just looking to test the functionality of calling a permutation? Are there high value permutations we need to test? Some hypothetical cases: 

Search a sample

In the first hypothetical, we’re testing the search functionality of an e-commerce site. In this scenario, testing every permutation would be costly in terms of both time and energy. A likely approach would be to select a few key terms to test the scenario with and potentially rotate our test cases based on seasonal trends or best-selling products. In this example, we’re prioritizing efficiency by using a small sample to test functionality as well as a few prioritized items.

Prioritize the middle of the bell curve

Our second case is a bit more complex. We’re a consulting group testing a government application with several compliance pieces. A key requirement is a searchable county directory that can link users to county resources. To further complicate matters, this directory of counties contains over 200 counties. For this example, let’s consider two testing strategies, each of which are aimed at a specific goal.

Our first strategy is aimed at testing counties with the highest traffic with the goal of gaining feedback as quickly as possible. In this strategy, we’ll take advantage of mabl’s unlimited parallel test executions to create a test case that utilizes a data table with the top 50 counties with the highest web traffic. Each row of our data table will contain a variable with the value of the county’s name, which we will use as a search term for our application to locate the resources for that county. Next, our test will validate the results by asserting against the expected number of resources listed for that county. 

With this strategy we can focus our efforts on our highest priority counties and our results will be available very quickly thanks to running our tests in parallel. This strategy prioritizes speed and efficiency, which is easier to determine since we had clear-cut criteria (traffic rates) for determining which counties were most critical. Without data-driven testing, running tests for all 200 counties would have required more resources. With that said, depending on priorities, testing all 200 counties in parallel is not only possible, but an incredibly easy way to validate a test with hundreds of permutations through increased parallelization.

Let’s consider a second strategy that would test all 200 counties in a more cost effective manner. One approach we can take is to use a Javascript step in mabl to store a hardcoded array of 20 counties. Using a looping flow in our mabl test, we can iterate through this array and set a variable to the value of the array item at the index of our looping flow index. This step may look something like this:

After calling up the resources available to the county searched for from the array, we can validate the expected results and restart the loop with the next county while setting our validation steps to continue on failure. To expand our test coverage, we can also opt to create 10 copies of this test, each containing 1/10 of the counties to test within our hardcoded array. All 10 of these tests can be run in parallel to improve the efficiency of  our run time. This strategy will ensure that we’re able to test all 200 permutations of this test case. However, due to the length of time needed to iterate through all 20 loops in our test, this strategy is less time efficient compared to testing the top 50 counties.   

With the full-range of testing available on the mabl platform, it becomes critical to have data-driven strategies that can adapt to different goals. With our top 50 strategy, we prioritized efficiency; in the loop flow strategy, we prioritized test coverage.

Whether you’re testing a simple scenario with a few cases or a complex one with thousands of variations, there’s no shortage of ways for mabl to solve the problem available to creative testers. At the end of the day, when we have clear goals, our testing strategy is improved. At mabl, we’re focused on giving you the tools to embrace data-driven testing that maximizes your time and makes your product better.

See for yourself how data-driven automated tests work in mabl. Sign up for a free trial today!