Testing is a mission-critical aspect of the software development lifecycle (SDLC). You’d be hard-pressed to find a software development professional who disagrees. Yet, for all the importance that testing has in the SDLC, there is a misconception among many about the difference between a testing tool and a testing system. Understanding the difference is important, particularly if you’re a decision-maker who is responsible for ensuring that your company’s software development teams have the resources they need to implement software testing that is effective and reliable.

Hence, the purpose of this article is to explain the difference between a testing tool and a testing system. 

A Tool is Not A System

A testing tool is intended to test in a particular area of concern. For instance, there are tools intended for executing unit tests. Examples are JUnit and TestNG for unit testing Java code, Mocha for Node.js programming environments and, MSTest and NUnit for .NET. These are but a few of many tools available for the wide variety of programming languages in use today.

There are also tools intended for UI testing such as Selenium and mabl. When performance testing, you can use a tool such as the very popular open-source JMeter. Also, there are commercial products such as NeoLoad, Rational Performance Tester, and Load Ninja. There are also tools for API testing, security testing, and regression testing. The list is long. But the important thing to understand is that each tool has a specific purpose. This is good - a tool that tries to be everything to everybody rarely does one thing well.

While a tool such as Java’s TestNG or Node.js’s Mocha can be used for unit testing, they can also be used as a test runner. For example, you can use them as test runners to execute Selenium UI tests. In this case, you’ll have the TestNG or Mocha code use the Selenium WebDriver component to navigate through a website, locate elements in a web page and run assertions and perform actions against the element such a mouse click or type text into a textbox. The test will check the results of an action and make assertions against those results.

Typically these tests executing under a test runner are created in the programming language's particular testing framework, in this case, Java for TestNG or JavaScript for Mocha. Creating the code to execute under the test runner can be a laborious undertaking. Some tools will, such as the standalone Selenium IDE will allow you to record UI tests and then export recorded actions as scripts that can be run in a particular test runner. For example, Figure 1 below shows the test export capabilities of the standalone Selenium IDE.

Figure 1: A tool such as the Selenium IDE supports exporting recorded UI tests to code.

A tool such as mabl takes a more integrated approach to UI test execution, and covers the test creation process and test execution process by running tests from within a standalone environment. There’s no need to use a test runner, test framework, or testing library in conjunction with mabl. The built-in testing environment takes care of it all.

Regardless of whether you’re using a testing tool stack or a standalone testing tool, it takes more to be build a testing system. Testing systems go beyond the scope of any particular tool. 

Understanding Testing Systems

A system is a group of parts that interact in concert to form a unified whole. A system has an identifiable purpose. For example, the purpose of a school system is to educate students. The purpose manufacturing system is to produce one or many end products. In turn, the purpose of a testing system is to ensure that features and functions within the scope of the software’s entire domain operate to specified expectations.

Typically a testing system is made of parts that test specific aspects of the software under consideration. However, unlike a testing tool, which is limited in scope, a testing system encompasses all the testing that takes place within the SDLC. Thus a testing system needs to support all aspects of software testing throughout the SDLC in terms of execution, data collection, and reporting.

Features of a Testing System

First and foremost, a testing system needs to be able to control testing workflows. This means that the system can execute tests according to a set of predefined events. For example, when new code is committed to a source control repository, or when a new or updated component is ready to be added to an existing application. In the case of the former, the testing system will execute the unit tests that get checked in the source control management system along with the new code. In the case of the latter -- adding new components -- the testing system will be able to execute integration tests. (See Figure 2, below)

Figure 2: A testing system needs to support all testing throughout the SDLC in terms of execution, data collection, and reporting.

Also, a testing system can be configured to execute tests around a given concern at a predefined schedule, for example, executing integration and API tests once a day or once an hour. Running tests periodically is particularly useful when trying to identify side effects that can be caused when components share state and a state caused by one component corrupts the functioning of another.

In addition to being able to control the execution of tests within the testing system’s workflow, the testing system must be able to handle test failures. This means sending out notification events upon failure as well as implementing rollback activity.

Finally, a testing system needs to be able to collect test result data from all testing activity and aggregate that data into reports that target the needs of the various parties involved in creating the software. This includes not only test practitioners but developers, project managers, and project sponsors as well.

A testing system's requirements go well beyond the scope of any single testing tool. For example, a unit test will report when things go wrong, but a unit test usually will not contain the logic that will perform rollback behavior or notify interested parties about test success or failure. This work is done by the testing system.

CI/CD Integration

Another important aspect of a testing system is that it’s usually part of an automated Continuous Integration/Continuous Delivery (CI/CD) process. You can think of the CI/CD process as the high-level controller of a testing system.

Let's consider the more popular framework commonly used for CI/CD automation: Jenkins. One of the things that makes Jenkins popular is that a number of testing tools integrate with Jenkins through its plugin ecosystem. Just about all the major testing products publish dedicated Jenkins plugins, mabl included.

The Jenkins mabl plugin integrates well with the mabl cloud testing environment. You can configure the plugin to execute UI and API tests associated with a given run time environment such as DEV, STAGING, or PROD. Also, you can have the plugin take advantage of mabl’s labeling feature to run tests according to a finer grain of execution.

The important thing to understand about CI/CD and testing tools is that a comprehensive testing system uses CI/CD as the means to organize the disparate concerns of each testing tool into a unified, purposeful whole. Any of the well-known CI/CD platforms will provide the wherewithal to execute testing throughout the SDLC. Also, they’ll provide the means by which to capture test results from all manner of testing sessions. Once the given CI/CD platform captures test data and stores it in common formats such as XML, JSON or YAML, that data can be aggregated into meaningful reports using tools such as  Splunk, Kibana, or Grafana to name a few. In the case of a testing system like mabl, test data can still be exported into formats like BigQuery, or simply viewed within the mabl system dashboards.

Putting it All Together

Taking a comprehensive approach to testing is critical for any company developing software for professional use. Whether the company is a startup doing open-source work and one with a deep legacy of commercial activity, those in the know understand that testing is essential.

However, just using testing tools alone is not enough. A comprehensive testing system that unifies all aspects of software testing throughout the entire software development lifecycle is required. Or to put it another way, a software development business without a unified, effective testing system... isn't.

The key features of an effective testing system are:

1. All testing activity throughout the SDLC is controlled by the testing system and observable to the testing system.

2. The testing system must be able to accommodate the failure of any test anywhere in the SDLC. Remember, for as many tests that will pass there will be just as many that will fail. Failing code cannot be promoted onward. The failure must be accommodated by the system.

3. The testing system needs to be able to collect test results everywhere from within the SDLC and make is so those results can be presented in meaningful reports to the variety of professionals on the software development team.

Moving beyond a tools centric approach in testing to one that takes a system-oriented view can be hard work but it’s work worth doing because quality counts. History has shown that although you can’t test quality into a system, a comprehensive testing system is, in the long-term, the easiest, most effective way to ensure that companies meet the standard of quality required to deliver software that matters. As history has shown, successful companies do indeed make software that matters.