Next Generation Codeless Automated Testing Tools - Alternatives to Selenium
Testing has long been one of the most thankless tasks in the world of development. Doing it manually can result in hours of repeating the same tasks over and over again trying to find the bugs that developers hope don't exist. With the 90-90 rule in full-effect, it seems like it takes as long to test software as it does to create it. The larger the site gets, the greater the burden of testing it becomes. Worse than that, it creates a bottleneck in the development process. No sites can be moved to production before this process is completed (unless, of course, you don’t care about quality). Test automation solutions came along as a response to this.
Selenium and Selenium IDE - why do we need an alternative?
Selenium is a powerful framework that enables developers to automate all the tests that they want. However, it's user hostile for someone without a coding background. One needs to be pretty much a full-fledged developer to use it, but many organizations tend to tread down the dangerous path of hiring developers or interns without testing experience to create test scripts.
To be fair, there's Selenium IDE. With Selenium IDE, all one needs to do is fire up the Selenium plugin in the browser and start recording the user journey. You can play the test back at any time to see the test run live. You can also manually add steps, assert the existence of elements, and use a number of other nifty features.
However, reality being what it is, nothing is ever as we plan it (otherwise why would need to be testing in the first place?). The love-hate relationship with Selenium is a real thing. It has its merits, but Selenium tests, whether recorded or coded, fail a lot for all sorts of reasons; a slight slowdown in the internet connection or an errant mouse movement while recording the test cause strange test results. Tweaking the tests often break them, and you'll need to start from scratch.
They're also at risk of becoming quickly outdated in agile and DevOps software development teams where changes occur frequently to the app under test. Almost any change on a site requires re-recording or rewriting of the entire test. The larger the project, the worse it gets. Maintaining Selenium scripts quickly becomes no more efficient than manually running through tests on your own. If not done right (which is easily done!), Selenium can result in low-confidence tests and become a huge burden.
To summarize, Selenium has a lot of the following problems:
- Tests are unstable
- Tests are fragile to app changes
- It is far too complex for beginner users
- Being a tester becomes a full-time highly skilled job
- The specialized skill set silos testers from the rest of the development team
- It creates a new expense items for development firms
- Qualified testers can be hard to find
Record and Playback Tools - Yes, we know, they suck.
So, of course, many companies saw an opportunity here. First generation record-and-playback tools came along, but suffered from some of the same problems as Selenium IDE. Record and playback tools were still fragile and broke with minor app changes. Record-and-play tests aren't modular, which means each test case with similar flows in it have to be modified across the board. This generation of record-and-play quickly got a bad reputation. They were still not scalable. Cue the next generation.
Next Generation Automated Testing Tools - codeless automation that doesn't suck
So a new demand arises: a solution that can harness more of the power of Selenium that:
- Doesn't require coding
- Runs stably
- Adapts to code changes in the app
But this generation of test automation tools has gone beyond these 3 requirements, and incorporated many delighters like cross-browser testing, advanced features like email verification, and automated visual regression insights from your test runs.
Joe Colantonio has talked about this third wave of test automation tools that utilize up-and-coming technologies like machine learning. The point of this article is to look over some of these well-known codeless testing tools on the market and to evaluate them based on this list of features that the next generation of automated testing tools need by Angie Jones. Many of the items in this article have already been addressed by most of these major testing tools on the market. However there are a number of features that may not be as common, and each tool handles these a little differently.
It is important to state that this article is not a review of these tools, but instead a brief overview of some of their capabilities. Unless stated, all features discussed are available innately or in the tool's UI, accessible without needing to code.
The following tools will be examined:
A feature-by-feature assessment
Feel free to skip to sections you care about the most. We will be comparing the following features:
- Smart element locators
- Conditional waiting
- Control structures
- Easy assertions
- Reusable steps
- Cross browser support
- Custom code injection
- Integration in the development pipeline
Smart Element Locators
As we already discussed, one of the flaws in first generation record and playback tools was tied to the fact that even minor modifications to code causes test scripts to break. One factor that would improve these tools is the ability to check for more than one element locator for every element interacted with in the test script. If element locators in the code change or no longer exists, the test can fall back on other locators, meaning scripts would be more stable and require less modification in the event of code changes.
mabl searches for closely matching instances of a specific element instead of looking for it at its last known XPath. CSS selectors and dozens of other identifiers are derived from the website automatically. mabl will run the test using the closest matching element, and rerun the test with all closely matching elements if there are many likely candidates.
TestCraft uses what they refer to as “smart binding.” This enables it to assign tests to multiple elements at the same time. TestCraft provides a few ways to doing this. This first is technical, which captures multiple attributes at once, such as IDs, and contextual, which captures an element's position, text value, or size. It also allows elements to be rebound during a run, to match changes that have occurred in the code. This is handled manually.
A common issue with first generation record and playback tools is that any delays or unexpected problems with a browser or internet speeds could cause tests to immediately break. What was necessary was to be able to create conditional pauses, and to allow scripts to only proceed if a condition is returned as true, such as a page finishes loading or an API call has completed.
mabl handles this automatically with built-in heuristics that assign different wait times for different elements so that if a specific element usually takes longer to load, mabl will wait the extra few seconds for it. It is also possible to to add “wait” steps to specify an exact amount of time to pause before attempting the next action.
TestCraft has some built-in “if” statements. One can say that the test should wait for an entire page load before proceeding. It also has functionality to check to make sure all AJAX requests and CSS animations have completed as well. TestCraft also allows setting a configurable amount of time for waits.
EndTest handles most of this process automatically. It does not require adding wait steps; the application identifies whether to wait or not. It should automatically detect whether or not, for instance, a page load has occurred. It allows setting max wait times, and will also allow entering a pause into a test.
One of the advantages of coded automation frameworks was the ability to create control structures such as loops and other conditional clauses. Being able to loop through many different conditions is a very valuable factor for making tests scalable. This is one area where codeless tools have lagged behind coded frameworks. While most have advanced to the state where they can allow simple conditional checks, with a few exceptions, looping is not something that is typically available as of this point, without some additional coding.
TestCraft has a number of built in codeless control structures, such as “if” statements, and has a reusable test flows structure. It also allows the running of concurrent tests with different datasets.
EndTest has a built-in “if" tool, and they can be nested. It currently does not have looping but it is under development.
While this seems like a relatively easy task to accomplish, many first-generation codeless tools did not allow one to easily assert the existence of an element. This is important because being sure that an element has actually loaded is an important part of any automated test. While it could be done by writing custom code, this could slow down automated testing considerably. For this reason, this is a fairly crucial need for codeless automated testing tools and it should be a required feature of next generation codeless tools.
mabl has tens of easy assertions built in for any element: validating the inner text, CSS properties, and advanced assertions such as cookie and URL validation.
TestCraft has a number of basic assertions like validation its presence as one of its core features.
Adding assertions is part of the basic functionality of Endtest. One can assert an element is present, visible, or not visible, text matching, and more.
As anyone who has done any testing can testify, there are many functions that require repeating. Having to run through a segment of a test each time can often seem like a time-waster. There have been a number of ways that this has been handled with coded tools, ranging from the creation of libraries for a project, up to a fully object-oriented approach where individual functions common through multiple applications can be drawn upon to create much more efficient testing processes.
For codeless tools to be truly worthwhile, they should be able to provide the ability to reuse common functions, such as logins without having to record it each time. If these functions need to be updated, they should be updatable globally.
mabl allows recording sequences which can be bundled into “reusable flows.” These can be stored as a library and can be inserted into either tests within the same application, or can be used again in other tests.
TestIM has an ability to create small modules which can be reused, called “shared steps” which can be inserted into tests. They can also be cloned so modifications can be made if necessary for individual tests.
TestCraft is a little different than some of the other testing tools described here. Typically tests are constructed manually in a flowchart instead of recording them (though there is a recording function). As a result, the modular approach is built into it, and individual nodes or modules can be reused for new flows.
With EndTest, tests can be divided up into “test cases,” such as “logging in” or “searching” which can then be reused by importing a test case into a “suite” which can be used in multiple applications.
GhostInspector uses “custom variables” to perform functions which are not already built-in to the application. These can be simple variables, but also can be full functions (or objects) which can be modularized and be reused.
Cross Browser Support
Many of the first generation testing tools allowed testing in only one browser, typically Chrome or Firefox. However, as we know, users use a wide variety of browsers, and unfortunately, not all of them behave the same way. A common headache in front-end web development involves dealing with variations between browsers.. Whether you are dealing with Chrome or Firefox or Edge, Or IE this is necessary. This also gets more complicated if we have to check against older versions. (Maybe some of your target market is using IE6? Anyone remember the old AOL browser? <shudder/>)
mabl tests creation is limited to Chrome, however the tests are platform-agnostic, so the recorded tests are created once but can run in Firefox, Internet Explorer, and Safari in parallel or in different test plans. You can configure the display size for responsive web testing.
Running cross-browser tests in TestIM requires using the CLI, which requires custom setting up of node.js. However once this is set up, the CLI is available through their interface. The scheduler also appears to allow configuring for different browsers.
TestCraft is platform agnostic and allows running on multiple platforms simultaneously and will run tests in Chrome, Firefox, and IE.
EndTest shows support for Chrome, Firefox, IE 11, and Edge, as well as various mobile devices.
Ghost Inspector allows running tests in Chrome and Firefox if set up in a Linux environment.
Of course when you are running many tests, it would be a lot easier to see a high-level view of whether or not test runs have succeeded. Ideally, when a test fails, it should be easy to see exactly where the failure occurred without having to dig through log files or to have run the test again manually. On top of this, sometimes failures occur under certain circumstances that can't always be easily reproducible with a manual test.
With a good reporting system, ideally we should be able to see all information regarding a failure so as to understand what circumstance caused the problem. Some tools provide reports on which tests failed and where, and some even go as far as to provide visual evidence of the failures in action, so as to not need to run these again manually.
TestIM has some very high level managerial reporting tools, and it reports common failures. This requires digging into each test to see the failure results. This may be sufficient for most users.
TestCraft has some nice features, including detailed reports of every run, and it also records screenshots and and videos of moments when they failed, so it's possible to visualize the problem.
EndTest only provides a record with the results of a test, and displays pass/fail indicators for each run.
GhostInspector provides reports on every test run. It will provide a screenshot which compares the results the last passing test. As a result, identifying any differences is quite easy.
Custom code injection
Frameworks such as Selenium are often language-agnostic. Granted, while this is something that might only benefit the most advanced users, it could be helpful if some of the codeless tools would allow other languages.
Integration in the development pipeline
As most web development teams work in an Agile and/or DevOps environment, testing has typically been integrated into the full development process. As a result, it is typically necessary to be able to connect to different tools which will enable tests to run throughout the entire workflow and by different teams.
With coded frameworks, typically custom coding is needed to integrate into existing processes. For automated tests to be truly part of the development lifecycle, it needs to have some integrations with other tools, or include a few basic triggers that can be called via API.
mabl has integrations with several CI/CD platforms, and for non-natively supported platforms, there's an API that you can use to trigger tests and send test results back from.
With TestIM it is unclear. While it has internal tools for running tests on schedules, all information about APIs seems to suggest it is calling external APIs, and not providing triggers. That said, TestIM has an integration with another product, Testrail, which will provide most of the necessary integrations.
TestCraft provides full integrations into CI/CD workflow, with many integrations for the entire development cycle. Integrations include Jenkins, Slack, Zephyr, Jira, Gitlab, and more.
EndTest integrates with various DevOps tools, such as Jenkins, Github, Bitbucket, and a few other pipeline tools.
GhostInspector provides integrations with AWS Code Pipeline, Bitbucket, Docker, Gitlab, Jenkins, and many other tools.
While there is still no perfect replacement for the power of fully coded tests, codeless automated testing software has come a long way since the first record-and-playback generation of tools. Many of the complaints about the limitations of these original tools have been addressed, and new changes are occurring regularly.
As seen above in the selections we examined, almost all have addressed some of the basic needs, such as easy assertions, a few basic conditional operators and the ability to integrate with other development pipeline tools. The speed at which software companies have adopted many of the needs for testers gives us reason for optimism.