Building a Secure Test Automation Experience

Last week mabl announced that we achieved SOC 2 compliance making it easier for our customer community to understand how we ensure that their test data is protected. A major part of this initiative includes the mabl desktop application. Launched in February this year, we made major strides in capabilities within our test automation platform by introducing API testing and mobile web testing, as well as improving testing speeds. In addition to these improvements, the mabl team focused on increasing user control over their data, making the automated testing process more secure. After all, testing is ideally integrated into the overall development process, and we want to ensure that quality teams have the support they need to meet the security requirements of their organizations. 

The Secure Testing Experience

A core component of maintaining a secure test automation solution is the secure testing experience in the mabl app. As we built the app as a better, more powerful testing experience compared to the Chrome extension, security was a priority to ensure a positive experience. 

The Rise of the Chrome Extension

The Chrome extension test recorder has been a staple of low-code testing. Nearly every vendor in the space has one, in addition to open-source alternatives like Selenium IDE.  

The reason is simple: robust test capture requires deep access to the actions that users take in the browser. A test automation solution needs access to DOM elements and events, file download operations, cookies, tabs, navigation actions, and web requests. For playback, the testing tool also needs to be able to simulate and recreate these things, as well as execute arbitrary JavaScript provided by the customer. Chrome extensions by default run inside the browser and can request broad permissions to read and modify data.

The Secure Test Automation Problem

Typically, extension developers can restrain permissions in two ways: restricting the depth of access by limiting what permissions it asks for, or restricting the breadth of access by limiting the sites the test automation solution can access. A common example are plugins designed only to work with Google Calendar.

However, the core capabilities that support user-friendly, robust test creation and playback also make test recording extensions fundamentally high-risk. Due to the factors discussed above, test recorders require deep access to the browsing experience, with few ways to limit the data they collect. Since extension permissions are requested globally, they need broad access and typically request access to all sites. In other words, the default installation settings of test recorder extensions allow it to access almost everything users do in the browser.  

That lack of nuance is a significant challenge to create a secure test automation framework without sacrificing functionality, one that mabl set out to solve as we built the low-code test automation experience from the ground up for the mabl app. 

Existing controls

The ecosystem provides some degree of protection against malicious or poorly behaving code. Reputable commercial vendors with solid privacy policies and popular, actively maintained open source solutions should only be capturing data when test recording is explicitly on, and even then the test recorder should only collect the data needed to recreate and maintain the test. Furthermore, the Chrome team routinely reviews extension submissions and has been raising their standards for privacy and security with the introduction of Manifest V3 by more thoroughly reviewing extensions that request broad permissions.

But even the best intentioned companies and well-run open source projects have security vulnerabilities. Good security practices would require that test automation solution providers restrict the extension’s access to only the essential data needed to do its job. But, as seen too often in real-world security practices, these ideal standards become untenable when applied organizationally.  

As an individual tester, there are a few options to limit the automated testing extension’s level of access. The most comprehensive would be to install it in a dedicated Chrome profile that’s only used for testing, which would effectively isolate your normal browsing from your testing activities. Chrome 80 also introduced the ability to explicitly configure which sites an extension can access. But even these solutions have drawbacks: it’s easy to confuse Chrome profiles and it’s possible to have both testing and non-testing use cases for individual websites, undermining any limitations placed on extension access.

But the biggest problem with these solutions is scalability, or lack thereof. QA leaders, particularly those at mature DevOps organizations with a culture of quality that have everyone contributing to software testing, can’t expect every end user to modify their browsing habits or explicitly configure restricted sites. Adding to the challenge is that the managed tools available for extensions are much more limited. While it’s possible to specify organization-wide allow or block lists for an extension, the most secure version of this list would require centralized maintenance of which sites are being tested, a major challenge and productivity blocker. Blocking access to sensitive sites also could impede usability as well, since the organization risks blocking the ability to test those sites using test accounts and data.  

A Better Solution: The mabl App

The fundamental problem is that Chrome extensions are designed as a way to augment the browsing experience. Low-code test automation tools are sizable software programs that require taking over the browsing experience completely, but only when explicitly asked to do so.  As the mabl team explored new ways to improve the secure testing experience, it became clear that a dedicated application that wraps the browser would be a better fit compared to an extension that sits inside the browser.

This model has a number of clear benefits. The most obvious is that a dedicated automated testing application completely removes the testing tool from the browser instances used for general browsing. This eliminates any concerns about appropriately sandboxing the test automation tool. Given how many organizations have policies limiting local filesystem storage of sensitive data and the amount of sensitive data accessed purely online, this is a significant improvement and enhances our ability to respect user privacy, a core component of SOC 2.  

Though desktop applications open up a new set of security risks by moving the test automation tool outside the browser sandbox, modern operating systems have a robust set of security controls around sandboxing and/or limiting the access of individual applications. Testers ultimately have much more control compared to the traditional Chrome extension model, again supporting mabl’s commitment to empowering testers in all aspects of software quality. Security and privacy are further enhanced since those security controls can be applied at the individual level or by a centrally managed IT team, depending on the general endpoint management policies of an organization. 

Experience secure testing with a free trial of the mabl app.