Back to the blog

Reshaping the Testing Triangle: Making UX the Guiding Light to Testing

Picture of Chou Yang
byChou Yang

End-to-end test automation has been generally avoided for what once were good reasons. Fortunately today’s technology allows us to automate end-to-end tests effectively and without creating too much overhead. 

End-to-end testing vs. Unit testing

Automating end-to-end tests is a tricky business. Many leading practitioners over the years have advised against doing ANY end-to-end test automation. They recommend doing a good job of automation at the lower levels, and do end-to-end testing manually once and trust your lower level tests to protect against regressions.

As a result of this, we often see the test automation triangle pictured below. The shape of the test automation triangle is predicated on traditional associations about the cost of troubleshooting test failures and feedback cycles at each level. Traditionally, the closer you were to the bottom, the easier it was to find the root cause of the issue. Therefore, they recommend that you have a 70/20/10% split between unit, integration, and end-to-end tests, respectively.

The traditional testing triangle

But no matter how much testing you do at lower levels of granularity (unit, integration, etc.), end-to-end tests are the only thing that brings together all the pieces and configurations while focusing on the end-user functionality and experience.

No matter how much granular testing and tracing is done at the unit and integration level, the result of foregoing end-to-end testing is spending less time on the user experience and more time testing how we hope the application works (happy path testing). We’re also losing the ability to surface problems in a way that anyone - non-technical QA, customer success, business leaders - could understand. These team members may not be waist-deep in building the actual product, but they are intimate with your customer's pain points and wants and can provide truly insightful feedback.

Another thing to note is that granular testing is becoming less and less of the whole picture. Today's apps have incredibly feature-rich UIs. They're becoming more and more decoupled and less monolithic. Cindy Sridharan makes it clear in her story about distributed tracing. The service graph she draws below depicts dependencies well, but when it comes down to debugging a scenario where a specific service is experiencing issues, such a granular view isn’t particularly useful. If there is a slowdown with the front page, there are many dependencies you’d need to start digging into to find the root cause. Which microservices are not meeting SLAs? Which integration points are failing?

Source: https://medium.com/@copyconstruct/distributed-tracing-weve-been-doing-it-wrong-39fc92a857df

Putting more focus on unit or integration testing puts us into the same scenario. We lack test coverage from the user's perspective. It becomes harder to troubleshoot issues when tracing issues from the user-experience point of view. 

So the case to be made is – does today’s technology allow us to automate end-to-end tests effectively and without causing too much overhead?Well, there’s a new wave of test automation solutions that are changing the shape of the automated testing triangle. And mabl’s leading it.

 

Times have changed: automation is here to stay

There is a class of manual tests that are reserved for manual execution because they were traditionally too difficult to automate. It wasn’t because they really require human observational powers and reasoning, but that automation can no longer be avoided as development teams are moving towards shipping multiple times a day. Manual end-to-end testing alone isn't keeping up.

So the questions we’ve asked are: Why avoid automating end-to-end tests to avoid the problems with it? The end-user experience results in customer happiness and revenue, so why not fix the problems with end-to-end test automation instead?

Creating reliable end-to-end tests with mabl’s unified SaaS platform is easy and removes the constraints that suggest the end-to-end tests should be extra credit. The steep learning curve of coding end-to-end tests have been removed with an intuitive UI that allows you to create automated tests by simply using your app just like your users would. Using big data and machine learning, mabl provides comprehensive test output, logging everything that happened before, during, and after a test failure, so failures are easy to identify and resolve. The reputation of UI-based tests being flaky or unreliable is not a problem with mabl's AI-driven auto-healing. As your web UI evolves with development, your tests learn and automatically evolve along with it. 

By eliminating the cost and difficulty of end-to-end test automation, mabl broadens the top of the testing pyramid with the aim of making the end-user experience the guiding light to how we test software.

The user-centric model of testing

End-to-end testing has become stigmatized because of reasons that simply no longer exist with the innovation we’ve seen in test automation offerings in recent years. End-to-end tests are way too valuable to avoid doing because they best align with the user experience.

If you're up against the recurring argument that end-to-end testing is too expensive… the cost of not doing it well is easily calculated with poor app experience resulting in customers that don’t return or never become customers to begin with. The solution is intelligent test automation, and the ROI is clear.

“Using hiring and traditional automation tools like Selenium, it would have taken two years and $240K to accomplish what mabl helped ITS do in just four months at 80% cost savings.” -Samar Khan, VP of Software Development and DevOps, ITS

Create a free account of mabl today at app.mabl.com/signup.

Back to the blog