Homegrown testing framework strained by complex product and test maintenance
Faster delivery without risking quality
Testing democratized for 45+ team members
Team is executing 80K test runs per month
Priceline is the home of modern travel experiences. Whether it’s finding the perfect place to stay for a much-needed getaway, snagging a reservation at the hottest spot in town, or taking to the open road in a rental car, Priceline can get you there.
Behind these seamless travel booking experiences, however, is a complex platform delivering highly customized user experiences.
When Test Automation Architect Antony Robertson joined Priceline in 2015, their software testing strategy was 100% manual. Regression testing, feature (A/B) testing, web/API testing, and end-to-end testing were all performed manually by a dedicated team of QA professionals. Between product complexity and rapid-fire delivery cycles, the team simply didn’t have the bandwidth to build or buy a test automation tool that would have a steep learning curve and require ongoing maintenance. Yet manual testing was increasingly proving to be too time-intensive and ineffective for scaling quality across dynamic user experiences.
Like many QA teams, Antony solved these bottlenecks by building his own test automation framework. Fondly known as Autobot, this framework went through several major transformations from 2017 to 2022, with each iteration introducing a greater range of integrations, functionalities, and abstraction layers that democratized automated testing across the Priceline team.
The maturation of Antony's testing framework eventually triggered a paradigm shift, a Prometheus moment that convinced the Priceline team that automation could shift their focus from mundane, repetitive tasks to the rigorous evaluation of edge cases—those issues most valuable and yet most difficult to test.
By the time Antony built Autobot V3 in 2021, his testing tool featured APIs to trigger tests via the UI, GitHub Actions to trigger tests, and APIs to return results to everyone's preferred platform (such as Slack or Splunk). Automated testing was fully integrated into CI/CD pipelines, and more importantly, the entire Priceline organization was bought into automated testing.
But managing quality for an organization like Priceline came with an even broader set of challenges, especially since the product itself had evolved significantly in the years it took to build Autobot. As Antony explained it:
"We had a really nice platform. But seven-eight years later, our products had matured. It's now very bespoke. It has a lot of inertia behind it. But people are always looking for what's next. How can we do what we're doing already? Then, by reducing or removing the friction points that we have with the challenges, the maintenance, security vulnerabilities that pop up that we need to address. I joke with the mabl team that I was 80% there before mabl, but the last 20% is really costly.”
Given their long history with Autobot, Antony and the Priceline team had a precise set of requirements for their next test automation tool, including ease-of-use for developers and manual testers alike, API testing, reusable test steps, CI/CD integrations, a Slack integration, reduced test maintenance, the ability to track and manage tests results, and dashboards for data visualization.
Some of these considerations were purely practical, after all, why should Priceline invest in a test automation tool that offered fewer integrations than Autobot. But some, like the ease-of-use requirement and the need to reduce test maintenance were essential for maintaining momentum for the culture of quality Antony had painstakingly worked to build since 2015.
Mabl's low-code user interface, AI-powered auto-healing, range of DevOps tool integrations, and unified platform with API testing, browser testing, UI testing, and end-to-end testing meant that Priceline could continue building on the success of Autobot while reducing the burden of test maintenance and overcoming infrastructure challenges.
Antony was part of the proof-of-concept process with mabl, partnering with the mabl team to understand the nuances of low-code test automation and how a unified platform could support Priceline's quality goals. Immediately, he created a straightforward test that validated a hotel booking happy path. Once he knew that essential test could pass with confidence every time, Antony worked with mabl's Solutions Engineering team to build reusable test snippets that could help anyone at Priceline test more complex user journeys.
Reusable flows in mabl were similar to test templates that Antony had created for Autobot. Both features made it easier for anyone on the Priceline team, whether they were a manual tester or developer, to create automated tests.
His first project was the date selector, which can be difficult to test since date ranges are constantly changing as customers book more trips. Selecting dates is a universal - yet dynamic - step in the Priceline user experience. Having this feature work 100% of the time is essential. Antony was able to encapsulate a date selector test into a simple JavaScript snippet that was turned into a reusable flow, democratizing this crucial test step proved that mabl could pick up where Autobot left off.
Once those initial evaluations were complete, Antony scaled automated testing in mabl by packaging the Priceline homepage and different user interactions into their own reusable flows. The homepage, hotel searches, flight searches, rental car searches, and more were captured in flows so that testing the enormous range of Priceline user journeys was possible through automation and collaboration. This process included configuring data inputs so that the team could inject randomized search criteria, which ensured that tests covered the broadest possible range of likely scenarios, including probable advance purchase windows.
The last step to setting mabl up for truly scalable software testing was creating test templates with data tables that allowed testers to create numerous scenarios with little effort. This allowed QA to support A/B testing across all aspects of the user experience, a fundamental part of Priceline's culture and the organization's ability to deliver premium booking experiences to everyone.
Antony and the Priceline team completed their migration in early 2023, shifting approximately 5000 web tests into mabl's unified test automation platform. Despite the volume of tests and number of people engaged in automated tests, the transition was fairly seamless. Antony explained: "We created deploy triggers, we made it easy to pull test results, and all of it matched what we already had in Autobot. This made it possible for my team to migrate from Autobot to mabl at their own pace with the same testing strategy."
Antony's efforts to streamline the migration into mabl paid off. Approximately 45-50 people are running 80,000 tests per month, unleashing a culture of quality across the company. Testing has been democratized and integrated into CI/CD pipelines for quality that matches the speed of Priceline's development cycles and their innovative development culture.
Mabl has allowed Antony to build a seamless system where tests are automatically triggered post-deployment, results are monitored, and progression to the next stage is automated based on pre-defined success criteria. Slack and Splunk notifications alert relevant stakeholders, smoothing out communication processes and virtually eliminating the need for developers to wait for manual testing interventions. Everyone is able to participate in testing with confidence.
Managing quality for a product as complex and dynamic as Priceline's isn't easy, but Antony's journey with automated testing shows that it's possible. From manual testing to Autobot to mabl, his team has been able to democratize and scale software testing across the organization for a true culture of quality that engages everyone from the CEO to developers and testers. Mabl made it possible to build on the successes of Autobot - integrating testing into CI/CD pipelines, democratizing automated testing, and automating 80% of Priceline's testing - while removing the infrastructure and test maintenance barriers that plagued the Priceline team.
It’s the only SaaS solution that tightly integrates automated end-to-end testing into the entire development lifecycle.
Mabl is the leading unified test automation platform built on cloud, AI and low-code innovations that delivers a modern approach ensuring the highest quality software across the entire user journey. Our SaaS platform allows teams to scale functional and non-functional testing across web apps, mobile apps, APIs, performance and accessibility for best-in-class digital experiences.
