Get a Free Trial
Creating, executing, and maintaining reliable tests has never been easier.
Wurl is a not-so-secret pillar of the Streaming and Smart TV world. Their massive network connects video producers, video services, and advertisers, enabling creators to reach a global audience and advertisers. Needless to say, managing quality for this large and growing ecosystem is a considerable challenge.
Using Low-Code Test Automation To Ensure Video Quality at Scale
Jagadeesh is a Lead QA Engineer at Rebaca Technologies, where he consults on Wurl’s quality strategy. Jagadeesh has years of experience in automated testing, including building his own custom software testing frameworks. For Wurl, custom automated testing frameworks were only useful to team members that knew how to code, limiting their ability to scale automated testing.
Wurl provides a feature-rich experience for media companies to schedule their media content. In order to ensure that the ad and promo placements for their video content play at the right location and that the transition is smooth during playback, Wurl has developed a video player that ensures ad placements are seamlessly placed in existing content
Validating Video Player Functionality with JS snippets
Most streamed television has approximately 24 frames per second, so the average 30-minute show has around 43,200 frames. Even a 30-second ad features over a thousand frames, making it difficult to assure quality across an entire program without automated testing. With mabl, Jagadeesh and his team are able to use automated smoke tests to validate text in different video frames.
Consider a 30 minute video: there may be 10 minutes of show content, followed by one or more ads, followed by more content. This happens multiple times throughout the day as the content lineup shifts, and each time, a smooth transition and consistent quality between a changing queue of ads and changing episodes is expected. Using manual visual and audio testing combined with automated testing with mabl, the team is able to validate the functionality of each sequence.
To test each scenario, the Wurl team designed their mabl test to pause the video at specific durations (5, 10, and 15 mins) for different types of asset playback like promos, ads, and episodes. At each given point, the video frame is expected to contain particular text. They validate the location of the text within the player by using custom JS Snippets, which utilize OCR technology. Each test can contain any number of steps that validate videos at multiple points to cover more complex scenarios, depending on the needs of the content. A library of reusable flows simplifies the test creation process as different team members look to test different assets.
The following screenshot demonstrates the playback sequence validation of Promo, Ad and Asset types. In each step, text within the video will be validated.
Before mabl, scaling testing to this extent was difficult since Wurl's QA team had different areas of expertise, in addition to the challenge of maintaining testing infrastructure. Low-code automated testing makes it possible for everyone to contribute to Wurl’s quality strategy, even for difficult test scenarios like validating video.
Measuring Quality and the Viewer Perspective
QoE (Quality of Experience) describes metrics that measure the performance of a service from the perspective of a user or viewer. Common streaming video related QoE metrics include rebuffering, playback failures, and video startup time. For example, when a viewer is streaming video, the network might experience a 2-second outage that would be captured by a QoS availability metric. However, if the video player had buffered enough video data ahead of time, the viewer won't experience any issues that would be measured by a QoE metric, such as rebuffering.
QoS (Quality of Service) refers to metrics that measure the overall performance of a service. In the context of streaming video, this often refers to networking and the ability of a network to deliver video bytes quickly and consistently. Common metrics include bandwidth, throughput, latency, and availability. With mabl, Wurl has been able to improve both QoE and QoS metrics, thereby ensuring that viewers have a seamless, enjoyable streaming experience.
Manual and Automated Testing Join Forces to Keep Wurl Going ‘Round
Managing quality at scale is no easy feat, but a combination of manual and automated testing helps Wurl’s QA team move quickly without missing a single frame. Since starting with mabl in March 2021, Wurl has been able to grow their automated testing strategy to 300+ test cases, and get more cross-functional team members engaged in test automation. Wurl’s network of creators, advertisers, and viewers is growing quickly in the streaming era and thanks to an effective testing strategy created by Wurl’s QA team, they’ll be able to ensure quality across all videos.