Test Results Best Practices

Introduction

This lesson is made to help you more easily understand test results and isolate issues faster.

Before taking this lesson

• Know test creation basics with the trainer (Lesson: mabl Trainer Intro)
• Understand basic mabl concepts such as workspace, plans, tests, credentials, applications and environments (Lesson: mabl App Intro)

Learning Objectives 🧠

To understand:

• What the tools used for debugging are 👾
• How/when to use those tools when approaching a failure 🎯
• Common next steps once an issue has been isolated ⛑

What we will cover 📚

• Debugging tools overview
• Test review best practices
• Common next steps after issue isolation

Ultimately, this lesson will guide you towards a repeatable process of test review, so that your team can feel confident finding issues and communicating them.

Why is this important?

When using mabl to test the frontend of an application:

• Tests are created (by a user)
• Tests are run in the cloud (by mabl)
• Results are provided to show what is working and what is broken

For things that work: rock on 🤘

For things that are broken: find the root cause, communicate it, and get the problem fixed ASAP 🛠

mabl's debugging tools help you:

• Understand what happened during cloud execution so that you can find the root cause quickly and effectively.
• Communicate/report the issue to your team easily so that it can be fixed

With clear, repeatable troubleshooting practices, your team can find and fix bugs faster, which leads to improved overall app quality and, most importantly, happy users 🙂

Debugging Tools Overview:

3 Primary Tools

1. Step details 📝
2. Screenshots 📸
3. Logs 📋

1. Step details 📝

1. Step Status/Description - tells you what action the step performed, and whether it either Passed, Failed, or Passed with warning
2. Timing - how long it took the step to execute
3. Visual Change - visual change detected
4. Element thumbnail - a small image of what element the step interacted with

Network Issue - network issue detected

Auto-heal - element found based on secondary search criteria

ⓘ 'Passed with warning' but no Visual Change/Network/Auto-heal?

This means mabl had to use an alternative method of interaction (Ex. javascript) to complete the step - shown in mabl Activity tab under logs

2. Screenshots 📸

1. Current, Baseline, and Side-by-side comparison
2. Screenshot zoom - click on area of screenshot that you would like to zoom
3. Highlighting - toggle highlighting of visual changes in screenshot

https://vimeo.com/491812498

Bonus: Visual Change Mini-Lesson

There are two different types of pixel-based visual change detection:

1. Run-by-run detection

Compares each current screenshot to the last executed test.

(Automatic for each test run)

2. Machine Learning driven detection

Uses the last 10-15 executions of a test to establish a baseline, and highlight whether areas deemed as 'static' differ from the baseline - the baseline is updated on an ongoing basis.

(Must enable in Plan under 'Advanced Settings', and run test ~10 times)

Learn more about mabl's visual testing here!

3. Logs 📋

1. Mabl activity 📊 shows details of mabl's process in passing or failing a step (Also shows what interaction mabl used when it 'Passed with warning').
2. Network 📡 shows what network calls were made in your app during the step execution. Search for specific network calls in the provided HAR file.
3. DOM 📄 - shows a snapshot of the DOM during step execution. Search to see whether certain elements appeared in the DOM during that step.
4. Performance ⏳ - shows all recorded execution times of that particular step, which indicate how users are likely to perceive the load speed of your application UI. The lower the score, the better.

Check out how to interact with each tool in the logs:

https://vimeo.com/491830259

Best Practices

Disclaimer ❗️

The following tips are meant as guidelines for the majority of test failures that you will experience, but they are not exhaustive.

Some of these practices will depend on different types of failures.

As you test more with mabl you will develop a cadence that works for you and your team 😎

ⓘ Test failed and not sure why? Reach out to our support team! Click the button in the lower-left corner of the mabl app (that looks just like the one below) and start chatting with us ANY time! 👍

Overview:

1. Start with the failure ⛔️

2. Work your way backward ⏪

3. View the last passed test for context ⬅️ ✅

4. Reproduce the failure on your own 🔂

5. Reach out to mabl support for help 🙂

Let’s dig into each one!

Step 1: Start with the failing step ⛔️

Leverage the following Step Analysis Framework:

1. Look at steps details to gather information about the failure
2. View screenshot to understand state of application at failure
3. IF on the step you see an icon for:
1. Visual changes - notice the pink visual change highlighting on the current screenshot, then compare the current screenshot to the baseline
2. Network - navigate to the network tab under logs and investigate the status code (using CMD or CTRL + F) to search in the HAR file for a particular status code
3. Auto-heal - navigate to the mabl activity tab, skim the timeline to the bottom where mabl will detail which element it could not find, then go to the DOM and search (using CMD or CTRL + F) for the attribute
4. Look at the element thumbnail on the step to see if it matches what mabl interacted within the screenshot, or if it is even present on the page at all. Here you are comparing the intention of the step vs. what actually happened during execution
5. Read mabl activity log to understand the process of how mabl failed the step

Step 2: Work your way backward ⏪

Clicking on the steps before the failure, work your way backward to understand what happened leading up to the failure using the Step Analysis Framework for each step.

ⓘ Be sure to look for the yellow circle in the screenshot to see what element mabl interacted with during that step execution.

Step 3: View the last passed test for context ⬅️ ✅

Using the dropdown under 'Time of run' at the top of the results page to see previous runs of the same test.

Look at screenshots to compare the behavior.

ⓘ Quick Tip 

Virtual Tabs

Use Virtual Tabs to go back and forth between tests easily when viewing multiple test output pages.

Step 4: Reproduce the failure on your own 🔂

First, edit the test and try reproducing the failure.

Next, go into your application WITHOUT using the trainer and try reproducing the failure.

Step 5: Reach out to mabl support for help 🤗

Use the chat icon in the lower left-hand corner of the mabl App.

Next Steps

Overview

After finding a failure, the approach will differ depending on the nature of the failure.

The following are good next steps to be aware of:

• Add failure reasons
• Leave comments for teammates
• Share results/data with your team
• Create a Jira ticket

Let’s expand on each one 🤠

Add failure reasons❗️

- From the dropdown, select the reason that most closely describes why the test failed

If the reason you're looking for isn't there, select 'Other', or submit a feature request here!

ⓘ Did you know?

You can filter on failure reasons from the Results page (left navigation bar). You can also view the percentage of each failure reason in the past 30 days from the Dashboard!

Leave comments for teammates 💬

Help give context to your teammates by leaving comments on the test output page.

Shows details of the commenter, and timestamp of comment.

Communicate by exporting results or data 📝

Select the type of information you would like to export by downloading it and sharing it with the necessary people on your team

To export results as a PDF - click the three dots net to Edit

ⓘ Did you know?

If you navigate to the failed step, copy the URL, and then share it with teammates it will bring them directly to the step of the failure!

Create Jira Ticket 👾

Since everyone's Jira instance is a bit different, click the links to our help docs below, which have more specific instructions for how to approach that!

Jira Cloud Integration

Jira Server Integration

Recap

What did we learn?

What mabl's debugging tools are, and how to best use them for reviewing test results and isolating issues.

What will you go on to do?

- Form a process around reviewing test results that is known and understood by the team

- Find those pesky bugs faster and more effectively than ever before! 👾]

Feedback

Was this helpful?

Take this 2 minute survey to let us know how we can make mablU better!