I used to open a pull request, watch the smoke tests pass, and hit merge. Then I moved on. Anything that happens post-merge takes a couple of hours, so by the time a Slack alert popped up 90 minutes later, my only thought was, "What is this?"
Sometimes it was a test I should have retrained. My code change tweaked the user interface, but the pull request only ran a lightweight smoke suite checking basics like login. The full regression suite didn't run until the code reached our dev environment. I forgot to update the test in mabl to match the new behavior before merging.
AI has sped up how we build, the same way it has for our customers. Since November 2025, we've gone from shipping around 235 changes a month to more than five times that. Now we push around 300 a week. More changes means more chances for bugs and regressions, and AI writes its share of them.
We build mabl, and we test mabl with mabl, so we felt the pain of scaling at the same time our customers did. In some ways we were looking at their future. Blocking gates turned out to be a good answer. They moved ownership of our tests from five people to whoever makes the change, and that's how we started listening to our own test results again.
For a long time, five of us maintained 70 to 80% of our mabl-on-mabl tests. Whenever another engineer or a product manager updated the user interface and forgot to update the corresponding test, one of the five of us usually ended up fixing it.
It was exhausting. Tests fell out of sync, and the five of us were burning out.
It also changed how we treated failures. Somewhere before July, we stopped listening to our own results. They failed constantly because so few of us were keeping them up to date, and we couldn't keep up.
The reaction became "it failed again, let's ship it," which is how things break.
Believe it or not, we now have three gates. That sounds like a lot, but it means we're protecting our customers.
Once a pull request clears the gates and merges, the code lands in our dev environment, where we run the full regression suite of 294 tests. If a change looks risky, the author can also run the full suite on the pull request by leaving a comment. We wrote about how we chose and stabilized the tests in our PR gate “Why our PR gate has 17 tests, not 288”.
We're not in GitHub all the time. When a check fails, the author gets a private Slack direct message, usually about 20 minutes after opening the pull request. The alert links directly to the GitHub failure.
By the time the Slack ping arrives, mabl has already analyzed the failure. If a test fails, the alert includes the deployment ID, failure status, next steps, and a direct link to the mabl test results.
One failure I looked at was labeled "recurrent test issues." The test had been failing for a while, so my code probably wasn't the culprit. The alert suggested verifying the test's reliability and checking the environment. Because we have test impact analysis enabled, the failure also shows which tests ran, why they ran, where they touched my code, and whether my code had any coverage at all.
When a test turns out to be flaky, we ask our quality engineer to help make it more reliable, but when a test isn't reliable yet, we pull it from the gate until it is. I think the most important thing about a gate is that you trust it, because if you don't, you end up skipping it all the time. We'd rather have a gate with fewer tests we trust than more tests we don't.
Every one of our gates has an emergency exit, and I think every gate should. If you're shipping a fix for a severity-one customer issue, you don't want a gate stopping you for anything.
Ours is a label on the pull request. Every use gets tracked, and whoever adds it has to justify going around the gate. A label is how we built it, but there are plenty of other ways to do the same thing.
The other option is waiting it out. In October, a third-party security scanner we use as a pull request gate went down. GitHub logs showed 500 errors, and the vendor's status page and a quick check of other pipelines confirmed it was a wider outage.
Our norm is to wait and rerun the check once the outage is over. If it drags on for hours and our change is low risk, we use the emergency exit, and the outage is the justification.
The test impact analysis gate went live on October 7. In its first four weeks, it caught three bugs before merge, and it flagged three tests that needed retraining to match the change in the pull request.
That is the exact mistake I mentioned at the beginning. A pull request should only merge when its tests reflect the new behavior. Otherwise you get failures that only mean a test needs retraining.
It also helps me remember, because by the time the dev results come back, my mind has gone somewhere else, and I've forgotten I merged an hour and a half ago. Getting that Slack message while I'm still on the pull request means the change is still fresh in my head.
We chose to make these gates block merges, and that changed who fixes a broken test. If you want your code to ship, you have to fix the failing test. The gate turns the pull request author into the owner of the fix. If you don't know how to fix it, you ask for help in Slack.
It no longer depends on someone's goodwill, and ownership is more straightforward.
Reactions to this shift vary. It depends on who you ask. There's a gap between engineers and the PMs and designers who also open pull requests. That second group asks for more help and can feel more concerned or lost. I think it's a matter of time, because it means relearning a process.
For this to work, you need tests that aren't flaky, so we made a big investment in reducing flakiness. Patrick recently wrote about [what some of our flaky timing failures actually turned out to be](LINK: Patrick's post). Today, when I merge code, I feel confident. If a test fails, it's most likely a real failure, not a fake one.
A few weeks ago, a performance test in our smoke gate blocked a pull request from our co-founder, Dan. The app's actual performance hadn't regressed. Instead, the test caught a legitimate bug in our performance testing feature itself.
A failing performance test pointing to a bug in the performance testing feature is a headache to untangle, so it took some back-and-forth in an engineering channel. We fixed it, and Dan's change waited until we did. A gate is a gate for everyone. That bug never reached our customers.
It concerned me that we'd lost the value of mabl because we weren't listening to it ourselves. Now we trust mabl again and we listen to it, and blocking merges has been a good solution for that.