A test fails. You rerun it. It passes. You move on, blaming an under-provisioned staging environment, drifted test data, or a timing issue. But a lot of ‘timing issues’ are race conditions, a problem computer science named and studied decades ago. Once you call it that, you can go find the root cause.
We put a small set of tests in front of every pull request as a merge gate, so one red test blocks the merge, which makes small failure rates add up fast. If each test passes 99% of runs, a 17-test gate fails about 16% of the time, so three out of every 20 pull requests would fail for no reason. Add higher development velocity, and developers hit these false failures daily. They stop investigating and blindly rerun the test.
So we stopped allowing reruns and investigated every failure. That took a couple of months. Each failure turned out to be something specific, and a lot of them were the same thing: an assumption about order that nothing enforced.
We constantly assume sequential order in our code. We expect a component to read a cache only after a previous component updates it, or a UI element to be clickable only after its data loads.
Independent systems run at their own pace. Database queries take varying times depending on load. Writes aren't instantly visible, and network requests interleave unexpectedly. When code relies on which operation finishes first without explicitly enforcing it, it works until the race goes the other way.
A flaky test is a sampling machine aimed at exactly the places where your software assumes an ordering it never enforces, running enough trials to turn a rare interleaving into a reproducible rate. A user might see a glitch once every few months and refresh the page, but an automated suite running hundreds of times a day catches it constantly. More runs make flakiness more expensive, and give you enough samples to find what’s behind it.
In Anthropic’s playbook, the coding agent's plan.md names "the tests that prove it," and the agent writes those tests alongside the code. Coverage comes from the coding agent's own reading of the change, decided while it builds. The feedback loop confirms the work against those checks, or a screenshot against the mock, and the session ends.
One of our tests occasionally returned a blank account name right after setting it. We assumed a linear process: the write commits, the cache clears, and the next reader hits the database to cache the new record.
What actually happened:
The cache now holds pre-write data until the entry expires. Someone sets an account name, reads it back blank, and keeps reading it blank long after the write succeeded. Waiting wouldn’t have fixed that test.
Another test clicked a credential but intermittently failed to load the details page. We assumed the click would hold the old page briefly, fetch the new data, and render the new UI.
The data arrived in 287 milliseconds. Then the new page needed translation files the list page had never loaded. The transition had no fallback for that wait, so the app stayed on the list page for 17 seconds with no spinner and no error. The click only succeeded if the test arrived via a path that happened to pre-load those specific translation files into memory.
A race condition means somebody assumed an ordering that was never guaranteed. Sometimes the application assumes it, and sometimes the test assumes it.
When our suite's flake rate jumped 6% in a single week, we found two two tests sharing a hardcoded record name, each assuming it would get there first.
When triaging a failure, check the run artifacts to find out who assumed the order. If the test did it, rewrite the test to wait for explicit state changes rather than arbitrary timeouts. If the application did it, you just caught a customer-facing bug for the cheap price of a failed pull request.
Developers often blame the framework when a test goes red but the app seems fine. We assume the runner is slow or the tooling is brittle.
The test system is doing what you built it to do. Every cause in this post was in the product or the tests, and none of them were the runner or the framework. Staging can surface problems too. If it’s overloaded enough to change the order events arrive in, your production users are one traffic spike away from hitting the same interleaving.
We hit the gate’s reliability bar by refusing to let any failure go unexplained. That turned out to be the same work as finding our race conditions before our customers did. The gate was the forcing function, and the bugs were the payoff.
To learn more about mabl, take out a free 14-day trial or schedule a demo with our team.