A while back I wrote about how we stopped guessing and started measuring. We built a Session Review Agent (SRA) to read thousands of Test Authoring Agent (TAA) sessions and tell us, in structured detail, why an agent succeeded or failed in messy, real-world conditions.
Running that reviewer at scale taught us something we didn't expect. The SRA is almost perfectly reliable about what went wrong: every error string, failing tool, and message index it cites matches reality. It's far less reliable about why. Its root-cause theories are warm, but wrong about a quarter of the time.
That asymmetry is the entire design. If you want agents to fix production bugs without a human babysitting every step, you can't build the system on the one thing your reviewer is bad at. So we didn't. We built a pipeline that treats the SRA's symptoms as ground truth and its hypothesized causes as leads to be proven or killed against the source code before a single line of a fix gets written.
That pipeline now runs on every cloud test generation session. It clusters failures by symptom, sends an investigator with full code access to confirm the actual cause, writes a targeted fix and a regression test, puts it through a panel of simulated reviewers, and opens a PR with a full chain of evidence waiting for a human to hit merge.
The interesting part isn't that it runs, it's what building it taught us about handing agents the keys to your own codebase. Here's how it works, how the agents collaborate, and everything that broke along the way.
To fix failures automatically, you first need a reliable source of truth that says exactly what went wrong. That is still our SRA.
The SRA reads a completed authoring session (every message, tool call, DOM snapshot, and reasoning step) and emits a highly structured JSON review. It flags agent bugs, categorizes suboptimal choices, and provides narrative summaries.
But I built it with one deliberate constraint: the SRA has zero code access. It reasons entirely from the session transcript.
That single constraint shapes everything downstream. The SRA is spectacular at identifying what happened, but it's only decent at guessing why. After monitoring this across thousands of sessions, we mapped out its accuracy profile:
|
Metric |
Accuracy |
Engineering Observation |
|
Symptom Detection |
~100% |
Highly accurate. Every quoted error string, failing tool, and message index matches reality. |
|
Routing Flags |
100% |
Even if its root-cause theory is off, it always points investigators to the right tool and session context. |
|
Mechanism (Why) |
~75–80% |
Root-cause guesses are usually warm, but wrong about a quarter of the time. |
Our operating contract is simple: trust the symptoms unconditionally, but treat the SRA's hypothesized cause as a lead for downstream agents to verify against the source code.
There's a reason we like this division of labor beyond the raw accuracy numbers. Keeping the SRA on behavior and Claude on code lets each do the thing it's actually good at. Cut off from the codebase, the SRA reads a session the way a person would by watching what the agent did, where the rails came off, and how a real user would experience the failure instead of getting dragged into implementation details it can only guess at.
That's not a constraint we tolerate; it's what makes the diagnostics trustworthy and keeps investigators pointed in the right direction. The why still matters, and that's Claude's job downstream. But the symptom is what the user actually feels, and keeping one agent aimed squarely at it is what keeps the whole pipeline honest.
We quickly learned that classic telemetry isn't what makes a review actionable for a downstream developer (or another agent). The highest-value signals turned out to be:
The SRA generates a "quality score" for each session, but we quickly discovered that the score is actively misleading for triaging system health. A high score just means the agent recovered gracefully (e.g., used a manual workaround, logged a clean TODO, asked for help). Our single most expensive system defect scored a 95/100 because the agent handled the failure beautifully. Cluster size and wasted execution effort are our true severity metrics; the raw score is noise.
Once we had a continuous stream of structured failure profiles running on every cloud session, we paired the SRA with Claude to close the loop. We built a specialized, multi-agent pipeline designed to turn production symptoms into shipped code.
On its first real overnight run, this pipeline took an 82-session backlog, collapsed it into four distinct symptom clusters, and had eight root-caused, fully tested PRs waiting for us the next morning.
Here is how the automated squad handles the loop:
The most valuable architectural decision I pushed for was making the investigation phase explicitly adversarial. Because the SRA has no code access, it's prone to making assumptions that sound logical but are structurally wrong.
A great example: The SRA reported a bug stating, "the agent keeps trying to delete a step, but the delete operation is failing on the backend server."
If you handed that brief directly to a code-writing LLM, it would have shipped a useless API retry-limit patch. Instead, our Claude-powered investigator read the transcript, cross-referenced it with the server code, and proved the exact opposite: the delete operation succeeded perfectly. The step was gone by message 463. But a validation replay path silently re-created the step at message 467, because the replay mechanism was unintentionally reinstalling the recorder state.
Same symptom, completely opposite engineering mechanism. The SRA pointed us to the relevant session and critical debugging info. The adversarial check saved us from shipping a ghost fix that would have masked the real architectural bug.
When you give agents the keys to the codebase, you quickly realize that managing agent behavior is just systems engineering with a different set of constraints.
1. Multi-Lens Reviews Catch Human Flaws, Too
Layered reviews aren't just theater; they generalize to whoever holds the pen. In one instance, an orchestrator agent had to finish a stalled PR itself, ran the persona reviews on its own code, and caught a bare catch {} block that was swallowing critical errors.
Even better: I wrote a manual fix for a tricky edge case and decided to run it through our automated six-lens review panel. Two of the simulated reviewers independently flagged a bug in my logic. I had written a generic "step skipped because it's disabled" message that was flatly incorrect for cases where a workspace feature flag was turned off, breaking the downstream diagnostic loop for the agent. If you are the author, you lose your objective second set of eyes. The review panel gives it back to you.
2. Agents Need Liveness Rules
An agent's natural resting state is idle. Unlike human engineers who feel a momentum-driven impulse to finish a task, an agent treats "CI Passed" as a terminal state. It finishes the step and just stops. We had to implement explicit heartbeat monitors and execution liveness rules to keep the pipeline actively walking from one stage to the next.
3. Infinite Refinement is a Trap
A tireless reviewer needs a hard stopping rule. Our machine reviewer is relentless. During one run, it iterated on a single PR through nine consecutive rounds. Each individual critique was technically valid, but by round nine, it was attempting to rewrite the semantics of a raw data accessor to accommodate an incredibly implausible edge case. We had to introduce a strict convergence policy: after three rounds on the same theme, the system must weigh the engineering risk against the threat model, document the residual technical debt, and stop.
4. The Night the System OOM'ed Itself
On our very first overnight run, four autonomous fix workers spun up their local test suites completely in parallel on the same host machine. They instantly exhausted the host's RAM, causing a catastrophic hard crash of the system.
While it forced a new rule (all functional verification must happen isolated in CI, never on the orchestration host) it highlighted a massive win: recovering the entire pipeline's state from pushed branches and open PRs took less than fifteen minutes. The system was crash-resilient by design.
By taking the SRA from a sampling tool to an always-on reviewer for every single cloud session, we've shifted from reacting to bugs to systematically eroding entire categories of failures.
We've reduced the human intervention requirement down to exactly three high-leverage touchpoints: kicking off a batch, making occasional high-level architectural policy calls, and reviewing/merging the final PR.
The next win isn't better detection, it's proving a cluster's recurrence rate actually falls after a fix ships and deploys. That's the measurement that separates "we shipped N PRs" from "we eradicated N bug classes." It's the bar we're building toward, and if you're pointing agents at your own codebase, it's the one I'd tell you to hold too.
For now, an agent that stumbles on a complex, messy web app on Monday afternoon has a fully reviewed, path-verified code fix waiting for a human engineer's approval by Tuesday morning. And the system gets a little better at fixing itself every single time it runs.