What does it actually take to build autonomous agents for a discipline that depends on exact, repeatable outcomes? mabl spent three years finding out. From early PaLM experiments to the Agent Framework running in production today, mabl engineer and tech lead Lauren Leidal shares what building at the edge of generative AI and software testing taught her team. [Click here to read Part I]
At our annual mabl Experience conference in the fall of 2025, we took the stage and demoed something real: the test planner, the authoring agent, running autonomously in the cloud. No smoke and mirrors, this was the mabl-as-a-teammate vision actually coming to life.
Everyone was holding their breath.
The reception was everything we hoped for. But backstage, the feeling was more complicated. We knew what the audience didn't: the infrastructure underneath that demo wasn't built to last. The authoring agent still carried all the same fundamental issues we'd been wrestling with. We had proven the vision was real, and simultaneously made it undeniable that we weren't ready to deliver it at scale.
Then November hit, and coding agents started taking off across the industry. The window for figuring this out quietly was closing fast. That's when we made the call: stop building new features, fix the foundation.
We were hitting a severe organizational bottleneck. Our broader engineering team was working incredibly well together, but building and scaling these agents still relied heavily on the ingrained expertise of a few key individuals.
If a product team wanted to build an agentic feature, they constantly had to pull in those experts for consulting. There were simply too many architectural and design decisions to make from scratch—from how to properly query the RAG pipelines to setting up the evaluation suites correctly. Relying on a handful of people for every major design decision wasn't a sustainable way to scale. We had to stop hand-crafting individual agents and build a factory for agents instead.
From December 2025 through January 2026, we embarked on a massive winter sprint. We paused net-new feature development to focus entirely on consolidating our fragmented, early approaches into a single, unified Agent Framework. We codified everything we had learned—abstracting away the painful complexities of memory management, cross-platform state, and model whiplash.
Crucially, while this framework wasn't a shiny new feature itself, it immediately paid dividends by upgrading our one true existing agentic capability: generative test creation. By codifying all our hard-won lessons into the framework, we were finally able to address the hallucination, context, and reliability issues that had plagued our earlier versions of the tool.
The authoring agent you see today is the sixth version. We stopped counting. But each rewrite made the next one faster—because the framework meant we weren't starting from scratch anymore.
While we were codifying the Agent Framework for our product, our engineering culture was undergoing a simultaneous transformation. In early 2026, we took the internal coding agent experiments we started in the summer of 2025 and scaled them massively. We deployed a system of AI agents capable of writing, evaluating, and shipping real code across our own 75+ repositories. Today, every engineer at mabl uses them daily.
What surprised us most wasn't the AI part. The biggest bottlenecks to agent velocity turned out to be the same ones that slow down human engineers: build speed, CI/CD setup, test coverage. The fundamentals. Fixing those made the biggest differences, for people and agents alike. It was the same lesson the Agent Framework was teaching us on the product side: error handling, reusability, and standardized testing don't become less important when you introduce a powerful non-deterministic tool. They become more important. AI amplifies whatever foundation you've built, good or bad.
The two initiatives ended up being perfectly complementary as a result. As engineers embraced agentic workflows to write code, the framework ensured that when they turned to build AI features into the product, they were starting from a cohesive, well-engineered foundation rather than parroting outdated practices.
With the team already acclimating to AI-assisted workflows, we initiated the next massive phase of the pivot: onboarding roughly two-thirds of the entire mabl engineering team onto the Agent Framework.
Software testing and backend engineering are deeply deterministic fields. We were suddenly asking an entire team of traditional engineers to step out of their comfort zones, think probabilistically, write prompts, and embrace non-deterministic workflows.
The reality was messier than any framework slide would suggest. When one of the first teams shipped results analysis on the new framework, we ran a retro. The sentiment was positive—but the chaos was real. Staffing transitions mid-project, async decision-making that left engineers feeling siloed, and a new wrinkle nobody had anticipated: when AI coding agents are doing the heavy lifting, how do you make sure engineers actually understand the code they're shipping?
Scaling also exposed problems that expertise alone couldn't solve. Teams were independently building overlapping tools without realizing it. PRs were landing that broke things in subtle ways that only someone deep in the agent architecture would catch, and I was the one catching them. It felt like whack-a-mole.
Every time I fixed one subtle issue, another would surface somewhere else, and the team was moving too fast for me to keep up manually. So I codified what I knew into specific agent review standards for our PR review bot. When it started catching the things I would have caught, it was a genuine relief—less because it saved time, and more because I could finally focus on moving things forward instead of just keeping them from breaking.
When we started, only the authoring agent ran on the framework. Within months, six did: test planning and authoring, test recovery, collaborative results analysis, automatic failure diagnosis, application modeling, and agent session review. The framework didn't just make individual agents better. It made building new ones dramatically faster.
We had successfully transitioned from an organization experimenting with AI features to a fully agent-driven engineering team. The friction of the early days was gone, replaced by a scalable system that allowed us to move at the speed of the broader generative AI market.
But what exactly does that system look like under the hood? In the fourth and final part of this series, we'll open the black box and deep-dive into the actual architecture of the mabl Agent Framework.
The Winter Sprint gave us the factory. Now we're opening the black box.
If there is one massive misconception in the generative AI space right now, it's the idea that scaling AI is about writing "better prompts" or discovering magical new ways to talk to the model. It isn't. Scaling AI is actually about returning to foundational software engineering practices. We stopped treating AI like magic and started building foundational robustness—focusing on reusability, shared libraries, and standardized error handling.
Here is exactly how the framework translates non-deterministic AI into a reliable, enterprise-grade testing engine.
An agent is only as smart as the context it's given. But LLMs have strict context window limits. Dumping an entire application's DOM, a user's workspace history, and test execution logs into a prompt will cause the model to choke, hallucinate, or simply hit token limits.
Before the framework, every team trying to build an AI feature was reinventing the wheel—writing their own custom scripts to trim data and query our RAG pipelines.
The framework solved this by centralizing our context engine and introducing shared compaction strategies. We built standardized, reusable modules that automatically summarize, prune, and manage token limits before the data ever reaches the model. Now, whether an agent is looking at a single element or analyzing thousands of test runs, it uses the exact same battle-tested compaction pipeline to ingest user-specific workspace data efficiently.
Early on, it was tempting to build monolithic agents—one massive prompt telling the model how to do everything from finding a button to evaluating a visual state.
We abandoned that approach entirely. Instead, we broke agentic capabilities down into a shared library of Tools and Skills.
Shared Tool Definitions: We standardized the exact schemas we use to present capabilities to the LLMs. Crucially, tools aren't just for taking action (e.g., "click element"). We also use shared tools to dynamically pull in context on the fly (e.g., "query the DOM," "fetch previous test history"). Defining these once means that when we improve how a tool gathers context, every agent using that tool gets smarter instantly.
Skills: These are reusable cognitive abilities built on top of those tools (e.g., "evaluate a visual state against a baseline," "determine if a login was successful").
This composability is the secret to our velocity. But standardizing these inputs also allowed us to build a cohesive message handling framework.
Because LLMs are probabilistic, they will occasionally hallucinate a tool call, lose the conversational thread, or pass malformed JSON. Our message handling framework standardizes how we communicate back and forth with the models across all features. It acts as a universal catcher's mitt. When a model hallucinates, the framework catches the failure, normalizes the error, and automatically forces the model into a standard self-correction loop—all without individual developers needing to write bespoke retry logic.
There is an inherent risk to shared libraries: update a core context tool to improve one agent and you might accidentally break three others.
This is where the evaluator suites we built back in Summer 2025 proved their worth. We baked "Evals-as-Code" directly into the pipeline.
Because we measure AI performance quantitatively, we can guarantee robustness. If an engineer updates the "query DOM" tool, the framework automatically runs our LLM-as-a-judge evaluator suites across simulated agent workflows. It acts as an ironclad safety net, preventing regressions from hitting production.
The true power of this framework isn't just that it works—it's the universal portability it unlocked. Because everything is built on shared context strategies, composable tool definitions, and unified message handling, spinning up a new AI capability went from a massive architectural undertaking to an assembly line process.
The framework now acts as the engine for all our embedded, day-to-day Workflow Agents, including the Recovery Agent, Chat-based Results Analysis, and the Test Creation Planner. It also allowed us to port this architecture to the cloud, powering four dedicated, autonomous Cloud Agents: the Authoring Agent, Failure Analysis, Application Modeling, and the Review Agent.
The Review Agent is perhaps the most perfect realization of what we set out to build. Every week, it analyzes hundreds of real authoring sessions and produces a structured quality report—synthesizing both scored metrics and free-form narrative analysis into trend data neither could surface alone.
When we moved to Gemini 3, it told us reasoning loops and premature failures had dropped by 2-4x depending on the failure type. But it also flagged that hardcoded values actually increased, because Gemini 3 was now completing tests that Gemini 2.5 never finished. A regression hiding inside an improvement, caught automatically across hundreds of sessions.
The other half of the value is more subtle. The report doesn't just surface trends—it flags the specific sessions worth a human's attention. Finding the one catastrophically broken session in a thousand used to be a needle-in-a-haystack problem. Now it's a list. An engineer can go directly to the exact session logs that matter, combine the agent's structured analysis with their own judgment, and synthesize the actual fix. The agent handles the scale. The human handles the nuance. That combination is what makes it work.
Looking back at the summer of 2023, when we were struggling to get a model to navigate a basic login screen, it's hard to overstate how far the architecture has come. But the more important question isn't where we've been. It's where companies going through their own agentic transformation are going to run into trouble next.
We think about two things. The first is fit. Agents need to feel like teammates, and teammates have context about what you care about and work within your existing tools. Either half of that breaking down creates friction that compounds fast. The second is efficiency. AI bottlenecks are becoming less about model intelligence and more about infrastructure, availability, and token costs. A good teammate doesn't burden the team. They share what matters, skip what doesn't, and make the people around them faster. Agentic systems need to clear the same bar.
That's the next frontier. The last three years taught us that the teams who navigate it won't be the ones with the best predictions. They'll be the ones who build the infrastructure to adapt faster than the problems change.