Agentic QA in Practice: Using Autonomous Bug-Catching Agents in Client Automation Projects

Most "agentic QA" content I read is written by people selling agentic QA. It reads like a brochure: agents plan, execute, adapt, and heal, and your QA team finally sleeps. That has not been my experience running automation projects for clients across a handful of distributed teams in different time zones.
My experience is messier. Agents are genuinely useful, but only when you treat them like a junior engineer with a lot of enthusiasm and zero context about your client's business. I have shipped agentic testing automation into three client projects over the last year, and I want to write down what actually worked, what hallucinated, and what I would do differently. This is a practitioner's field notes on agentic QA and autonomous testing, not a vendor pitch.
What agentic QA actually means (and what it isn't)
The clean definition: agentic QA is when an AI agent owns a chunk of the test lifecycle. It reads a goal ("make sure checkout works for a logged-in user with a saved card"), decides which steps to run, executes them in a browser or API, observes the result, and adapts when the UI changes. The planning loop is the important part. The agent is not just generating a Playwright script once and walking away.
It is not the same as AI-assisted testing. Copilot-style tools autocomplete your test code. That is useful, but it is still you driving. A true autonomous QA agent closes the loop on its own.
Here is how I separate the two in practice (the table scrolls horizontally on narrow screens):
| Capability | AI-assisted (copilot) | True agentic QA |
|---|---|---|
| Writes test code | Yes, with you prompting | Yes, and runs it |
| Decides what to test | No, you decide | Yes, from a goal statement |
| Handles UI changes | You fix selectors | Agent re-locates the element and retries |
| Reports failures | You triage | Agent classifies (real bug vs flake vs selector drift) |
| Cost per run | Near zero | Real token spend per run |
| Debugging when wrong | Straightforward | Needs guardrails and logs |
That last row is where most teams get hurt.
Why I started using autonomous bug-catching agents
I build a lot of Next.js apps with Node/NestJS backends, and the pattern I kept hitting was this: a client asks for a small change, I ship it, and two weeks later something in a totally unrelated flow breaks. A date picker in the admin panel. A Stripe webhook that silently retries. A form that now submits twice because of a race condition I introduced while "just" upgrading a dependency.
My old Playwright suite caught maybe 60 percent of that. The rest slipped through because writing good E2E tests is slow, and clients do not want to pay for tests they cannot see. Agentic QA changed the economics. I can point an agent at a flow and say "explore this and tell me what breaks" and get a report in ten minutes instead of spending a day writing selectors.
That is the honest pitch. Not "replace your QA team." It is "cover the long tail you were never going to test by hand." Subtle regressions, the kind that hide in caching layers and dependency upgrades, are exactly what an agent is well suited to catch.
The architecture I actually use
I run agents three ways depending on the project. Here is the comparison I wish I had when I started (also scrolls horizontally on mobile):
| Approach | Best for | Cost | Reliability | Setup time |
|---|---|---|---|---|
| Hosted agentic platform (Momentic-style, Autify, Virtuoso) | Teams without deep test infra | Subscription, predictable | High, vendor-maintained | Hours |
| Self-hosted agent + Playwright MCP server | Devs who already have CI and want control | Token spend only | Medium, you maintain it | 1 to 2 days |
| Claude Code / Cursor agent driving a headless browser | Exploratory bug hunting, one-off audits | Token spend only | Medium, needs supervision | Minutes |
| Traditional Playwright suite + AI triage | Stable regression on known flows | Cheapest | High | Already done |
For most of my client work I combine the middle two. I keep a lean Playwright suite for the critical paths (login, checkout, payment webhook), and I run a self-hosted agent on top for exploratory passes and regression on the long tail.
Wiring up a Playwright MCP server
If you are building your own stack, the cleanest pattern I have found is exposing Playwright as an MCP server and letting a Claude Code or Cursor agent call it. The agent gets tools like "navigate," "click," "snapshot," and "screenshot," and it figures out the sequence from the goal.
Here is roughly what my MCP config looks like for a Next.js project (this follows the standard mcpServers config shape used by Claude Code and Cursor; see the Playwright MCP docs for the current flags):
{
"mcpServers": {
"playwright": {
"command": "npx",
"args": ["-y", "@playwright/mcp@latest", "--headless"]
}
}
}
Then I give the agent a goal in plain English:
Goal: Verify that a new user can sign up, confirm their email,
add a product to cart, and complete checkout with a test card.
Report any console errors, network 4xx/5xx, or visual regressions.
Do not use production data. Use the staging environment at
https://staging.client-app.com. Stop if you hit a CAPTCHA.
The agent runs, and I get back a report with screenshots and step logs. That is the whole trick. The prompt is the test. It is not magic, it is a well-scoped task. If you want the full production version of this pattern, auth, tool schemas, observability, deployment, see my guide to building a production MCP server in Next.js.
The failure modes nobody warns you about
I want to be blunt here because I have been burned.
Hallucinated passes
An agent will sometimes tell you a test passed when it did not actually complete the flow. It clicked something, the page did not load, and it assumed success. I have seen this on slow staging environments where a 3-second timeout fired and the agent "recovered" by clicking whatever was on screen.
Fix: require the agent to assert a specific end state. Not "the page loaded," but "the confirmation element with text Order # is visible." No assertion, no pass.
Selector drift masquerading as a bug
An agent that "self-heals" selectors is great until it heals around a real bug. I had an agent skip a broken login form because it found a different button that happened to submit the form in a degraded state. The test passed. The bug shipped.
Fix: log every healing event. If the agent re-located an element, that is a yellow flag that needs human review before it goes green.
Token spend creep
Running an agent on every PR sounds great until you see the bill. One of my clients had a CI setup that triggered an exploratory agent on every push. On a busy day that was 40 runs at roughly 80k tokens each. That is not free.
Fix: run full agentic passes nightly, and a cheap Playwright subset on every PR. Reserve the agent for the critical flows and the exploratory pass.
Context starvation
An agent that does not know your app's business rules will happily test the wrong thing. It will fill in an email field with a random string, submit, and mark it a bug because the app rejected it. The app was right. The agent was wrong.
Fix: give the agent a system prompt with the client's business context. "This app requires a 10-character password with a symbol. Emails must be from the client's own domain for staging tests. Payments use Stripe test cards, never live keys."
Tool definitions that drift out from under you
One more that bit me later: agent tools change. A Playwright MCP update renames a tool, or a schema shifts, and your agent silently starts failing or, worse, passing on stale assumptions. If you are running agents in CI, version and test your tool definitions the same way you version code, my writeup on agent skills drift and versioning covers the discipline.
A realistic rollout playbook
This is the sequence I use when onboarding a client project to agentic testing automation. It takes about two weeks end to end.
- Audit existing coverage. List the flows you already test and the ones you do not. The untested long tail is where the agent earns its keep.
- Pick three flows to pilot. Not ten. Three: one critical (checkout), one complex (multi-step form), one flaky (anything with a third-party widget).
- Write the goal prompts. Plain English, with explicit end-state assertions and business rules.
- Run in a sandbox first. Staging environment with test data only. Never point an agent at production without a hard kill switch.
- Review every result manually for one week. You are training your own judgment about what the agent gets right and wrong.
- Introduce the nightly run. Full agentic pass at 2am, results in Slack by 3am.
- Add the PR subset. A small, cheap Playwright run on every PR for the critical paths.
- Set up the triage loop. Failures go to a queue; a human classifies each as real bug, flake, or agent error.
- Measure. Track how many real bugs the agent caught that your old suite missed. That number is your ROI.
I keep the pilot tight because the failure mode I see most is teams going all-in on day one, drowning in false positives, and abandoning the whole thing. Three flows, one week, honest review. Then scale.
What this costs, realistically
Nobody talks about this honestly, so I will. The figures below are example ranges from a handful of my own client projects in the past year, not published list prices or current vendor quotes, treat them as ballpark and re-quote against current vendor pricing and model rates before you budget.
| Line item | Example monthly cost | Notes |
|---|---|---|
| Hosted agentic platform | $500 to $2,000 | Per-seat pricing; scales with team size |
| Self-hosted token spend | $100 to $600 | Depends on run frequency and model |
| Infra (staging, CI minutes) | $50 to $200 | You probably already pay this |
| Human triage time | 4 to 8 hours/week | The real cost. Do not skip it. |
For a mid-size client, that triage time is the number to budget for. Everything else is noise. If you cannot afford a human to review agent output, you cannot afford agentic QA. You will just be paying to generate noise.
A worked example
Take a typical mid-size client: a Next.js storefront with a hosted agentic platform, one engineer doing triage, and a nightly run. Using the ranges above:
- Hosted platform (mid-tier): $1,200/month
- Token spend on the nightly pass: $300/month
- Staging + CI minutes: $100/month
- Triage at 6 hours/week at a blended $60/hour: ~$1,440/month
That is roughly $3,000/month, and more than half of it is the human triage, which is the point. If the agent catches even two or three real regressions a month that would otherwise have shipped and cost a client a support fire drill, the math works. If it catches nothing for two months, the honest move is to cut the nightly run back to a weekly audit rather than keep paying for noise.
For smaller projects, the honest answer is often "not yet." If a client is paying me for a $5k website, the agent overhead does not make sense. I run a lean Playwright suite and a one-off agentic audit before launch. That is it.
Where agentic QA genuinely shines
Three places where I have seen it pay for itself:
- Pre-launch audits. I run an agent against a staging environment before go-live and catch edge cases the client never thought to test. The agent found a broken password reset on a recent Next.js build that my manual QA missed entirely.
- Regression on legacy CMS sites. Sites with old plugins, weird redirect chains, and inconsistent admin UI are a nightmare to test manually. An agent handles the inconsistency better than I do.
- Third-party integration drift. Stripe, Twilio, and Mailchimp all change their UI and behavior. An agent that notices "the webhook payload shape changed" is worth its weight in gold.
Where it does not shine: brand-new UIs with zero stable selectors, apps with heavy CAPTCHA or bot detection, and anything where the "correct" behavior is a judgment call (design, copy, tone). Keep a human on those.
The MCP angle for automation developers
If you are building automation for clients, not just testing, this is the part that matters. The same MCP pattern that lets an agent drive Playwright lets it drive your business tools. I have built MCP servers that let an agent query a client's CRM, trigger a Zapier flow, or check a Shopify order status. The agent does not need to know how those tools work internally. It just needs the tool schema and a goal.
That is the real shift. Agentic QA is a specific application of a much bigger pattern: give an agent a goal, give it tools, and let it work. Testing is just the first place most teams see the value because the feedback loop is tight and the failures are obvious.
A quick reality check on the hype
I read a lot of vendor posts claiming agentic QA "replaces manual testing." It does not. It replaces the boring 20 percent of manual testing that nobody wanted to do anyway (regression on flows you already know work). It creates a new 10 percent of work called "triaging agent output." Net time saved: real, but not the 10x the brochures promise.
The teams I see succeeding with this treat the agent as a very fast, very literal intern. They give it clear goals, they review its work, and they do not let it near production without a leash.
If you are thinking about this for your project
I have been building websites and automation for clients for over six years, and I lead Web Engineering and AI Automation at AKIJ iBOS. If you are trying to work out whether agentic QA makes sense for your stack, the practical next step is to audit your current test coverage and identify the untested long tail, the flows you know are fragile but have never had a test. That audit is useful whether or not you ever run an agent. If you want a second opinion on what it turns up, you can reach me through the contact page on this site and tell me what you are building.
Frequently asked questions
Is agentic QA real or just vendor marketing?
It is real, but narrower than the marketing suggests. Autonomous agents can genuinely plan, execute, and adapt tests, especially exploratory and regression passes. They will not replace your QA team, and they will hallucinate passes if you do not require explicit end-state assertions. Treat them as a fast, literal junior engineer that needs review.
Can I use agentic QA if I am a solo developer or a small agency?
Yes, but keep it lean. Skip the hosted platform and run a self-hosted Playwright MCP server with Claude Code or Cursor. Use it for pre-launch audits and nightly regression on the long tail. Budget 2 to 4 hours a week for triage, and do not point it at production without a kill switch.
How does agentic testing compare to traditional Playwright tests?
They solve different problems. Playwright is deterministic, cheap, and great for known critical flows. Agents are non-deterministic, more expensive per run, and great for exploratory coverage and UI drift. Most teams I work with run both: a lean Playwright suite on every PR, and an agentic pass nightly.
What is the biggest mistake teams make adopting agentic QA?
Going all-in on day one. They point an agent at the whole app, get 200 false positives, and abandon the tool. Start with three flows, review every result for a week, and scale only after you trust the output. The triage discipline matters more than the tool.
How do I stop an agent from reporting a false pass?
Require an explicit end-state assertion in every goal prompt, for example, "the element containing Order # is visible" rather than "the page loaded." Also require the agent to fail loudly on timeouts instead of retrying, and log every element re-location as a healing event that a human must review before the result can go green.
What does a first-week pilot actually look like in terms of hours?
Budget roughly 12 to 16 engineer-hours for week one: about 4 hours writing goal prompts for the three pilot flows, 4 to 6 hours reviewing every run result manually, and 2 to 4 hours wiring the staging environment and kill switch. After that, steady-state triage settles around 4 to 8 hours a week depending on how many flows you run nightly.
Which flows should I never hand to an agent?
Anything involving live payment keys, real customer data, or a CAPTCHA/bot-detection wall, and anything where "correct" is a judgment call like copy, tone, or visual design. Also skip flows with no stable selectors yet, a brand-new UI that changes daily will generate more healing events than useful signal.