Back to blog
November 9, 2026•32 views

Agentic QA in Practice: Using Autonomous Bug-Catching Agents in Client Automation Projects

agentic QAAI e2e testingautonomous testingMCP serversAI automationNext.jsPlaywrighttest automation
Agentic QA in Practice: Using Autonomous Bug-Catching Agents in Client Automation Projects

Most "agentic QA" articles I read are written by people who have never had a client call them at 11pm because a checkout flow broke in production. They explain what autonomous agents could do. I want to talk about what they actually do when you point them at a real client project with a messy WordPress plugin, a Next.js app, three third-party APIs, and a staging server that occasionally forgets it has a database.

I have spent the last several years building web apps and automation systems for clients across Bangladesh, the USA, the UK, and Australia. Over the past year I have been layering agentic QA and autonomous bug-catching agents on top of my normal Playwright and Jest suites. Some of it has been genuinely excellent. Some of it was expensive theatre. This article is the honest version.

What agentic QA actually means (and what it doesn't)

A normal test suite does exactly what you told it to do. An agentic QA system gets a goal ("make sure a new user can register, verify their email, and complete a purchase") and then figures out the steps itself. It reads the page, decides what to click, handles the cookie banner that appeared last Tuesday, and reports back on whether the goal was met.

That distinction matters more than the marketing suggests. Here is how I separate the categories in my own work:

Level What it does Who owns the logic Good for
Traditional E2E (Playwright/Cypress) Runs fixed scripts with selectors You Stable, high-value flows you control
AI-assisted testing Suggests selectors, heals broken locators, generates test drafts You, with AI help Reducing maintenance pain
Agentic testing automation Plans steps from a goal, adapts to UI changes, explores The agent, within your guardrails Exploratory coverage, fast-changing UIs
Fully autonomous QA Owns the whole cycle: coverage decisions, execution, triage The agent Rarely appropriate for client work without heavy oversight

Most vendors blur levels 2 and 3. When someone says "AI e2e testing" they usually mean level 2 with a chat interface. True autonomous bug-catching agents are level 3, and they are only useful if you build the scaffolding around them.

Why I started using agents at all

Two reasons, both boring and practical.

First, flaky tests. On a client project with a React frontend and a NestJS backend, our Playwright suite was failing roughly a fifth of runs for reasons that had nothing to do with real bugs: animation timing, a toast that covered a button, a date picker that rendered differently in CI. Every flaky failure costs a developer 15 to 40 minutes of investigation. Over a month that adds up to real money.

Second, exploratory coverage. Clients rarely pay for a QA engineer to click around and find edge cases. An agent will happily try to submit a form with emoji in the name field, double-click a payment button, or navigate back mid-checkout. Those are exactly the bugs that reach production.

The architecture I actually deploy

I keep it deliberately simple because client budgets are finite. The shape looks like this:

Goal definition (Markdown spec)
        |
        v
Agent runner (Claude Code / LLM with tool access)
        |
        +--> Browser tool (Playwright MCP)
        +--> DB inspection tool (read-only Postgres MCP)
        +--> Log query tool (read-only log MCP)
        +--> API client tool (OpenAPI-driven)
        |
        v
Structured report (JSON + screenshots + repro steps)
        |
        v
CI job + Slack/email notification + human triage

The key design decision is that the agent gets tools, not credentials. It never touches production directly. It runs against a seeded staging environment with a read-only database user and a scoped API token.

Wiring it up with MCP

MCP (Model Context Protocol) servers are the cleanest way I have found to give an agent controlled capabilities. Instead of hardcoding browser logic, I expose a Playwright MCP server and a small custom server for our own app's domain actions. If you want the full build-out, auth, tool schemas, observability, deployment, I wrote a separate walkthrough on building a production MCP server in Next.js.

A minimal goal spec in the repo looks like this:

# Goal: Guest checkout succeeds with valid card

## Steps the agent should attempt
1. Add any in-stock product to cart
2. Proceed to checkout as guest
3. Fill shipping with a valid BD address
4. Pay with the test card from env
5. Confirm order appears in the account-less confirmation page

## Must not do
- Never use real payment credentials
- Never modify product prices or inventory
- Stop and report if a CAPTCHA appears

## Success criteria
- Confirmation page shows an order number
- Order exists in DB with status 'paid'

The agent reads this, plans, executes through the browser tool, checks the database, and returns JSON. Here is roughly how I call it from a Node script:

import { runAgent } from "./agent-runner.js";

const result = await runAgent({
  goalFile: "goals/guest-checkout.md",
  tools: ["playwright", "postgres_readonly", "logs_readonly"],
  maxSteps: 40,
  budgetUsd: 0.60,
});

if (!result.success) {
  await postToSlack({
    channel: "#qa-alerts",
    text: `Agent failed: ${result.summary}`,
    attachments: result.screenshots,
  });
}

Notice maxSteps and budgetUsd. Both are non-negotiable. Without a step cap, an agent stuck on a modal will loop until your API bill becomes a conversation with your accountant.

The guardrails that keep agents from becoming a liability

The step cap and budget are only the first layer. Here is the full set I run on every client project, in rough order of how often they save me:

  • Scoped credentials. Every tool gets the minimum permission it needs. The Postgres user can SELECT and nothing else. The API token is scoped to a test tenant. If the agent goes rogue, the blast radius is a staging database it can only read.
  • Action allowlists. The custom MCP server exposes named actions (add_to_cart, apply_coupon) rather than a generic "run SQL" or "click anything" primitive. Anything not on the list simply isn't callable.
  • Environment isolation. Agents never see production URLs. Staging is seeded from a fixture snapshot before every run so the agent starts from a known state.
  • Irreversible-action blocks. Refunds, deletions, and email sends are hard-blocked in the tool layer, not just discouraged in the prompt. Prompts are suggestions; tool schemas are laws.
  • Structured, bounded output. The agent must return JSON matching a schema (pass/fail, summary, screenshots, repro steps). Free-form prose is rejected by the runner and retried once, then escalated to a human.
  • Human triage gate. Nothing the agent finds gets filed as a bug automatically. It lands in a #qa-alerts channel with screenshots and a repro, and a person decides whether it's real.

That last point is the one people skip and regret. An agent that files bugs directly into Jira will, within a week, have your client's project manager asking why there are forty tickets titled "button looks weird."

What it costs, honestly

Agentic runs are not free, and the cost is lumpy. A single exploratory run on a mid-sized flow typically lands between $0.15 and $0.60 in model and tool calls, depending on how much the agent has to explore and how many screenshots it takes. A nightly suite of twenty goals is maybe $5, $12 a night. That's cheap next to a QA engineer's hourly rate, but it's not nothing, and it only pays off if the agent is finding real bugs or replacing flaky-test investigation time.

The hidden cost is the one nobody budgets for: the time to write good goal specs and guardrails. The first time you do this on a project it's a day or two of work before the agent is trustworthy. After that, each new goal is maybe twenty minutes.

When autonomous testing is a waste of money

I'll be blunt, because this is the part the vendor blog posts leave out. Agentic QA is a bad fit when:

  • Your flows are already stable and well-covered. If your Playwright suite is green and fast, an agent adds cost and a new failure surface for little gain.
  • The app has heavy anti-bot defenses. CAPTCHAs, bot fingerprinting, and aggressive rate limits turn the agent into an expensive way to get blocked. You'll spend more time whitelisting than testing.
  • The domain is high-stakes and irreversible. Anything touching real money movement, medical data, or legal compliance needs deterministic, auditable tests, not a model deciding what to click.
  • You can't give it a clean, seeded environment. An agent exploring a dirty database will report a dozen "bugs" that are just stale fixtures.
  • The client won't pay for exploratory coverage. If the budget only covers "make sure the login works," write a Playwright test and move on.

The honest summary: agentic QA is a force multiplier on top of a solid deterministic suite, not a replacement for one. The teams that get value from it already have their critical paths covered by real tests and use agents to explore the long tail.

Where it has genuinely paid off

Two concrete wins from the past year:

  1. A checkout regression caught before launch. An agent, given a goal to "complete a purchase with a discount code applied after adding a second item," found that the discount silently dropped when the cart total changed. No human had tested that combination. It would have shipped.
  2. Flaky-test investigation cut dramatically. Instead of a developer babysitting a failing run, the agent re-ran the flow, captured a screenshot at the failure point, and reported "toast element intercepted the click at step 4." That's a two-minute fix instead of a forty-minute hunt.

Neither of those required "fully autonomous QA." Both required a tight goal, good guardrails, and a human reading the report.

The takeaway

Agentic QA is real and it works, but it works the way most powerful tools work: in a narrow band, with sharp edges, and only when you've done the unglamorous setup first. Give the agent tools instead of credentials, cap its steps and budget, isolate its environment, and keep a human in the triage loop. Do that, and autonomous bug-catching agents earn their keep. Skip it, and you've built an expensive machine for generating noise.

If you're wiring this into a Next.js stack, the production MCP server walkthrough covers the plumbing in detail. And if you want the short version: start with one goal, on one flow, on staging, with a step cap of twenty. See what it catches. Then decide whether to scale it.

Frequently asked questions

What is agentic QA, in plain terms?

It's a testing setup where an AI agent is given a goal (like "a guest can complete checkout") and figures out the steps itself, reading the page, deciding what to click, adapting to UI changes, instead of running a fixed script. It's most useful layered on top of a solid deterministic test suite, not as a replacement for one.

Do autonomous bug-catching agents replace Playwright or Cypress?

No. Traditional E2E tests stay your source of truth for critical, stable flows. Agents are best for exploratory coverage and for investigating flaky tests. Think of them as a force multiplier on top of your existing suite.

How do you keep an agent from doing something dangerous?

Give it tools, not credentials. Use read-only database users, scoped API tokens, action allowlists on your MCP servers, hard blocks on irreversible actions, a seeded staging environment, step and budget caps, and a human triage gate before anything is filed as a bug.

How much does agentic QA cost to run?

A single exploratory run on a mid-sized flow typically costs $0.15, $0.60 in model and tool calls, and a nightly suite of twenty goals runs around $5, $12 a night. The bigger hidden cost is the day or two of upfront work to write good goal specs and guardrails.

When is autonomous testing a waste of money?

When your flows are already stable and well-covered, when the app has heavy anti-bot defenses, when the domain is high-stakes and irreversible, when you can't provide a clean seeded environment, or when the client's budget only covers basic smoke tests. In those cases a deterministic test is the better spend.

Share this post

Native Web Share is available on mobile, so that share sheet includes Instagram and any other app you have installed.

Let's Connect

Get In Touch

Let's build something premium together.

Contact info

Follow me
Chat on WhatsApp