Agentic QA: Engineering Quality for Autonomous Test Systems

Quality engineering is entering a structural transition. Traditional automation assumes that humans specify the exact sequence of actions and assertions, while the test runner faithfully executes them. Agentic QA changes that contract: humans define intent, constraints, and approval boundaries, while an AI system plans and adapts the path to the goal.

That shift is more than adding an LLM to a browser test. It changes how test intent is represented, how execution is coordinated, how state is remembered, how evidence is captured, and where humans must remain in control.

The central thesis is simple: coordination is an architecture, not a behavior. Reliable agentic QA requires explicit layers for planning, execution, memory, tools, governance, provenance, and review.

1. From Manual Testing to Agentic Execution

The evolution of QA can be understood as three control models. Manual testing puts execution entirely in human hands. Scripted automation moves execution into code, but humans still dictate exactly how every step should run. Agentic QA moves one level higher: humans define the outcome, while the system reasons about how to achieve it.

This distinction matters because brittle automation is often brittle precisely where it is most explicit. A locator changes, a workflow moves, a page is reorganized, and the script fails even though the user outcome is still achievable.

An agentic system can instead reason from intent: ensure the user can check out, verify that a permission boundary is enforced, or confirm that a critical workflow remains accessible. The implementation path may change while the test objective remains stable.

That flexibility is powerful, but it introduces a new requirement: the system must be constrained well enough that adaptation never becomes uncontrolled behavior.

2. The Emerging Agentic QA Stack

Agentic QA does not eliminate existing test tools. It reorganizes them into a coordinated stack.

Execution tools such as Playwright and Cypress remain useful because they provide deterministic browser control and repeatable assertions. Orchestration platforms coordinate suites, environments, retries, and reporting. Visual AI systems add perceptual checks that are difficult to express as fixed selectors. Autonomous agents contribute planning, adaptive execution, and self-healing behavior.

The architecture therefore becomes layered:

• Execution engines perform concrete actions.
• Orchestration and management coordinate environments, runs, and reporting.
• Visual and semantic analysis extend what can be observed.
• Autonomous agents plan and adapt.
• Human reviewers set intent, approve risky actions, and judge ambiguous outcomes.

The important design choice is not which single tool “wins.” It is how responsibilities are separated so that each component does what it is best at.

3. Replace Procedural Intent with Outcome Intent

Traditional automation encodes a procedure: find element X, click it, then check text Y. That can be appropriate when the interaction is stable and deterministic. It becomes expensive when the UI changes faster than the automation can be maintained.

Agentic QA expresses the goal at a higher semantic level. Instead of instructing the runner exactly which locator to use, the system can pursue an outcome and regenerate the path when the interface changes.

That does not mean all tests should become free-form. A robust strategy combines:

• stable deterministic assertions for safety-critical invariants,
• semantic goals for workflows that change frequently,
• explicit exclusion zones where the agent may not modify state,
• bounded repair policies for locator and navigation failures.

The best candidate for agentic execution is not “everything.” It is the subset of test work where the intent is stable but the path is volatile.

4. Build Readiness Before Autonomy

An organization should not begin with autonomous execution. It should first make the surrounding test system legible to an agent.

A practical readiness review asks whether the environment has:

1. Structured, machine-readable artifacts so requirements and test metadata can be consumed reliably.
2. Clean page-object organization so interface abstractions are consistent.
3. Isolated CI/CD hooks so the agent can run tests without gaining uncontrolled pipeline access.
4. Clear exclusion lists that define what must never be edited, deleted, or bypassed.
5. Unbypassable human-review gates for consequential changes and ambiguous outcomes.

These are not administrative niceties. They are the control surface for autonomy. An agent with rich reasoning but poor boundaries is a liability; an agent with explicit boundaries can become a dependable collaborator.

5. Use AI as a Drafter, Keep a Human Anchor

Many QA tasks begin with messy, unstructured input: meeting transcripts, requirement notes, support tickets, and partial user stories. AI is well suited to converting this material into structured drafts.

The drafting layer can help with transcription, summarization, requirement extraction, and INVEST-style user-story splitting. But draft generation should remain separate from acceptance.

The human anchor ties generated output back to actual project context. Reviewers confirm that the draft reflects the intended behavior, preserves business constraints, and does not silently turn assumptions into requirements.

A useful operating principle is: AI may propose structure; accountable humans approve meaning.

This becomes especially important when generated artifacts drive downstream automation. A flawed user story can produce a flawless test for the wrong behavior.

6. The Core Agent Architecture: Perception, Planning, Memory, Tools

A testing agent is easier to reason about when it is decomposed into four functional systems.

Perception converts the current application state into a representation the model can reason over. That may include accessibility trees, DOM snapshots, visual structure, and structured page state.

The planner uses an LLM or other reasoning model to decide what should happen next. Its job is not to manipulate the browser directly, but to select actions under constraints.

Memory preserves the state that is needed across steps or sessions.

Tools perform concrete operations such as browser control, API calls, file inspection, or test execution. Protocols that expose tools through clear schemas reduce ambiguity and make agent behavior easier to audit.

This decomposition is useful because failures can then be classified. Did the agent perceive the page incorrectly? Plan the wrong sequence? Forget a constraint? Or invoke a tool incorrectly? Without this separation, every failure looks like “the AI was wrong.”

7. Prompt Design Is Part of the Test Architecture

In agentic QA, the prompt is not merely natural-language glue. It is an executable policy surface.

A strong QA prompt has at least four layers:

1. Invariant — unconditional safety rules and non-negotiable constraints.
2. Role — the operating identity, such as explorer, executor, or auditor.
3. Task — the expected output and schema, often JSON or another structured form.
4. Frame — the current DOM, application state, retrieved context, and relevant evidence.

This organization separates long-lived policy from short-lived state. It also makes prompts easier to test because each failure can be traced to a layer: missing rule, wrong role, ambiguous task, or incomplete frame.

For production use, treat prompt changes like code changes. Version them, review them, test them, and tie their outputs to observable execution traces.

8. Progressive Disclosure with SKILL.md

Large agent systems need a way to keep context focused. One pattern is to separate lightweight routing metadata from detailed instructions.

A SKILL.md structure can use YAML front matter to describe the skill, triggers, and routing metadata, while the Markdown body contains the detailed operational instructions that are loaded only when relevant.

This enables progressive disclosure:

• metadata is scanned cheaply and frequently,
• full instructions are loaded only when the skill is selected,
• the working context stays smaller,
• unrelated instructions do not compete for attention.

This pattern also encourages modularity. Instead of one giant system prompt, teams can maintain test-generation, browser-exploration, audit, reporting, and remediation skills as separate reviewed units.

9. Design Memory Deliberately

An agent without memory repeats itself. An agent with uncontrolled memory accumulates confusion. QA systems need explicit memory tiers.

Working memory contains the current step: temporary observations, local variables, and the immediate plan. It should be small and disposable.

Session memory persists across a single test run. It can hold discovered routes, prior actions, failed attempts, and intermediate evidence.

Persistent memory survives across runs and can contain semantic knowledge, vector indexes, known failure patterns, historical test results, and stable project facts.

The separation matters for both correctness and privacy. Not every observation deserves long-term retention. Persistent memory should be curated, permissioned, and version-aware so stale assumptions do not become invisible test dependencies.

10. Move from One Agent to a Reviewed Multi-Agent Workflow

A single agent can plan, execute, and judge its own work, but that concentrates failure modes. A more defensible design separates duties.

A three-role pattern is particularly useful:

• Planner writes the scenario and chooses the intended path.
• Executor runs the browser or API workflow using tools such as Playwright.
• Auditor inspects the execution trace for incorrect assumptions, side effects, and missed evidence.

Separation of duties improves debuggability and creates natural review points. The executor does not need to reinterpret the original goal continuously, and the auditor can evaluate what actually happened rather than what the planner intended to happen.

In higher-risk workflows, the auditor can also act as a policy checker that blocks actions violating explicit constraints.

11. Provenance Turns Agent Output into Evidence

Agentic QA becomes trustworthy only when it can explain where its claims came from.

Every load-bearing conclusion should trace back to a source: a requirement, binding box, execution log, browser trace, screenshot, or structured artifact. This is not merely for postmortems. It is part of the acceptance mechanism.

A useful report should let a reviewer answer:

• Which requirement did this test evaluate?
• Which actions were actually executed?
• What evidence supports the pass or fail decision?
• Which parts were inferred?
• Where did the system encounter ambiguity?
• Which decisions required human confirmation?

The principle is straightforward: trust requires receipts. If the system cannot connect its conclusion to evidence, the conclusion should not be treated as production-grade assurance.

12. Human-in-the-Loop Is a Risk Control, Not a Fallback

Human review should be designed into the architecture before the first autonomous run.

A practical gating rule is to require explicit human approval when a wrong decision would be:

• expensive,
• irreversible,
• security-sensitive,
• destructive,
• difficult to detect before production.

This elevates QA professionals from repetitive executors to editors, approvers, and exception handlers. The agent absorbs routine navigation and synthesis; the human focuses on judgment.

The goal is not to maximize autonomy. The goal is to maximize useful autonomy while keeping consequential decisions governed.

13. Local Models Can Trade Cloud Dependence for Hardware Constraints

Some QA workloads benefit from local inference, particularly when source code, test data, or application state should remain close to the development environment.

The slide gives rough model-to-VRAM sizing bands: smaller 7B–8B models around 6–8 GB VRAM, 14B-class models around 10–12 GB, and 27B–32B models around 16–24 GB. These should be treated as planning heuristics rather than guarantees because real requirements vary with model architecture, quantization, context length, batching, runtime, and hardware backend.

For QA architecture, the practical decision is not simply “local versus cloud.” It is a workload partition:

• local models for sensitive, repetitive, or low-latency tasks,
• larger hosted models for harder planning or broad reasoning,
• deterministic tools for concrete execution,
• explicit policies for what data may leave the local environment.

The best system can route work by sensitivity and capability instead of forcing every task through the same model.

14. Coordination Is the Real QA Architecture

The final design principle is the one that connects the rest.

Agentic QA is not achieved by asking an AI to “be autonomous.” It is achieved by coordinating human oversight, agentic execution, and continuous alignment through explicit interfaces.

The production architecture therefore needs:

• structured requirements and machine-readable artifacts,
• deterministic execution tools,
• perception and planning layers,
• bounded working and persistent memory,
• modular skills and prompts,
• planner/executor/auditor separation where useful,
• evidence and provenance,
• human approval gates,
• ongoing alignment between changing product intent and automated behavior.

Autonomy is only one component. The system around it determines whether it is safe, explainable, and maintainable.

Engineering Principles

The architecture ultimately depends on several principles:

1. Express stable intent, not fragile procedures
   Use semantic goals where workflows change often, while retaining deterministic assertions for invariants.

2. Separate reasoning from execution
   Let models plan and interpret; let well-defined tools perform concrete actions.

3. Treat prompts, skills, and memory as production assets
   Version, test, review, and scope them just as carefully as test code.

4. Make evidence first-class
   Every important test conclusion should be traceable to requirements and runtime artifacts.

5. Bound autonomy with explicit human gates
   The more consequential the action, the stronger the required review.

Final Synthesis

Agentic QA is best understood as a re-architecture of quality work rather than a replacement for existing automation. Deterministic frameworks still execute actions. Structured requirements still define what matters. CI/CD still provides repeatable environments. What changes is the control layer above them: an agent can perceive current state, plan against intent, adapt the path, call tools, use memory, and assemble evidence.

A production-ready system combines:

• deterministic foundations for repeatable browser and API execution,
• adaptive testing layers for workflows whose paths change,
• agent planning and modular skills for reasoning and task decomposition,
• security and governance through exclusions, isolation, and approval gates,
• observability and provenance through traces and requirement links,
• human authority over meaning, risk, and acceptance.

Closing Thought

The defining question for agentic QA is not whether an AI can click through an application. It is whether the complete system can adapt without losing control, evidence, or accountability.

The future of QA is not autonomous testing alone; it is engineered coordination between intent, agents, tools, and humans.