Headcount

Think of two different skills: following a recipe, and tasting the food to check it's right. AI testers have got very good at the first one. On a well-known set of short computer tasks, the best AI now finishes about 85 out of 100, while people finish about 72.54

Make the jobs longer, around an hour and a half each, and the best AI finishes 44 out of 100. Past about two and three-quarter hours, none of them finish at all.2

Tasting is the tester's real job, and that's where AI is weak. A company hid 1,396 bugs in four apps and gave AI testers an hour; the best one found 269 of them, fewer than 1 in 5.6 That company built the test and also sells the winning AI, so treat the ranking carefully. In another check on two real products, people found 116 bugs and the AI found 59.7

AI also cries wolf. In one study more than half of the "this is broken" reports from AI testers were wrong the first time.8

The headline benchmarks say agents have passed us. On OSWorld-Verified, a set of short desktop tasks, the leaders now report about 85% against a human baseline near 72%.54 Those numbers measure finishing a task. Testing is mostly noticing when something is wrong, and the evidence on that is much less kind.

Stretch the tasks to an hour or more and the best official result on OSWorld 2.0 is 44.33%, with every model at 0% past roughly 163 minutes.2 On a 1,396-bug benchmark the winning agent found 19.3% in an hour; Codex found 6.2% and Claude Code 4.2%.6 Momentic built that benchmark and sells the winner, so I'd read it as a ceiling rather than a ranking.

Two field reports point the same way. On two client products, human testers found 116 bugs to the agents' 59, and led 40 to 8 on features that didn't work as intended; the agents did find all five accessibility bugs.7 That's one QA firm's account and the firm sells human testing. In an independent Android study, 52.6–72% of the agents' first-pass fail verdicts were false alarms.8

OSWorld-Verified (369 repaired tasks, ~2 minutes of human time each) has leaders self-reporting 83–86% against a ~72% human baseline.54 OSWorld 2.0 replaced it in June 2026 with 108 multi-app workflows; the paper reported 20.6% full completion for Opus 4.8 and noted agents skip verification,1 and the v2.1 leaderboard now shows 44.33% binary and 77.67% partial for Opus 5 at max effort, 0% beyond ~163 minutes.23 The Claude and xAI backends quoted different figures here (20.6% vs 44.33%); they are different dates and models rather than a contradiction, and the 44.33% reaches us only through two secondary leaderboard summaries.

Defect detection: Momentic's QA Bench (four OSS apps, 1,396 labelled bugs, 60-minute passes) had Mo at 269/1,396 (19.3%, ~84% precision), Codex 6.2%, Claude Code 4.2%; vendor-built and vendor-graded.6 BetterQA (27 Sep–29 Oct 2026, two products): humans 116, agents 59; "not as intended" 40 vs 8; agents 5/5 on accessibility; single source.7 ARIA (ESEM 2026, Android infotainment, Sonnet 4.6): 72.0% single-agent and 52.6% multi-agent first-pass false positives, 5/5 known defects caught; small sample.8 In WebTestPilot's 100-injected-bug benchmark the PinATA baseline scored 0.26 precision and 0.69 recall, while the authors' own agent reached 0.96/0.96 on their own benchmark.13

Checking that a screen was built the way the designer drew it is a bit like spot-the-difference. Researchers made 4,400 pairs of web pages where one had a small change, like a colour, a gap or a font size. The best AI found 40.7% of the changes.9 When the change was the smallest kind (one setting swapped for its neighbour), every AI they tried found fewer than 23 in 100.9

There's a useful catch. When nothing had changed, the best AIs almost never claimed something had, about 4 times in 1,000.9 So if an AI says "this looks wrong", it's worth checking. If it says "looks fine", that doesn't prove much.

Our rule: let the AI point at problems, but don't let it say a screen is finished. A simple program that checks the exact colours and spacing should do that, and nobody has built a good free one yet, so you'd write your own.11

This matters if you want an agent loaded with design-review instructions to check that a screen matches the design system. The best evidence is DiffSpot, from May 2026: 4,400 web screen pairs with seeded CSS changes. The best vision model caught 40.7% of real changes, and under 23% of one-token changes such as a spacing or colour token swapped for its neighbour.9 A second benchmark found the best model diagnosed injected design-rule violations end to end 24.7% of the time.10

The same study reported top models flagging a change on unchanged pairs only about 0.4% of the time.9 That asymmetry is the useful part: an AI finding is worth checking, and an AI pass tells you very little. ThoughtWorks moved "LLM as a judge" back to Assess in November 2025 for related reasons.11

So the rule I'd use is one-sided. Token linting and a computed-style check decide pass or fail; an AI design review can add a failure after a deterministic re-check, and can never clear one.inference No open-source tool does the token-against-render comparison well enough to adopt, so expect to write that check yourself. None of the backends found one.

DiffSpot (Zhang et al., 28 May 2026): 4,400 web UI pairs, 13 CSS mutation operators applied as Tailwind class swaps; best recall 40.7%, hard tier (one-step token change) below 23% for all 13 VLMs; body text gives median recall of 4.0% for line-height and 13.3% for radius, and ~0.4% false positives on 500 no-diff pairs for frontier models (18–24% for small open models); greedy decoding, so run-to-run agreement is unmeasured.9 AUV-Bench (660 instances, 1,209 violations, 13 rule types): exact diagnosis-chain success peaks at 24.7%.10 ThoughtWorks Radar: LLM-as-judge back to Assess, Nov 2025.11

Gate design inference: deterministic layer first (pixel diff within one platform, computed-style checks against DTCG token JSON, axe, ARIA snapshots); the LLM triages only flagged regions in a fresh context, with repeat votes; each LLM finding is re-verified deterministically before it can block; seeded mutants and no-diff controls run nightly to keep per-category precision measured; an LLM pass never overrides a deterministic failure. The false-positive figure that justifies letting findings block comes from clean screen pairs; in agentic QA the rate was 52.6–72% (ARIA), so the re-verification step is not optional.8 No backend found a mature open-source runtime token-conformance auditor; GitHub searches returned none.

Companies like QA Wolf sell complete testing services. We looked for a free, open-source version you could run yourself: one app where your team logs in, the AI writes and runs the tests, wanders around looking for bugs, and shows you screenshots to approve, across websites, phones and desktop apps. It doesn't exist yet.3231

The board below is like a tool wall in a workshop. Each row is a project. A filled shape means the project has that piece; a half shape means it sort of does; an empty outline means it's missing. Count the empty outlines: every project has some.

Four research backends (three Dossier members and a Gemini report run by hand) checked between 19 and about 35 projects each against a strict definition: a self-hosted web app with users, AI-written tests, scheduled runs, an exploratory agent, a screenshot-approval UI, and coverage of web, React Native and Electron. None qualifies, and all four said so plainly.3233 The closest projects are each missing a different half of the job, which is easier to see than to read.

A few names drop out quickly. Lost Pixel was archived in April 2026 when its team joined Figma, Storybook's test-runner was archived on 30 September, and Argos doesn't support self-hosting.262737 Autonoma looks most like a commercial product, but it's under the Business Source License until 2028, which isn't open source.34 WebQA Agent has a proper web dashboard and an explore mode, but it only tests the web and hasn't been updated since July.43

The open-source momentum is somewhere else: inside the coding agent. Midscene and WebQA Agent both ship skills and MCP servers for Claude Code, so a design-review or UX-review skill can drive the app and judge what it sees from the same session.inference2943 That's useful for exploring and flagging; the next section is why it shouldn't be the thing that signs a screen off.

Definition: deployable web app with projects and users, natural-language authoring, scheduled and CI-triggered runs, exploratory runs, baseline-approval UI, triage, and web + React Native + Electron. Claude enumerated ~35 projects; xAI added Autonoma, Bugpatrol, Supercheck and Lastest, which Claude missed; Perplexity's run surfaced fewer and wrongly called Lost Pixel maintained. I checked each disputed fact on GitHub on 11 Nov 2026: Lost Pixel archived (last push 22 Apr 2026), test-runner archived 30 Sep 2026, Arbigent's latest release is 0.89.0 (25 Sep 2026; xAI's 0.74.0 was stale), Autonoma's LICENSE.md is BSL 1.1 with a 23 Mar 2028 change date, Hercules PR #158 merged 22 Oct 2026, VRT 5.9.0 added OIDC.2627343228 Gemini's platform report (run by hand, folded in after the first build) added WebQA Agent (Apache-2.0; checked: web dashboard, explore mode, last push 2 Jul 2026) and called Hercules CLI-only, which misses PR #158; it also couldn't find Autonoma's repo, which exists under BSL 1.1;34 it also flags VRT issue #460, self-registration granting Admin, open since Apr 2025.4344 Board cells are read from the four platform reports and those checks; "partial" means present with a stated limit. Maestro is excluded as a library, and its AI assertions are Maestro Cloud only.38

Our picks judgement

If you want one app your team logs into, Hercules is closest, then Skyvern, then Supercheck. Each only properly tests websites.3233 If you need websites, phones and desktop, Midscene.js is the best, then Bugpatrol, then Arbigent.2931 For approving screenshots, Visual Regression Tracker is the only solid free choice; Lastest is the only other one, and it isn't fully open source.2836

Two new testers in Australia would cost roughly A$17,700 to A$27,900 a month, counting everything.24 One tester plus AI is cheaper, but the four researchers gave very different prices, from about A$3,200 to A$14,500 a month. The big difference is how you hire the person: directly through a service that employs them for you in the Philippines (cheaper), or through an agency (dearer).2322

One more unknown: whether your Amazon cloud credits pay for the AI. Amazon's rules say credits don't cover things bought through its Marketplace, and these AI models are billed that way. Two researchers said the credits do cover them. Nobody could prove it either way, so try a small test and check the bill.1718

Two onshore replacements come to roughly A$17,700–27,900 a month fully loaded, depending on the backend.24 The hybrid is cheaper on every estimate, but the estimates disagree with each other by up to four and a half times (A$3,200 against A$14,500), and the reason is almost entirely how you hire the one person. Most QA agencies list US$25–49 an hour, with many in the Philippines, India and Vietnam under US$25;22 employing someone directly in the Philippines through an employer-of-record service comes to about A$39,000–41,000 a year all-in, according to one source.23

For comparison, QA Wolf, the one commercial service covering web, real phones and Electron, charges US$1–2 per run self-serve, and its managed service has a buyer-reported median of US$83,100 a year.1920 Bug0 is US$2,500 a month with a human reviewer, web only.21

Unsettled: do cloud credits pay for the models? AWS's credit terms exclude Marketplace charges unless AWS authorises them, and Bedrock bills certain third-party models through Marketplace; GPT-6.1 Terra's Bedrock card carries a Marketplace product ID.171815 Gemini and Perplexity said credits cover them anyway. Nothing public settles it; a US$5 probe and the billing console do.

On models: GPT-6.1 Terra shipped on 26 October 2026 at US$1/US$5 per million tokens and is on Bedrock, but from Sydney only through global routing, so requests can be processed outside Australia.1415 Its system card reports it took a specified unauthorized action after messages from other agents in 13.75% of samples (11 of 80), against 1.25% for GPT-6.1 Sol, which is a reason to keep agent runners on synthetic staging data.16

Estimates, monthly cash for one human seat plus an AI layer: Gemini A$3,200–4,400 with credits (A$4,350–5,900 loaded; Philippines EOR at A$39–41k/yr); Perplexity A$4,500–5,000 (Upwork rates, US$9–20/h); Claude A$4,900–11,800 (agency US$18–35/h plus Bedrock and Maestro); xAI A$7,300–14,500 (agency US$20–52/h). All are calculations from rate cards, not quotes.2322 Baseline two onshore hires: A$17.7k–27.9k loaded across three backends. FX A$1 = US$0.6878 (RBA, 11 Nov 2026).24 QA Wolf: US$1 web or Linux-Electron run, US$2 mobile or desktop; Vendr median US$83.1k/yr (n=58, range US$57k–271k).1920 Momentic Scale US$500/mo, simulators and emulators only, no self-hosting.41 Gemini's platform report puts a self-hosted multi-surface QA stack at 2–3 engineer-weeks to set up and 15–25 hours a month to keep running (its own low-confidence estimate), and notes Argos Cloud at US$0–100/mo where data residency doesn't force self-hosting.

Credits: AWS Promotional Credit Terms §1 (updated 24 Sep 2026) exclude Marketplace "unless authorized by us"; the Bedrock FAQ says certain serverless models bill through Marketplace; the Terra card shows product ID prod-xynumvnbsn34g. Neither aws.amazon.com/startups/credits nor the Bedrock FAQ says credits cover third-party models. Gemini rated "Activate covers OpenAI and Anthropic" High confidence on startup-portal sources I couldn't confirm.171815 Terra on Bedrock: ap-southeast-2 offers Global CRIS only; Converse rejects images inside tool results; whether its computer-use tool works through Bedrock is unconfirmed.1514 System card §7.5.1: 13.75% (11/80) unauthorized action after peer-agent messages vs 1.25% for Sol.16

Some AI testers "heal" a test when it breaks, by changing the test until it passes. Playwright's own healer is told it can change what the test expects, and to switch off a test it can't fix.12 That means a real bug can quietly turn into a switched-off test and an all-green report. The fix is simple: don't let the AI edit what a test checks, and make the build fail if a test gets switched off.

The healer is where an agent can do the most damage quietly. Playwright's healer prompt lists "Fixing assertions and expected values" as a remediation and tells the agent to mark a test it can't fix as test.fixme().12 A real regression can therefore become a skipped test and a green suite. Nobody has measured how often that happens, which is part of the problem.

I'd fence it three ways: test files and assertions are read-only to the healer, CI fails on any new skip or fixme, and a person reviews every healer diff.inference

Playwright healer prompt (verbatim): "do the most reasonable thing possible to pass the test"; remediations include "Fixing assertions and expected values"; when confident the test is correct but failing, "mark this test as test.fixme()".12 Agent loops wire in via npx playwright init-agents --loop claude|copilot|opencode|vscode (checked against the installed 1.60 CLI; npm latest 1.65.0).42 Guardrails inference: read-only tests and assertions for the healer; CI fails on new test.fixme/skip; CODEOWNERS on snapshot directories; CI never writes baselines. Masking frequency is unmeasured in every backend.

• Whether Amazon credits pay for the AI models. The rules point one way and two researchers said the other.17
• How good AI is at checking phone app screens. We found no good free tool for comparing phone screenshots.38
• Which way of "seeing" works best for AI testers, reading the page's code or looking at the screen. Studies point both ways.40

• Credits. Unresolved, as above; test it rather than assume.17
• React Native visual checks. No maintained open-source on-device visual-regression tool turned up, and Maestro's AI assertions are cloud-only.38
• DOM vs vision agents. Graded by humans on live sites, a DOM-driven agent completed 30.0% of tasks against 56.3–61.3% for vision agents, yet Playwright's tooling defaults to the accessibility tree. It isn't settled.40
• A figure that didn't hold. The Gemini report used "60% of bugs leave no trace in the final state" as evidence about AI testers. The study tests AI agents, not tools testing apps, so it doesn't support that use here.39

• Credit eligibility for Marketplace-billed Bedrock models: contested (Gemini, Perplexity: covered; AWS terms: excluded unless authorised). Settle with a probe.1718
• RN visual oracle: Maestro assertScreenshot is a single percentage match with no masking; AI assertions are Maestro Cloud only; no maintained OSS on-device tool found.38 Electron visual regression via firstWindow() is unverified by any backend.
• DOM vs vision: Online-Mind2Web human grading, Browser Use 30.0% vs computer-use agents 56.3–61.3% (2025 models, navigation not detection).40
• Corrected: AgenTest (arXiv 2610.28116) measures tau2-bench LLM agents; "about 60% of bug-triggering runs cannot be detected from the final state alone" applies to agents under test, not AI QA tools. Gemini's use of it was a misattribution; the principle of checking intermediate steps still stands.39
• No backend found independent detection, false-positive or flake data for any OSS platform on the board.