# 75% Single-Run Accuracy Drops to 42% Across Three Trials: How Growth Companies Build AI Agent Evaluations and QA for Production

> AI agent evaluation and quality assurance: 75% single-run accuracy drops to 42% over 3 trials. Learn how growth companies build automated evals.

- Published: 2026-10-18
- Author: Master Mind
- Canonical: https://aimaster.fi/en/artikkelit/75-prosentin-tarkkuus-putoaa-42-prosenttiin-tekoalyagenttien-evaluointi-ja-laadunvarmistus

An AI agent that succeeds 75% of the time on a single test run will succeed across three consecutive runs only 42% of the time (Anthropic, 2026). That mathematical gap explains why an agent that looks impressive in a boardroom demo starts generating customer complaints the week it connects to your live ERP. Systematic AI agent evaluation and quality assurance is the architectural boundary that separates fragile prototypes from production systems.

According to Statistics Finland and Eurostat, 66% of Finnish companies already use generative AI tools, the highest rate in the EU (2025). Yet most growth companies still test their agents by typing five prompts by hand and eyeballing the answers. When you change a system prompt or upgrade the underlying language model, manual testing leaves your team flying blind.

## Why does a 75% single-run score fail in production?

Single-run accuracy hides compounding failure across repeated business workflows. In Anthropic's January 2026 study Demystifying evals for AI agents, researchers contrast two reliability metrics: pass@k, which measures the probability of at least one success in k attempts, and pass^k, which measures the probability that all k trials succeed. At a 75% per-trial rate, pass^3 drops to (0.75)^3, or 42% (Anthropic, 2026).

Anevaluation ("eval") is an automated test where an AI system receives a defined task input and a grading script scores both its reasoning trace and its final output. Without automated evals, teams get trapped in reactive debugging loops: a customer reports a failure, an engineer patches the prompt, and that fix silently breaks three other workflows.

The difference between developer tools and autonomous business operations comes down to which metric you optimize. When a software engineer uses a coding assistant, one working suggestion out of three tries (`pass@3`) saves time. When an agent processes incoming purchase orders, customer refunds, or warehouse claims, every single run must succeed (`pass^k`). Slide-deck consultants rarely mention that the tenth consecutive trial of a 75% agent passes less than 6% of the time.

| Metric or Concept | What It Measures | Business Impact in a Growth Company |
| --- | --- | --- |
| pass@k (at least 1 success in k tries) | Probability that at least one attempt out of k produces a valid result (Anthropic, 2026) | Fits human-assisted drafting where a specialist picks the best option |
| pass^k (all k trials succeed) | Probability that every consecutive trial succeeds; 75% single-run drops to 42% at k=3 (Anthropic, 2026) | Mandatory benchmark for customer-facing agents and automated ERP/CRM workflows |
| Capability evals (quality ceiling) | Hard tasks starting at a low pass rate to measure what the agent can achieve (Anthropic, 2026) | Reveals immediately whether a newly released model unlocks complex workflows |
| Regression evals (stability lock) | Previously mastered tasks requiring a near-100% pass rate on every change (Anthropic, 2026) | Prevents prompt edits and API updates from breaking live production processes |

## What should automated AI agent evaluation and quality assurance verify?

Automated AI agent evaluation and quality assurance must verify two separate layers: the execution transcript and the final environment outcome. According to Anthropic (2026), an agent may tell a customer that a refund or reservation is complete in the conversation transcript while the actual SQL database or ERP state remains unchanged. Grading only the text reply creates false confidence.

In production architectures, evaluations combine three types of graders (Anthropic, 2026). Choosing the right mix keeps testing fast, objective, and affordable:

- Code-based graders: Deterministic scripts that check exact database states, tool call parameters, static analysis rules, and response latency. They cost almost nothing to run and execute on every code commit. When [structured outputs and JSON schema make AI integrations reliable in production](https://aimaster.fi/en/artikkelit/structured-outputs-ja-json-skeema-tekoalyintegraatiossa-estavat-rikkinaiset-erp-kutsut), code-based graders verify schema compliance automatically.
- Model-based graders (LLM-as-judge): A separate language model grades open-ended dimensions such as tone, completeness, and source grounding against a strict rubric. When [API-level citations tie every AI agent claim to the exact source passage](https://aimaster.fi/en/artikkelit/hallusinoinnin-loppu-dokumenttiprosesseissa-nain-api-tason-lahdeviittaukset-sito), a model grader checks that the agent did not extrapolate beyond the cited paragraph.
- Human graders (domain calibration): Subject-matter experts do not read thousands of logs manually; instead, they audit a sample of transcripts to calibrate the LLM-as-judge rubrics and define reference solutions.

Anthropic (2026) also highlights a critical trap: rigid graders that punish valid solutions. In one benchmark audit (CORE-Bench), Claude Opus 4.5 initially scored 42% because the grading script rejected "96.12" when expecting "96.124991..." and had ambiguous file path rules. Once the broken grading rules were fixed, the model's score jumped to 95% (Anthropic, 2026). Grade the final business outcome in the target environment, not an overly rigid sequence of intermediate steps.

## How do you build an evaluation suite without months of engineering overhead?

A growth company does not need hundreds of test cases before launching an AI agent. According to Anthropic's 2026 engineering guide, 20 to 50 unambiguous test cases drawn from real failure modes and manual pre-release checks are enough to establish a reliable baseline. Teams with an automated eval suite can validate a new frontier model and upgrade production in days instead of weeks.

At AIMASTER, evaluation-driven development is built directly into our productized path so your team never deploys an unverified agent into core operations. **[Master Plan](https://aimaster.fi/tuotteet/master-plan)** is an AI strategy sprint that maps where AI generates the most value for your company — measured in euros. During this phase, we also define the exact pass/fail business criteria and target `pass^k` thresholds for your highest-value workflows.

**[Master OS](https://aimaster.fi/tuotteet/master-layer)** is the company's AI Operating System: a platform that safely connects your existing systems (CRM, ERP, documents), data, and agents for AI use. We embed the isolated evaluation harness directly into Master OS so every trial starts from a clean state without leftover cache or database collisions. **[Master Mind](https://aimaster.fi/tuotteet/master-mind)** is a suite of AI agents that runs on top of Master OS and handles business processes autonomously. **Custom AI solutions** are delivered using an agile sprint model: one sprint is 3 development days.

- Sprint 1 (20–50 baseline cases): Collect the 20–50 most common edge cases from your support queue, ERP exceptions, and manual checks. Pair each task with a known working reference solution and balanced negative cases (when the agent should act vs. when it must escalate).
- Sprint 2 (Outcome and rubric automation): Connect deterministic state checks to your sandbox environment and configure isolated LLM-as-judge graders with an "Unknown" escape hatch to prevent judge hallucinations (Anthropic, 2026).
- Continuous CI/CD and transcript audits: Promote high-scoring capability evals into your permanent regression suite and review a weekly sample of failed transcripts to keep the suite aligned with live production monitoring.

In our public reference deliveries — such as the Aini AI assistant built for entrepreneur Jaajo Linnonmaa and automation workflows for digital marketing agency Tagomo — reliability in daily use depends on repeatable behavior across real user inputs, not one-off demo prompts. When your evaluation suite runs automatically on every change, your team upgrades models in days and scales automation with mathematical confidence. Start by identifying which processes in your company have the highest euro potential and what reliability bar they require: book a [Free Master Mind Analysis](https://aimaster.fi/analyysi).

## Frequently asked questions

### What is the difference between pass@k and pass^k in AI agent evaluation?

Pass@k measures the probability that an AI agent succeeds at least once in k attempts, making it suitable for developer tools where one working draft is enough. Pass^k measures the probability that the agent succeeds across all k consecutive trials. If single-run accuracy is 75%, pass^3 is (0.75)^3 = 42% (Anthropic, 2026). Customer-facing business processes require measuring pass^k.

### How many test cases does a growth company need for its first AI agent evaluation suite?

You do not need hundreds of synthetic test prompts to start. According to Anthropic's 2026 evaluation research, 20 to 50 concrete test cases drawn directly from real error logs, support tickets, and manual release checks are enough to catch regressions and validate model upgrades in early production.

### Why must AI agent evaluations verify database outcomes instead of just the conversation transcript?

Transcript grading inspects the path the agent took, including reasoning steps, tool calls, and token consumption. Outcome grading verifies the final state in the target environment, such as confirming whether a credit note or order row was actually written to the ERP database rather than trusting the agent's text reply (Anthropic, 2026).

### When should you use deterministic code graders versus LLM-as-judge graders?

Deterministic code-based graders check exact rules such as JSON schemas, database states, and tool parameters fast and at near-zero cost. Model-based graders (LLM-as-judge) use a separate language model and a structured rubric to score open-ended qualities like tone, groundedness, and completeness, calibrated periodically by human domain experts.