Agent Evaluations: A Practitioner's Framework for Engineers

Agent evaluations measure end-to-end agent behavior, treating the full execution trajectory as the unit of measurement, not just the final output. If you take one thing from this article, it is this: teams that instrument trajectories and run persistent, automated regression suites catch regressions before they ship; teams that skip evals discover failures in production, one incident at a time.

Three things to lock in before you write a single grader:

• Define objectives and success criteria first. Every metric you collect should trace back to a stated objective. Without this, you end up with dashboards full of numbers and no clear signal.
• Select graders per objective. Code-based (deterministic) graders for structured outputs, LLM-as-judge for subjective or high-coverage checks, and human-in-the-loop for safety-critical or ambiguous labels.
• Control scaffolds and environments. Prompt templates, parser versions, and tool stubs must stay constant across runs so observed changes are attributable to the agent, not the harness.

Pro Tip: The highest-leverage investment in any evaluation program is a persistent, automated regression suite that runs on every pull request. Anthropic's engineering team documents that without it, teams fall into reactive debugging loops where regressions hide until they break something else.

---

Key Takeaways

Effective agent evaluations measure full execution trajectories, combine grader types per objective, and run persistently in CI to catch regressions before they reach production.

Point	Details
Measure trajectories, not outputs	Instrument every tool call, plan, and intermediate step; final output alone cannot support root-cause analysis.
Match grader type to objective	Use deterministic code checks for structured outputs, LLM judges for rubric scoring, and human review for safety-critical labels and calibration.
Control scaffolds for reproducibility	Record scaffold version, seed, and environment snapshot with every run so metric changes are attributable to the agent, not the harness.
Separate regression from capability suites	Regression suites target near-100% pass rates; capability suites explore new territory. Mixing them produces alert fatigue and hides real regressions.
Integrate CI gates and production monitoring	Run smoke tests on every PR, nightly capability sweeps, and continuous production sampling to catch regressions at two checkpoints.

---

Table of Contents

• What does an agent evaluation actually consist of?
• Which grader type should you use, and when?
• What metrics should you actually measure?
• How do you build a trustworthy evaluation pipeline from scratch?
• How do you get reproducible results when agents are non-deterministic?
• How should you tailor evals for different agent types?
• How do you scale evals without scaling your human review budget?
• How do you turn failing evals into engineering tasks?
• What tools map to which eval components?
• Running agent-targeted evaluations on a multi-agent deployment
• Production lessons and what actually matters in practice
• Sources
• FAQ

What does an agent evaluation actually consist of?

An agent evaluation is a pipeline with five distinct components, each with a specific responsibility. Conflating them is the most common source of confusing results.

Dataset (test items). A curated set of task inputs, each with enough context to initialize the agent's environment. Good test items cover representative tasks, edge cases, and known failure modes. They are versioned and stored alongside the eval harness.

Harness / runner. The orchestration layer that loads test items, initializes the agent runtime, captures the full execution transcript (plans, tool calls, intermediate reasoning, final output), and routes results to graders. The harness also manages environment setup: API stubs, sandbox containers, seed values.

Agent runtime (instrumented). The agent itself, running inside a controlled environment. Instrumentation hooks emit structured telemetry at each step: tool call name, arguments, return value, token count, latency, and any intermediate reasoning traces.

Grader layer. One or more evaluators that score each transcript against defined criteria. Graders can be deterministic code, LLM judges running a rubric, or human reviewers. Multiple graders can run in parallel against the same transcript.

Aggregator and result store. Collects per-item grader outputs, computes aggregate metrics (pass rates, mean scores, cost totals), and writes results to a queryable store. Dashboards and CI gates read from here.

A minimal eval run should emit a structured record for every test item. Here is a representative JSON schema:

{
  "run_id": "eval-2026-06-01-abc123",
  "item_id": "task-042",
  "transcript": [
    {"step": 1, "type": "plan", "content": "..."},
    {"step": 2, "type": "tool_call", "name": "search_web", "args": {...}, "result": {...}},
    {"step": 3, "type": "output", "content": "..."}
  ],
  "grader_results": [
    {"grader": "task_success_code", "score": 1, "pass": true},
    {"grader": "coherence_llm_judge", "score": 4, "rubric_dim": "coherence", "pass": true},
    {"grader": "safety_flag", "score": 0, "pass": true}
  ],
  "per_dimension_scores": {"task_success": 1.0, "coherence": 0.8, "safety": 1.0},
  "overall_pass": true,
  "metadata": {
    "model": "claude-opus-4",
    "scaffold_version": "v1.4.2",
    "seed": 42,
    "tokens_used": 3847,
    "latency_ms": 4210
  }
}

Component	Responsibility	Key output
Dataset	Provide task inputs and ground truth	Versioned test items
Harness	Orchestrate runs, capture transcripts	Structured run records
Agent runtime	Execute tasks under instrumentation	Telemetry, tool call logs
Grader layer	Score transcripts per dimension	Per-item grader results
Aggregator	Compute metrics, write to store	Aggregate pass rates, scores

---

Which grader type should you use, and when?

Anthropic's engineering guidance identifies three grader types that together cover the full range of evaluation needs. Each has a distinct cost, coverage, and reliability profile.

Code-based (deterministic) graders

These are unit-test-style assertions written in Python, JavaScript, or any language your harness runs. They check exact output values, schema conformance, state changes in a database or file system, or whether a specific tool was called with the correct arguments. Deterministic graders are fast, cheap, and perfectly reproducible. They are the right choice for any objective with a ground-truth answer: did the agent write a file to the correct path? Did it call the API with the right parameters? Did the returned JSON validate against the schema?

Their weakness is coverage: they cannot evaluate coherence, factual accuracy, or nuanced reasoning without brittle string matching.

Model-based graders (LLM-as-judge)

A rubric-driven LLM judge reads the transcript and scores it against a set of named dimensions, each with a description and a scoring scale. Azure Foundry recommends pairing rubric evaluators with built-in evaluators for safety and coherence, then integrating both into CI/CD pipelines. LLM judges scale well: you can run them against thousands of transcripts overnight at a fraction of the cost of human review.

The risk is calibration drift. An LLM judge that has not been validated against human labels will produce scores that look plausible but diverge from what a human would actually flag. Research on LLM-based evaluation methods confirms that automated judges require periodic human auditing to avoid false positives and negatives accumulating over time.

Human-in-the-loop graders

Human review is necessary for three situations: high-stakes or safety-critical labels where an error has real consequences, ambiguous cases where the rubric does not clearly resolve the score, and calibration of LLM judges. For calibration, the standard workflow is to have human reviewers score a representative sample (typically 100–200 items), compute inter-rater agreement, then compare LLM-judge scores against the human labels. Where disagreement exceeds a defined threshold, revise the rubric or the judge's system prompt.

Pro Tip: Set an explicit escalation rule in your harness: any item where the LLM judge's confidence score falls below a defined threshold, or where two LLM judges disagree by more than one scale point, routes automatically to the human review queue. This keeps human effort focused on genuinely ambiguous cases.

Grader type	Speed	Cost	Coverage	Best for
Code-based	Very fast	Very low	Structured, exact outputs	Schema checks, state assertions, tool-call validation
LLM-as-judge	Moderate	Moderate	Subjective, high-volume	Coherence, coverage, claim grounding, rubric scoring
Human-in-the-loop	Slow	High	Ambiguous, safety-critical	Calibration, high-stakes labels, edge cases

---

What metrics should you actually measure?

NVIDIA's technical guidance makes the point directly: high scores on foundation-model benchmarks like MMLU do not predict agent reliability. Agent evaluation must measure system behavior across the full workflow, including planning quality, tool use accuracy, and trajectory efficiency.

The five primary objective categories, with sample metric definitions:

Task completion. Task Success Rate (TSR) = (number of tasks where the agent fully resolved the stated intent) / (total tasks). This is the headline metric, but it hides failure modes, so always decompose it by task type and difficulty tier.

Capability metrics. Tool Call Precision = (correct tool calls) / (total tool calls made). Tool Call Recall = (correct tool calls) / (total tool calls that should have been made). These two together reveal whether the agent is calling the right tools, calling unnecessary tools, or missing required ones. Measuring tool selection accuracy matters especially in systems where agents have access to large tool catalogs.

Trajectory efficiency. Steps per successful task and tokens per successful task. An agent that solves a task in 4 steps where 12 are typical is not just cheaper; it is also less likely to accumulate errors across a long chain. AgencyBench documents that realistic long-horizon scenarios can average 90 tool calls and approach 1M tokens, which makes efficiency tracking a cost-control requirement, not just a quality signal.

Per-dimension rubric scores. Each rubric dimension (coherence, factual grounding, instruction following, safety) produces a score on a defined scale (e.g., 1–5). Aggregate these as weighted means across the test set. Tracking per-dimension trends over time reveals which capability is degrading even when TSR stays flat.

Safety flag counts. The number of transcripts that triggered a safety evaluator, broken down by flag type (harmful content, privilege escalation, data exfiltration attempt). This is a count metric, not a rate, because even a single safety failure in a production system warrants investigation.

Pro Tip: When comparing two runs, do not rely on point estimates alone. Compute a confidence interval around the TSR difference and check whether it crosses zero before concluding one configuration is better. Small test sets (under 50 items) routinely produce misleading comparisons because the variance is too high to distinguish signal from noise.

Capability evals can start at lower pass rates and improve over iterations. Keep these two suites separate so a capability experiment failure does not trigger a regression alert.

---

How do you build a trustworthy evaluation pipeline from scratch?

The roadmap below is ordered by dependency: each step produces an artifact the next step consumes. Skipping steps produces evals that look like they work but cannot be trusted.

1. Define objectives and acceptance criteria. Write down what "success" means for each task type before touching code. Acceptance criteria are the contract between the eval and the engineering team.
2. Collect and author test items. Start with real production traces where available. Supplement with hand-authored edge cases and adversarial inputs. Aim for at least 30–50 items per task type to get meaningful aggregate metrics.
3. Design rubrics and graders. For each objective, decide which grader type applies (code, LLM judge, human). Write rubric dimensions with explicit descriptions and scoring scales. A minimal rubric row: {dimension: "instruction_following", description: "Agent completed all stated sub-tasks", scale: "1-5", weight: 0.4}.
4. Implement the harness and instrumentation. Wire up the runner to load test items, initialize the agent in a controlled environment, capture the full transcript, and route to graders. Below is a minimal pseudo-code harness:

for item in test_dataset:
    env = setup_environment(item.env_config, seed=item.seed)
    agent = Agent(scaffold_version=SCAFFOLD_VERSION)
    transcript = agent.run(item.input, env=env)

    results = []
    for grader in graders:
        results.append(grader.score(transcript, item.ground_truth))

    store.write(RunRecord(
        run_id=RUN_ID,
        item_id=item.id,
        transcript=transcript,
        grader_results=results,
        metadata={"scaffold": SCAFFOLD_VERSION, "seed": item.seed}
    ))

5. Run initial campaigns and calibrate graders. Execute the first full sweep, then manually review a sample of LLM-judge outputs against human labels. Adjust rubric wording and judge prompts until agreement is acceptable.
6. Integrate regression suite and CI gates. Add the regression suite as a required CI check on every pull request. A failing gate blocks the merge. AWS recommends pairing CI gates with continuous production monitoring so regressions are caught at two checkpoints.
7. Monitor production and iterate. Route a sample of live production traces through the same graders. Track metric trends over time. When a dimension score drops, trigger a targeted investigation before it becomes a user-visible failure.

Recommended default rubric weights for general-purpose task agents:

• Instruction following: 0.40
• Factual grounding / accuracy: 0.30
• Coherence and format: 0.15
• Safety compliance: 0.15 (treated as a hard gate, not just a weighted score)

---

How do you get reproducible results when agents are non-deterministic?

Non-determinism is the central engineering problem in agent evaluation. It comes from four sources: model sampling temperature, scaffold differences (prompt templates, parser versions), live environment volatility (external APIs, web content), and asynchronous tool behavior. Each source requires a different control.

Fixed seeds. Set temperature=0 and a fixed random seed wherever the model API supports it. This eliminates sampling variance across runs of the same item. Note that some providers do not guarantee deterministic outputs even at temperature zero, so treat seeds as a variance reducer, not a guarantee.

Offline snapshots. For any tool that reads from an external source (web search, database, third-party API), record the response at dataset-authoring time and replay it during eval runs. A unified sandboxed evaluation framework demonstrates that standardizing the instruction-tool-environment triplet and using offline snapshots disentangles scaffold effects from intrinsic model capability. Live external sources produce intermittent failures that mask real regressions.

API stubs and deterministic tool simulators. Replace live tool endpoints with stub implementations that return recorded or synthetic responses. This also makes evals runnable without network access, which matters for CI environments.

Scaffold metadata capture. Every run record must include the exact scaffold version: prompt template hash, parser version, tool schema version, and any middleware configuration. Without this, a score change between two runs could be caused by a prompt template edit rather than a model change.

Decision guide for offline vs. live evaluation:

• Prefer offline snapshots when: you need reproducibility for regression testing, the live environment is volatile or rate-limited, or you are comparing two model versions.
• Prefer live runs when: you need to measure real-world latency and availability, you are testing tool-use correctness against a live API contract, or you are running production monitoring (not regression testing).

Pro Tip: Always version and record scaffold metadata with each run record. When a metric shifts unexpectedly, the first diagnostic question is always "did the scaffold change?" If the metadata is missing, that question takes hours to answer instead of seconds.

---

How should you tailor evals for different agent types?

The general framework applies to all agents, but each agent class has distinct failure modes, instrumentation needs, and grader mixes. Applying a one-size-fits-all rubric to a coding agent and a conversational agent produces misleading results for both.

Coding agents

• Grader mix: Unit tests and integration test harnesses as primary graders (deterministic), LLM judge for code quality and style as secondary.
• Success criteria: All specified tests pass; no unintended side effects (file deletions, permission changes); code executes without runtime errors.
• Typical pitfalls: Agents that pass unit tests but introduce security vulnerabilities; agents that write correct code but modify files outside the specified scope.
• Example test case: Input: "Add a calculate_discount function to pricing.py that applies a 10% reduction to any price above $100." Expected tool calls: read_file("pricing.py"), write_file("pricing.py", ...). Pass criteria: the written file contains a function named calculate_discount, the function returns price * 0.9 for inputs above 100, and existing functions in the file are unchanged.

Conversational agents

• Grader mix: LLM judge for dialog quality (coherence, relevance, tone), code-based grader for outcome state (was the booking made? was the ticket created?).
• Success criteria: Intent resolved within a defined turn limit; no hallucinated facts; user-stated constraints honored.
• Typical pitfalls: Agents that produce fluent, coherent responses but fail to actually complete the task; agents that resolve intent but violate a stated constraint (e.g., booking a non-refundable ticket when the user asked for flexibility).

Web and tool-using agents

• Grader mix: Environment-state checks (did the correct form get submitted?), schema validation on tool call arguments, LLM judge for decision quality.
• Success criteria: Target environment state reached; no unintended side effects on adjacent state; tool calls conform to schema.
• Typical pitfalls: Agents that navigate correctly but click the wrong button on the final step; agents that call tools with malformed arguments that happen to succeed due to lenient API validation.

Long-horizon research agents

• Grader mix: Trajectory-level rubrics (planning quality, source diversity, claim grounding), simulated user feedback for multi-turn interactions.
• Success criteria: Research output covers all specified sub-questions; claims are grounded in cited sources; trajectory does not loop or stall.
• Typical pitfalls: AgencyBench's benchmark scenarios show that long-horizon tasks can require extensive tool calls and large token budgets, making cost tracking as important as quality tracking. Agents that produce high-quality outputs but consume 10x the expected token budget are not production-ready.

---

How do you scale evals without scaling your human review budget?

Scaling evaluation is fundamentally a cost-allocation problem. Human review is accurate but expensive; LLM judges are cheap but require calibration; deterministic checks are free but narrow in coverage. The solution is layered grading.

Continuous evaluation workflow:

• PR smoke tests: A fast subset of the regression suite (10–20 items, deterministic graders only) runs on every pull request in under two minutes. This catches obvious regressions without blocking CI.
• Nightly capability sweeps: The full capability suite runs overnight, including LLM judges. Results are posted to a dashboard and reviewed each morning.
• Release candidate suites: Before any major release, run the full suite plus an expanded adversarial set. Human reviewers audit a stratified sample of LLM-judge outputs.
• Production continuous monitoring: A sample of live production traces (typically 1–5%) routes through the grader pipeline in near-real time. AWS's production guidance emphasizes that continuous monitoring is necessary to detect agent decay and that HITL audits are required to maintain golden datasets for judge calibration.

Synthetic data and user simulators. When real production traces are scarce (a new feature, a low-traffic task type), synthetic task generation and simulated users expand coverage. The key constraint is representativeness: synthetic items should match the distribution of real tasks in difficulty, ambiguity, and tool-use patterns. Periodic comparison of synthetic-set metrics against real-trace metrics catches distribution drift.

LLM-judge calibration at scale. Run a calibration batch of 100–200 items through both the LLM judge and human reviewers every time you update the judge's model or system prompt. Track agreement rate and flag dimensions where disagreement is systematic.

Pro Tip: Use layered grading to control cost: run cheap deterministic checks first and only route items that pass to the LLM judge. Items that fail the deterministic check are already flagged; running an LLM judge on them adds cost without adding information. Reserve human review for items where the LLM judge score falls in an uncertain range or where the safety grader fires.

---

How do you turn failing evals into engineering tasks?

A failing eval is only useful if you can trace it to a specific, fixable cause. The diagnostic workflow below moves from raw failure to prioritized fix in a repeatable way.

Diagnostic steps:

• Reproduce the trace. Re-run the failing item with the same seed, scaffold version, and environment snapshot. If it does not reproduce, you have a flakiness problem, not a capability problem. Fix the environment controls first.
• Classify as decision-level vs. execution-level failure. A decision-level failure is a wrong plan or wrong tool selection. An execution-level failure is a correct plan that fails during tool execution (API error, schema mismatch, timeout). The fix is different for each: decision failures point to prompting or planning logic; execution failures point to tool contracts or retry logic.
• Extract per-dimension scores. Look at which rubric dimensions are failing. A drop in "instruction following" with stable "coherence" scores points to a specific capability gap, not a general quality regression.
• Cluster similar failures. Group failing items by failure type, tool call pattern, or rubric dimension. A cluster of 15 items all failing on the same tool call with the same argument error is a single bug, not 15 separate problems.
• Propose targeted fixes. Match fix type to failure class: prompt revision for decision failures, tool contract update for schema mismatches, retry logic for transient execution failures, memory update for context gaps. Tracking agent memory design is especially relevant here, since memory gaps often surface as repeated decision failures on tasks the agent has "seen" before.
• Validate via targeted regression tests. Write a new test item that specifically covers the fixed failure mode. Add it to the regression suite. The fix is validated when the new item passes and no previously passing items regress.

Example failure cluster: Five items all fail because the agent calls create_ticket before calling check_duplicate, creating duplicate records. Classification: decision-level (wrong tool order). Fix: update the planning prompt to include an explicit pre-condition check for check_duplicate. Validation: add five regression items covering the duplicate-check scenario; confirm all pass after the prompt update.

Pro Tip: Keep regression suites strictly separate from capability experiments. A capability experiment is allowed to fail; that is how you learn where the agent's limits are. A regression suite failure means something that worked is now broken. Mixing the two produces alert fatigue and causes real regressions to get lost in exploratory noise.

---

What tools map to which eval components?

No single tool covers the full evaluation pipeline. The practical approach is to pick one tool per component and wire them together through a shared result store or CI integration.

Harness runners and orchestration:

• Inspect AI (open-source): a Python-based evaluation framework with built-in task runners, dataset loaders, and solver abstractions. Integrates with OpenAI, Anthropic, and local models. Well-suited for teams that want a code-first harness with full control over the execution loop.
• LangSmith (LangChain): provides tracing, dataset management, and evaluation runs with a UI for browsing transcripts. Strong CI integration via its SDK. Best for teams already using LangChain-based agents.
• Braintrust: a hosted eval platform with dataset versioning, LLM-judge configuration, and a scoring dashboard. Useful when you want a managed result store without building your own.

Instrumentation libraries:

• OpenTelemetry with an LLM-specific semantic convention layer (e.g., the OpenLLMetry instrumentation package) captures token counts, latency, and tool call spans as structured traces. These traces feed directly into the harness's transcript capture.

LLM-judge SDKs:

• OpenAI Evals and Anthropic's eval tooling both provide rubric-based judge configurations. Azure Foundry's built-in evaluators cover safety, coherence, and groundedness out of the box and integrate with Azure AI Studio's CI/CD hooks.

CI integrations:

• Most harness runners expose a CLI that returns a non-zero exit code on gate failure, making them compatible with GitHub Actions, GitLab CI, and Jenkins without custom plugins.

Dashboards and alerting:

• Grafana with a time-series backend (Prometheus or InfluxDB) works well for metric trend visualization. For teams that want eval-specific dashboards, Braintrust and LangSmith both provide built-in views.

Example SDK call to push an eval run and retrieve results (pseudo-code):

# Push results to a managed eval store
client = EvalClient(api_key=API_KEY, project="my-agent")
run = client.create_run(
    dataset="regression-v4",
    metadata={"scaffold": "v1.4.2", "model": "claude-opus-4"}
)
for item_result in run_results:
    run.log(item_result)

summary = run.finalize()

When evaluating citation quality in agent outputs, tools like the AI Citation Audit from BabyLoveGrowth can help verify that agent-generated research outputs are grounding claims in real, checkable sources, which is a useful complement to rubric-based grounding evaluators.

---

Running agent-targeted evaluations on a multi-agent deployment

The following walkthrough applies the framework above to a multi-agent system running on agent-swarm, where a lead agent breaks down tasks and delegates to specialized worker agents running in isolated Docker containers.

Step 1: Collect representative tasks. Pull 60 recent session traces from agent-swarm's session store, covering three task types: code generation, research summarization, and workflow orchestration. Stratify by outcome (30 successful, 20 partial, 10 failed) to ensure the eval set covers the full difficulty range. The agent-swarm examples library provides a starting point for representative session structures.

Step 2: Re-run stored traces through an offline harness. Configure the harness to replay each session trace against a snapshot of the tool environment (stubbed GitHub API, stubbed Linear API) rather than live endpoints. This eliminates environment volatility as a confounder.

Step 3: Generate rubric evaluators. Define four rubric dimensions for this deployment:

• task_decomposition_quality (1–5): Did the lead agent break the task into appropriate sub-tasks?
• tool_selection_accuracy (1–5): Did worker agents call the correct tools with valid arguments?
• output_completeness (1–5): Did the final output address all stated requirements?
• safety_compliance (pass/fail): No privilege escalation, no unintended data writes.

Step 4: Run LLM judge plus deterministic checks. Deterministic checks validate tool call schemas and output file structure. The LLM judge scores the three rubric dimensions. Safety compliance runs as a separate hard-gate grader.

Step 5: Aggregate per-dimension scores and cluster failures.

Pseudo-configuration for pointing the evaluator at stored traces:

eval_config:
  dataset: "session-traces-2026-06"
  harness: "offline-replay"
  env_snapshot: "snapshots/2026-06-01"
  graders:
    - type: code
      checks: ["tool_schema_validation", "output_file_structure"]
    - type: llm_judge
      model: "claude-opus-4"
      rubric: "rubrics/multi-agent-v2.yaml"
    - type: code
      checks: ["safety_compliance"]
  output_store: "results/eval-2026-06"

What the eval revealed:

• A cluster of 11 failures where worker agents called write_to_linear before fetch_linear_context, creating tickets with missing parent references. Classification: decision-level failure in the worker agent's planning prompt.
• Three sessions where the lead agent's task decomposition produced overlapping sub-tasks, causing two workers to write conflicting outputs to the same file. Classification: coordination failure, traced to agent density and composition patterns.
• Memory gaps in 7 sessions: the worker agent repeated a tool call it had already completed in the same session, indicating the session context was not being read correctly.

Changes made:

• Updated the worker agent's planning prompt to enforce a fetch_context pre-condition before any write operation.
• Added an overlap-detection check to the lead agent's decomposition logic.
• Fixed the session context reader to correctly surface completed tool calls.
• Created a targeted regression suite of 18 items covering all three failure patterns. All 18 pass after the fixes.

---

Production lessons and what actually matters in practice

Running evals in production teaches you things that no benchmark paper covers. Here are the operational lessons we have accumulated.

Invest in telemetry before you invest in graders. You cannot evaluate what you cannot observe. If your agent runtime does not emit structured traces with tool call arguments, return values, and token counts at every step, your graders are scoring summaries, not behavior. Instrument first.

Version your scaffolds like you version your code. A prompt template change that seems minor can shift TSR by several percentage points. If you do not record the exact scaffold version with every run, you will spend hours debugging a "model regression" that is actually a prompt edit.

Separate regression suites from capability experiments, and enforce that separation in CI. Regression suites protect the baseline; capability suites explore new territory. Mixing them means exploratory failures trigger production alerts, and real regressions get buried in noise.

Calibrate your LLM judges more often than you think you need to. Judge drift is slow and invisible. A judge that was well-calibrated three months ago may have drifted if the underlying model was updated. Schedule calibration runs quarterly at minimum, and always re-calibrate after a model version change.

Present eval results to product and SRE teams as trends, not snapshots. A single TSR number means nothing without context. A chart showing TSR over 90 days, annotated with deployment events, is a conversation starter. SRE teams respond to alert thresholds; product teams respond to trend lines. Give each team the view they can act on.

On ethical evaluation boundaries: any eval that involves safety-critical decisions (medical, legal, financial, or security-adjacent tasks) requires human review as a mandatory gate, not an optional audit. Automated judges are not reliable enough for decisions where a false negative has real-world consequences. Build the human review step into the pipeline architecture, not as an afterthought. Security-adjacent failure modes, including privilege escalation patterns, deserve dedicated threat modeling before you design the safety grader.

---

Sources

• Demystifying evals for AI agents - Anthropic
• Mastering Agentic Techniques: AI Agent Evaluation | NVIDIA Technical Blog
• A Unified Framework for the Evaluation of LLM Agentic Capabilities
• Run an agent-targeted evaluation (Azure Foundry docs)
• AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts - ACL Anthology
• Evaluating AI agents: Real-world lessons from building agentic systems at Amazon | Artificial Intelligence

---

FAQ

What is the difference between model benchmarks and agent evaluations?

Model benchmarks like MMLU test static knowledge and reasoning in isolation. Agent evaluations measure system behavior across a full workflow, including planning, tool use, and multi-step trajectory execution, which is what actually determines production reliability.

How many test items do you need for a reliable agent evaluation?

At minimum, 30–50 items per task type to produce meaningful aggregate metrics. Fewer than that and the variance in pass rates is too high to distinguish a real capability change from statistical noise.

When should you use a human grader instead of an LLM judge?

Use human graders for safety-critical labels, ambiguous cases where the rubric does not clearly resolve the score, and for calibrating LLM judges. Calibration batches of 100–200 human-labeled items are the standard practice for validating judge accuracy.

How do you prevent scaffold changes from corrupting eval results?

Record the exact scaffold version (prompt template hash, parser version, tool schema version) in every run record's metadata. When a metric shifts, compare scaffold metadata across runs before attributing the change to a model or data difference.

What is the right cadence for running evals in CI?

Run a fast deterministic smoke test on every pull request, a full capability sweep nightly, and a comprehensive suite before each release.

Recommended

• Your AI Workflow Has Too Many Agents | agent-swarm.dev
• We Hid 75 of Our Agent's 90 MCP Tools — And It Got Smarter | agent-swarm.dev
• Nobody Prompt-Injected Our Agents — They Escalated Their Own Privileges | agent-swarm.dev
• Why Your AI Agent Needs a Job Description: SOUL.md & Identity Architecture | agent-swarm.dev