Beyond the Score Table: Oracle Defects and Configuration-Dependent Measurement in Agent Eval Harnesses

When the Measuring Instrument Fails: The Hidden Risk Behind High Benchmark Scores

In enterprise software and manufacturing operations, engineering teams increasingly rely on automated agent benchmark tables to govern release readiness. If a candidate configuration achieves a high pass rate on multi-turn tool-use tasks, or if a prompt and schema revision produces a sharp single-run gain, teams treat the score table as objective proof to promote the route into production. Yet after deployment into live order modification, ticket dispatch, or inventory state machines, those workflows still trigger database mismatches and policy violations.

Often, this failure stems not from subjective judge bias, but from defects inside the evaluation harness itself. A benchmark success rate is an instrument reading produced by a harness (tasks, input generator, and grading oracle), not a direct measure of intrinsic model capability. Recent audits show that oracles can mutate graded database state, verifiers can score tool names instead of operations, and single-run comparisons cannot separate presentation changes from ordinary rerun noise^{[1, 2]}. Release gates need oracle-integrity tests, replayable tool-call logs, and repeated runs under a fixed claim about what the score means.

Inside Harness Failure: Oracle State Mutation, Configuration Sensitivity, and Verifier Brittleness

To evaluate agents across realistic multi-turn workflows, Yao, Shinn, Razavi, and Narasimhan (2024) introduced the τ-bench evaluation form, in which an agent converses with a simulated user, follows domain policy documents, invokes domain APIs, and is graded against the terminal database state^[3]. While this structure mirrors enterprise transaction systems, coupling an external runner with a state-based reward function without isolation allows the grading step itself to corrupt the measurement.

Studying the evaluation harness—especially the reward oracle—rather than intrinsic model capability, Chen (2026) audited a widely used τ-bench-form agent–user–database benchmark across retail and airline domains^[1]. The audit uncovered an oracle defect framed as an integration failure (unsafe upstream side effect plus a second state reading): the reward function resets the environment database to build a ground-truth hash and never restores it^[1]. Although the returned value can be correct for the benchmark’s own runner, a harness that scores the episode after the session ends re-reads the reference solution’s state^[1]. Replaying recorded tool calls from 4,320 archived episodes against a clean database reduced database passes from 4,194 in the archived labels to 2,883, lowering measured success from 94.3% to 65.5%^[1].

To detect such defects, Chen (2026) built a verification suite of five state-integrity invariants and three known-answer controls tested against seven injected oracle defect classes^[1]. Every injected class was detected on the studied implementations, and the repaired oracle shipped with the package yielded no false positives^[1]. The author bounds these claims to the studied implementations rather than claiming a general-purpose framework: a second independently implemented benchmark could not reconstruct the reference terminal state, producing a negative portability result^[1]. A secondary configuration ablation across 3,240 retail episodes localized environment effects across the three documents the agent reads (policy documents, tool descriptions, and error strings)^[1]. All three carried a protective effect on trajectory collapse, with tool descriptions largest (−9.2 percentage points collapse) and error strings smallest (−5.3 percentage points) in an exploratory ordering^[1]. Because the sign of the user-language effect reversed in the airline domain, the finding is strictly bound to the measured configuration and does not imply that a given language inherently makes agents better or worse^[1].

Whereas state mutation corrupts the graded environment, Gaonkar (2026) introduced TRACE to diagnose verifier brittleness via paired mutations, score comparisons, trajectory inspection, and counterfactual rescoring of unchanged trajectories^[2]. Across 25 synthetic research-operations tasks with scripted policies, renaming tools lowered a careful scripted agent's score by 0.250 even though it performed the same operations; restoring original names at scoring time closed the entire gap (0.250 to 0.000)^[2]. The same mutation exposed a genuine behavioral failure in a second scripted agent where the gap remained after rescoring^[2].

In a public τ²-bench follow-up, Gaonkar (2026) showed that identical reruns flip 15–36% of task outcomes (14.7–16.3% for three agents; 35.6% for one), so single-run comparisons cannot separate presentation effects from run-to-run variation^[2]. With repeats controlled, meaning-preserving tool renames and observation reformatting left reward unchanged within ±0.10 for seven of eight agent–change pairs, while deliberately misleading tool names lowered every agent’s reward by 0.20–0.44^[2]. Moreover, two frontier language-model judges disagreed on 57% of the same fixed records (one grading procedure rather than outcome), and presentation changes did not move verdicts beyond call-to-call variation once repeats were controlled^[2].

Engineering the Harness as Production Software: Replay Logs, State Assertions, and Human Gates

These empirical findings point to a clear architectural rule (the following mapping and verification rules represent our team’s engineering reasoning): teams must treat evaluation harness continuous integration like any other production software. Across the AI Gateway and Agent OS modules on rlab.works, this discipline maps onto three workflow patterns and atomic components:

First, persist tool-call events to enable clean-database replay. Using Recoverable agent runtime and context management (a working component), the runtime persists structured events, checkpoints, and replay logs. In harness continuous integration, teams should repeat-score the same finished episode, assert terminal state stability after session end, and keep tool-call logs replayable against a clean database so archived pass rates can be verified independently.

Second, freeze the measurement claim and quantify rerun noise before switching routes. In the AI Gateway module, the skill packs Compare and switch models (S12) and Connect and check services (S18) are transferable patterns for enterprise engineering. Respecting their adaptation boundaries is critical: Compare and switch models requires real samples to verify compatibility, relies on existing records for recommendations, makes no promise of an automatic optimal model, and includes human review of the switch proposal plus a post-switch sample check. Connect and check services checks node heartbeat, application response, and deployment separately without assuming streaming or auto-retry, ending in a mandatory human acceptance or rejection gate after registration, client preparation, and test requests. Supported by Gateway usage, financial, reliability and fleet analytics (a working component), teams should freeze what claim a score supports (system configuration, model comparison, or absolute performance), require multiple repeats before comparing configurations, and report the instrument failure rate beside the score.

Third, enforce a human gate before promoting routes on evaluation evidence. In the Agent OS module, the skill pack Plan and deliver work (S01) is a transferable pattern in manufacturing and a general governance workflow elsewhere. It places human gates at scope approval and delivery review via Auditable workflows, human confirmation and decision review (a prototype component that pauses workflows for human confirmation and logs review decisions, while tool steps currently act as placeholders and records). Reviewers inspect oracle integrity checks and repeat-run evidence here to accept or reject a release before promotion advances.

All referenced skill packs are proposed compositions demonstrated with synthetic data and not yet integrated with live services; deploying clean-database replay requires domain-specific database snapshot wiring.

Three Calibration Checks Before Trusting an Agent Eval Score

Before promoting a model or workflow based on benchmark scores, engineering leaders should apply three checks (team engineering reasoning):

1. Test oracle state invariance and clean replay: Repeat-score the same finished episode to assert terminal state stability after session end, run known-answer controls, and verify that recorded tool-call logs reproduce the score when replayed against a clean database.
2. Use counterfactual rescoring to isolate verifier brittleness: When tool renames or format edits shift scores, rescore unchanged trajectories under the original names to verify whether the scoring rule—rather than agent behavior—caused the drop.
3. Freeze the measurement claim, run repeats, and require human sign-off: Reject single-run comparisons, require repeated runs under a fixed claim with instrument failure rates reported beside the score, and require human gate approval before switching production traffic.