Same Request, Three Outcomes: Why Customer-Facing Agents Need pass^k Acceptance, Not Average Success

One Customer, Different Outcomes on a Retry

Consider a routine support scenario: a customer contacts an automated agent to modify an order or verify a fee adjustment. On the first try, the agent asks repetitive questions until the customer abandons the session. On a retry with the exact same request, it modifies the main item but misses a linked add-on. On another retry, it issues a return immediately—without the explicit user confirmation required by policy. Before launch, the team approved the system because its average single-run success rate looked acceptable. In production, however, as agents move into customer-facing workflows, the human oversight needed to clean up inconsistent runs can eat into the gains automation was expected to deliver.

The root cause is accepting customer-facing agents on average task success. An average smooths over run-to-run variance, whereas each customer experiences one run. When the same request must succeed on every repeat (pass^k), success falls sharply, and it falls further when user behavior shifts toward impatience, skepticism, or heightened emotion. Recent evidence shows that most failures are execution errors—misread intent, wrong action order, wrong tool arguments, and missed actions—and policy breaches, not missing empathy. In our view, acceptance must be repeated-trial and policy-checked, with consequential writes such as refunds and charges rehearsed in a sandbox and handed to human support when disputed.

Why Average Success Hides Fragility: Evidence from τ-bench, τ-Trait, and τ-Emotion-Bench

Three benchmarks on tool-agent-user interaction demonstrate how single-run averages overstate production readiness across repeatability, behavioral traits, and user emotion.

Yao, Shinn, Razavi, and Narasimhan (2024) introduced τ-bench, where a language agent with domain API tools and a policy document converses with a simulated user in two customer-service domains: retail and airline^[1]. Evaluation compares the final database state with an annotated goal state and defines pass^k—the chance that all k i.i.d. trials of the same task succeed—contrasting it with pass@k, where at least one succeeds^[1]. State-of-the-art function-calling agents succeeded on fewer than 50% of tasks overall; the best function-calling agent tested in 2024 reached about 61% pass^1 on retail and about 35% on airline, while pass^8 fell below 25% in retail^[1]. In retail failure analysis, wrong decision-making (not following domain rules) accounted for 25% of failures, and partial resolution of compound requests accounted for 19%^[1]. The authors also noted that a rule-based reward is necessary but not sufficient: an agent might issue a return without explicit user confirmation, violating policy^[1].

Standard evaluations also assume patient users. He et al. (2025) extended τ-bench to τ-Trait across four domains—airline, retail, telecom, and telehealth—using steerable user-trait directions^[2]. Small shifts toward more impatient, incoherent, or skeptical user behavior caused an average 2% to 30% performance degradation across frontier models^[2].

Singh, Saeidi, Hegde, Kumbhar, and Baral (2026) introduced τ-Emotion-Bench across 580 executable tasks in ten domains, including airline, retail, telecom, telehealth, bank, and travel^[3]. Holding the underlying task and success conditions fixed, it varies user emotion via five emotion families and 22 fine-grained emotions combined with politeness, urgency, and trust across 594 conditions^[3]. The simulated user may end the conversation as unsatisfied, which counts as failure before completion; each task runs for 3 trials to compare average success (Avg@3 / Pass^1) against three-trial consistency (Pass^3), with confirmed user-simulator errors in only 1.1% of failed trials^[3].

Consistency drops sharply across repeats. Under one user simulator, the best evaluated model reached 55.9% average Pass^1 (Avg@3) but only 32.2% average Pass^3; under a second simulator, the best model reached 78.6% Avg@3 but 66.5% average Pass^3^[3]. Most models lost 14 to 24 points between Pass^1 and Pass^3, and under the first simulator even the strongest agents solved only about a third of tasks in all three trials^[3]. Domain spread was also wide: one strong model reached 62.0% Pass^3 in one domain but 2.9% in telecom^[3].

Across 6,961 failed trajectories with at least one identified fault, failures concentrated in task execution rather than affective interaction: misread intent accounted for 15.9%, wrong action order 13.7%, wrong tool arguments 12.9%, and missed actions 12.2%, whereas policy and interaction faults were 4% to 10% per domain and emotional neglect was below 1.7%^[3]. Still, policy violations persisted in some conditions at roughly 67% in telehealth and 91% in telecom even among stronger models^[3]. Moreover, empathetic responses were consistently accompanied by a higher fraction of follow-up questions; the authors noted that empathy should acknowledge the user's state without adding unnecessary interaction, balancing tool execution with only necessary follow-up^[3].

Engineering for Repeatability: Sandbox Rehearsal, Policy Checks, and Human Gates

Because execution faults and policy breaches dominate agent failures, prompt-level empathy tuning alone is unlikely to make a workflow safe for production. In our view, engineering acceptance should run each scripted customer scenario several times with varied simulated user behavior, report pass^k next to average success, enforce a policy check before every database write, keep refunds in a sandbox until human approval, and route disputes to tickets with a named human owner.

Within the Commerce and Agent OS modules on rlab.works, three proposed skill packs and three atomic components illustrate this architecture:

• Benefit verification and fee adjustments in consumer support: In Commerce, Check benefits and support is transferable to consumer applications; adaptation requires replacing hotel pricing models and linking wallet or account identity, while ensuring sign-in never implies paid entitlement. It runs through four steps: check identity and benefits, calculate adjusted charges, pause at a human gate where support accepts and decides, and notify while keeping a record. At that gate, Support tickets and notification inbox (working maturity) manages tickets, replies and internal notes, assignment, priority, history, and notification states so disputed adjustments have a named human owner.
• Administrative intake and access control in healthcare: In Agent OS, Intake and handoff is transferable to healthcare strictly for administrative service—appointments, records access, and handoff to staff—with no symptom-based clinical triage or diagnosis. Adaptation requires connecting hospital identity and ticket systems across four steps: confirm account and role, create a service record, pause at a human gate to approve data access, and hand off and record. To curb policy breaches, Operations identity, permissions and audit trail (working maturity) enforces operator accounts and sessions, two-factor verification, role permissions, and auditable operations.
• Payment reconciliation and sandboxed refunds: In Commerce, Reconcile payments and refunds is transferable to consumer workflows under a strict boundary: use either a sandbox or a real adapter per run, never mix environments, and require valid authorization and payment integration for real refunds. Before the human gate that approves a refund or correction, Sandbox payment and refund state orchestration rehearses payment states and validates amount, currency, and idempotency.

Capability status must be stated honestly: all three skill packs are proposed compositions demonstrated on synthetic data, not live customer deployments. While the support ticket inbox and operations identity components are at working maturity, sandbox payment and refund state orchestration is a prototype with a sandbox provider only and no live payment adapter yet.

Pre-Release Acceptance Checklist for Customer-Facing Agents

Before deploying a customer-facing agent, engineering and operations leaders should enforce three checks:

1. Accept on pass^k across behavior-shifted simulations, not average success alone: Run each scripted customer scenario across multiple trials with impatient, skeptical, and emotional user simulations, and report pass^k (such as Pass^3) alongside average success. Block deployment in domains where multi-trial consistency collapses.
2. Enforce pre-write policy checks and cut unnecessary follow-up questions: Audit logs for misread intent, wrong action order, wrong tool arguments, and missed actions, and verify policy rules and explicit user confirmation before every database write. Keep empathetic acknowledgment brief so extra follow-up questions do not trigger unsatisfied user abandonment.
3. Rehearse consequential writes in a sandbox and route disputes to named human owners: Validate amount, currency, idempotency, and permissions in a sandbox first, and pause at a human gate backed by an auditable ticket queue whenever a refund, charge adjustment, or disputed request requires final sign-off.