Uncertainty Finds Failing Trajectories — Not the Step Worth Steering

Why Uncertainty Thresholds Misfire as Mid-Run Intervention Triggers

When engineering teams deploy multi-step agents, a common instinct is to wire runtime interventions directly to model uncertainty. If token entropy, sequence log-probability, or verbalized confidence crosses a threshold, the orchestrator interrupts the run and triggers a corrective mechanism—prompting reflection, invoking critic feedback, calling a stronger model, or injecting memory.

In multi-step workflows, however, treating an uncertainty threshold as an automatic steering switch often degrades results. Mid-trajectory interruptions add latency and cost while derailing runs that would have succeeded on their own; conversely, at the pivotal step where guidance would rescue a failing trajectory, raw uncertainty is often unremarkable. This paradox stems from conflating two distinct questions: uncertainty can flag which trajectories are failing, but no single signal reliably locates the step where intervention improves the outcome. Making runtime steering net-positive requires separating whether to steer, where to steer, and which mechanism to invoke; training monitors on measured counterfactual outcome deltas rather than failure labels alone; and budgeting how often successful runs may be disturbed.

From Uncertainty to Action: Stepwise Counterfactual Outcomes and the Value of Steering

Prior work on calibration and self-evaluation explains why correctness estimation and intervention localization diverge. Guo et al. (2017) showed that confidence in modern neural networks is often miscalibrated relative to accuracy; more fundamentally, even calibrated confidence is not automatically a locator of intervention value inside a sequential decision process^[3]. Similarly, Kadavath et al. (2022) studied P(True) self-evaluation of answer correctness^[2]. When such uncertainty and self-evaluation signals enter multi-step agent loops, can they tell an orchestrator when to step in?

Li et al. (2026) test this directly by building a Stepwise Outcome Table (SOT) that measures counterfactual steering at every non-terminal step^[1]. Across three benchmarks (app APIs, household text, and a shopping simulation), two open-weight agents, and four steering mechanisms—reflection, critic feedback, stronger-model assistance, and memory-based guidance—the authors take 1,864 base trajectories, apply steering at every non-terminal step, run each continuation to completion, and record the task score change (Δ) across roughly 82,000 counterfactual continuations^[1]. Decomposing runtime control into three decisions exposes why naive triggers fail:

• Whether to steer—and the harm of false triggers: At the trajectory level, uncertainty carries real failure-detection signal, reaching AUROC up to 0.73 and 0.71 on the two agents on AppWorld^[1]. Yet intervention can harm good runs: steering a successful trajectory at a random step lowers average task score by up to 0.14 and 0.32 for the two agents^[1]. Always-steering can actively hurt overall performance—for example, one agent loses 9.9 points on AppWorld when always steered compared with unmodified runs^[1].
• Where to steer—detection rankings oppose localization: Signals best at detecting failing trajectories are not the best step locators. Across estimators, failure detection rankings and step localization rankings are unrelated or opposed (Spearman rank correlation −0.12 and −0.41 on the two agents)^[1]. Within a trajectory, correlation between uncertainty and measured steering outcome is weak: the best median Spearman correlation is only about +0.15—achieved by P(True) self-evaluation, which is among the weaker failure detectors yet most consistently aligned with where steering helps^{[1, 2]}. Every estimator is negative on at least a quarter of trajectories^[1].
• Which mechanism—agreement on trajectories, disagreement on steps: The four mechanisms agree moderately on which trajectories are recoverable (best-outcome correlations 0.17–0.66), but barely on which step to steer (within-trajectory order correlations 0.02–0.19)^[1].

From the SOT, Li et al. (2026) train Value of Steering (VoS)—a value function learned directly from measured score deltas (Δ) across all signals, paired with a harm-budgeted trigger threshold limiting the false-trigger rate α on successful trajectories (α chosen on train in 5–30%)^[1]. Using memory-based guidance as its fixed mechanism (since routing across mechanisms did not reliably beat it), VoS improves over unmodified execution in all 12 settings (3 benchmarks × 2 agents × offline/online) by 7.8 points on average, and beats the strongest of five uncertainty-triggered methods from prior work in 11 of 12 settings by 2.9 points on average^[1]. Ablations show that training on measured outcomes beats failure labels, binary help labels, or process-reward-style targets; a tight harm budget beats steering almost every trajectory; and combining all signals is more robust than any single estimator family across the 12 settings^[1]. Scope limits remain clear: experiments require open-weight agents for token-level uncertainty, building the SOT via per-step re-runs is expensive, and the three testbeds are research benchmarks rather than production certifications^[1].

Engineering Mapping on rlab.works: Separating Decisions, Shadow Deltas, and Governed Gates

In our engineering view, these findings point to five production rules: write whether, where, and which decisions separately; collect counterfactual or shadow outcomes before trusting an uncertainty threshold in production; train monitors on Δ score rather than fail/success labels alone; set an explicit false-trigger budget on runs that would have succeeded; and recalibrate when the agent, tools, or score definition change.

Across Agent OS and AI Gateway on rlab.works, three proposed skill packs and three components illustrate how to structure these gates—and where current boundaries lie:

• Pre-action intervention gates: In Agent OS, Approve computer actions (S03) runs check readiness → approve the action list (human gate) → act and retain evidence → check the changes (human gate), analogous to deciding whether and where to intervene before computer actions execute. It draws on Auditable workflows, human confirmation and decision review (which pauses workflows for human decisions and records reviews) and Durable governed agent execution (which provides a governed execution loop). Transferable in legal workflows—restricting read/write folders and keeping external filing outside default permissions—it places explicit gates before side effects rather than relying on noisy mid-run thresholds.
• Model escalation and shadow policy comparison: Because stronger-model assistance is one steering mechanism and offline policy selection requires counterfactual comparison, Compare and switch models (S12) in AI Gateway pairs check candidate services → compare sample results → review switch proposals (human gate) → configure primary and fallback → recheck samples (human gate) with Model comparison and shadow evaluation for offline and shadow policy evaluation. Transferable in enterprise settings, it lets teams verify whether an escalation policy yields positive score deltas before affecting live runs.
• Staged review gates in multi-step plans: In Agent OS, Plan and deliver work (S01) structures multi-step execution into plan the steps → approve the scope (human gate) → run and collect → review and hand over (human gate). Transferable in manufacturing—decomposing maintenance into inspection, material preparation, and review tasks with handover evidence—it concentrates intervention at explicit plan and handover gates while keeping software records separate from physical repair work.

Adaptation boundaries must be stated plainly. All three skill packs are proposed, unintegrated compositions demonstrated on synthetic data and require domain contract wiring. Auditable workflows, human confirmation and decision review is a prototype, while Model comparison and shadow evaluation and Durable governed agent execution carry working ratings from static source inspection only. None of the site packs or components implement VoS or automatically eliminate bad interventions.

Three Pre-Deployment Checks for Runtime Agent Steering

Before enabling runtime steering triggers in multi-step agents, engineering leaders should apply three checks:

• Separate whether, where, and which in control logic: Decouple trajectory risk detection from step localization, and prefer a validated, fixed guidance mechanism (such as memory-based guidance or explicit workflow gates) over unverified dynamic mechanism routing.
• Train and select triggers on measured outcome deltas (Δ), not failure labels: Use offline replay or shadow evaluation to measure actual score changes from intervention, combining multiple signals—including self-evaluation—to locate steps with positive net value.
• Budget false triggers on successful runs and recalibrate on change: Cap the false-trigger rate on trajectories that would succeed unmodified (such as a 5–30% budget), and recalibrate whenever the agent, tools, or scoring metric changes.