Route or Defer? Cost-Aware Abstention as a Readiness Test for Automated Ticket Routing

Why Overall Accuracy Cannot Tell You Whether a Ticket Is Safe to Auto-Route

When enterprises pilot automated routing for internal IT desks or support queues, launch reviews tend to focus on overall test accuracy. Once a prompted language model returns plausible labels with high confidence scores, it is tempting to assume a confidence cutoff will auto-route clear cases and defer the rest to people.

Operational routing, however, is not a symmetric guessing game. Sending a ticket to the wrong resolver group triggers cross-team bounce-backs and queue delays—a penalty that can exceed the cost of human triage at intake^[1]. When a wrong route costs more than sending the item to a human, accuracy alone cannot say when an automated router should be trusted. Abstention exposes a weak confidence signal rather than repairing it. Grounding the “route or defer” decision in explicit costs and an offline, frozen policy evaluation provides a concrete readiness test before go-live.

From the Error–Reject Trade-Off to Cost-Aware Abstention: Mechanisms in the Literature

Deferring uncertain inputs dates back to Chow (1970), who formalized the optimum error–reject trade-off: rejecting (deferring) uncertain inputs trades coverage for fewer errors, with the optimal threshold governed by the relative cost of an error versus a rejection^[3]. Guo et al. (2017) showed that modern neural networks are poorly calibrated—confidence does not match correctness likelihood—and post-processing calibration can fix the probability level^[4]. Whether a better-calibrated score also ranks errors below correct answers is a separate question, and it is the one the next study tests.

Focusing on internal IT tickets, Gupta (2026) treats a Chow-style error–reject trade-off as an offline, cost-conditional readiness assessment: choose a route-or-defer policy on calibration data under explicit costs, freeze it, and evaluate once on held-out data^[1]. The study uses a public IT Service Ticket Classification Dataset of 47,837 internal IT tickets across eight classes, split 70/10/20 into train, calibration, and test sets (33,486 / 4,784 / 9,567)^[1]. Four trained non-LLM classifiers are compared against four large language models under zero- and few-shot prompting (eight prompted conditions, evaluated on a stratified 1,000-ticket test subset)^[1].

The paper defines illustrative, unmeasured per-ticket costs of 0 for a correct auto-route, 1 for human triage, and 5 for a wrong route—stressing that every conclusion is conditional on them and that an organization must estimate its own costs^[1]. Under these illustrative 0/1/5 costs, the break-even correctness probability is 0.80 (route only if the probability of being correct is at least 0.80)^[1]. Held to this protocol, the two families diverge:

• Substantial modeled cost cuts from a simple trained classifier: A simple trained text classifier using term-frequency (TF-IDF) features and logistic regression, with a 0.60 threshold selected on calibration data and frozen, lowers expected per-ticket cost on held-out test data from 0.763 to 0.472—a 38.1% relative reduction (95% CI: 35.7%–40.4%)^[1]. Wrong routes fall from 15.3% to 3.5%, while auto-routing 70.5% of tickets at 95.0% accuracy on the routed ones^[1]. Across all four trained classifiers, relative cost reduction clusters at 35.5%–38.1%^[1]. In post-hoc sensitivity checks across 16 illustrative cost settings, the frozen calibration-selected policy beat both always-route and always-defer on test data in 14 settings, selecting always-route at the equal-cost boundary^[1].
• Accuracy gap and fairness caveat: On the 1,000-ticket subset, the best prompted configuration reaches 0.477 accuracy versus 0.879 for a fine-tuned small encoder model on the same tickets^[1]. A crucial caveat applies: trained models learn the taxonomy from 33,486 labelled tickets, whereas prompted models get only label names and at most 24 examples without category definitions—not a pure capability comparison^[1].
• Recalibration cannot create ranking discrimination: Every prompted model was overconfident^[1]. Isotonic recalibration cut expected calibration error (ECE) by 86–98% relative, but changed correctness AUROC by at most 0.007^[1]. One prompted configuration gave the same confidence (1.000) to every ticket, keeping correctness AUROC at 0.500—recalibration cannot create ranking information that is not in the score^[1].
• Deferring almost everything at break-even: At the 0.80 break-even threshold, seven of eight prompted conditions route 0 of 1,000 tickets (defer all), and the best one routes 19 of 1,000 (all correct)—a low-coverage observation, not evidence of a broadly useful policy^[1]. In the paper’s framing, “abstention does not repair a weak classifier”; it is “a guardrail, not a fix,” and an evaluation framework scoped to this dataset, prompts, endpoints, and illustrative costs rather than a new algorithm or universal deployment certification^[1].

A legal companion study shows the same "commit or abstain" problem when the correct answer is missing. Demir (2026) extends a legal holding multiple-choice benchmark with an abstention condition by replacing the correct holding in 1,670 of 5,314 items^[2]. When no option is correct, generative language models still pick one in 38–86% of items, whereas the most cautious decision model (returning calibrated probabilities per option rather than text) does so in 30%^[2]. Accepting a generative model's answer only when a decision model agrees halves its wrong citations—showing in a research setting that legal AI should be judged by its wrong citations, not by accuracy alone^[2].

Engineering Mapping on rlab.works: Separating Prediction, Confidence, and Routing Policy

In our engineering view, teams should write the three costs down per queue; keep predictor, confidence signal, and policy as separate, auditable stages; select the threshold on calibration data, freeze it, and test once in shadow mode; report discrimination (AUROC, risk–coverage) next to calibration error; and treat “defer-all” as a legitimate go/no-go result.

Across Agent OS, Commerce, and AI Gateway on rlab.works, three proposed skill packs and three components map to this, with adaptation boundaries:

• Intake and handoff for deferred tickets: In Agent OS, Intake and handoff (S09) runs confirm account and role → create service record → approve data access (human review) → hand over and record. When a router defers a ticket, the workflow records it in Support tickets and notification inbox (assignment, priority, department, history) and pauses at Auditable workflows, human confirmation and decision review for human triage. In the catalog, this skill pack carries a transferable tag only in healthcare—strictly for administrative service, with no symptom-based triage. For internal IT desks, it is a transferable pattern needing local identity and case-system wiring, with no proven IT-desk claim.
• Human review in support and entitlements: In Commerce, Check benefits and support (S36) runs check identity and benefits → calculate charges → review and decide handling (human review) → notify and retain history, marked transferable in consumer apps (where sign-in cannot imply paid entitlement).
• Shadow evaluation before switching models: In AI Gateway, Compare and switch models (S12) runs check candidates → compare sample results → review switch proposals (human review) → set primary/fallback → recheck samples (human review), marked transferable in enterprise settings (verify with actual samples; no promise of automatic optimal-model selection). Teams can use Model comparison and shadow evaluation to compare models on datasets and collect shadow evidence before go-live.

Current limits must be stated plainly. All three skill packs are proposed compositions demonstrated on synthetic data. The ticket inbox and shadow evaluation components are rated working from static inspection only, while the auditable workflow component is a prototype whose tool steps are records and placeholders. Crucially, none implements the paper’s abstention framework or a cost-threshold router.

Three Pre-Launch Checks for Cost-Aware Routing Readiness

• Write down queue costs and calculate the break-even threshold: Never treat the paper’s illustrative 0/1/5 numbers as real costs; estimate the cost of a correct auto-route, human triage, and a wrong route per queue to set the minimum correctness probability required for auto-routing.
• Freeze the calibration threshold and report discrimination beside calibration error: Keep prediction, recalibration, and policy separate; if recalibration lowers ECE but correctness AUROC stays near random—or if the frozen policy defers nearly everything at break-even—treat that as a valid “no-go” result and consider a classifier trained on labelled tickets as the comparison.
• Make deferral to a person a first-class outcome: Route every abstained ticket into a human queue with candidate classes, calibrated scores, and an audit record, and re-check when costs, prevalence, or models change.