More Agents, Better Review? The Miss–False-Alarm Trade in Divided Agent Review

Why Splitting a Checklist Across Agents Does Not Simply Make Review Safer

When teams automate reviews of credit memos, legal filings, or shift logs, a single prompt checking several requirements at once can miss subtle defects. Common guidance is to divide the review among several agents, or wire agents into a debate-and-aggregate pipeline. When task scores rise on a benchmark, teams often read that gain as a more trustworthy system. In operation the result can look different: reviewers may face more false alarms on clean files, while polished summaries can still rest on unsupported claims in the stated reasoning.

This gap comes from mistaking an operating-point shift for higher reliability. Controlled studies show division mainly moves the operating point: it catches some more defects and raises more false alarms, costs more tokens, and does not by itself reduce hallucination in the stated reasoning. The right comparison is an equal-call single-agent baseline, scored on misses and false alarms separately on known-clean cases, with the reasoning instruction fixed and reported.

Operating-Point Shifts and Capability Redistribution: What Controlled Studies Show

Cemri et al. (2025) note that performance gains of multi-agent systems on popular benchmarks are often minimal^[3]. Analyzing 1,600+ annotated traces across seven popular multi-agent frameworks—and building a taxonomy with expert annotators from 150 traces—they identify 14 failure modes in three categories: system design issues, inter-agent misalignment, and task verification^[3].

Liu (2026) isolates division on a financial review task^[1]. Reviewing credit assessment memos against six written requirements across 300 synthetic memos (program-generated from a public credit-card dataset record, ~1,100 words each) yields 1,800 labelled requirement cells: 1,136 clean, 205 bare, 229 masked (with their own explanation), and 230 arithmetic defects—defects in 37% of cells, far more than in practice^[1]. With materials and analysis fixed by committed hash before runs and two primary predictions tested on four models at a Bonferroni-corrected level, a single agent checking all six requirements in one call is compared against a team of three on the same model checking two requirements each, in identical words, with no roles and no later stage^[1].

Under a “decide each requirement once” instruction, the team catches 0.08 to 0.10 more masked defects on the three models whose single agent misses some (significant at corrected level on two), but raises more false alarms on all four models (corrected level on three)^[1]. The share of the 300 memos with at least one false alarm is higher for the team on every model, by 0.04 to 0.19; exploratory balanced accuracy changes by 0.02 or less, while the team uses 2.2 to 2.4 times the tokens^[1]. Without that reasoning instruction, the team’s excess of false alarms is larger on all four models^[1].

Showing each agent the requirements it was *not* given ("scoped team") reduces added false alarms on two models (corrected level on one)^[1]. Meanwhile, the fair same-call baseline—sampling the single agent three times and flagging on any draw ("any-of-three")—gives higher exploratory balanced accuracy than the team on two models; in an exploratory expected-cost analysis at a 2% defect share, the team had the lowest expected cost at none of the miss/false-alarm cost ratios tried, on any model^[1]. As the author recommends: "A divided review should be judged on misses and false alarms separately, on cases known to be clean, and against the single agent given the same number of calls."^[1] Given short synthetic memos, an unrealistically high defect share, one pairing (two requirements per call), a strongest model at ceiling, and false-alarm cost probably overstated by misfiled flags on some models, these rates should not be assumed to carry over to long documents with rare defects^[1].

Jang, Joshi, and Yu (2026) evaluate four systems (a single-model baseline and three multi-agent topologies built from aggregate, reflect, and debate blocks, each ending in an aggregator) on four models and three gold-labelled clinical tasks^[2]. Using reasoning-only systems without tools or retrieval, they score hallucination inside the rationale (atomic claims under a factuality and faithfulness taxonomy) alongside the task metric^[2].

Collaboration buys task performance but not lower hallucination as a rule, and no topology is reliably better than another^[2]. Of 36 model–system–task comparisons, 12 are large enough to resolve: four decreases (all one model) and eight increases (spread over the other three models), with no model on both sides—the model, not topology or number of agents, sets the level and direction^[2]. Scaling each topology from three to seven agents has minimal effect; swapping only the aggregator’s model moves hallucination across most of its range, landing near what its stronger member achieves alone^[2]. Shown its own flagged errors in a self-correction test, every model except one reduces hallucination; the model collaboration helps gains most from correction, while the one it harms most gains nothing—consistent with, not proof of, self-correction as mechanism^[2]. Collaboration “redistributes what its models can already do; it does not supply what they cannot.”^[2]

Engineering Bounded Review in Agent OS and Intelligence

In our view, before adopting a split, teams should build a labelled set with known-clean cases; report miss and false-alarm rates separately (per file, not only per cell); compare against the single agent at the same call budget (three draws with an any-of-three or majority rule); fix and record the reasoning instruction; if splitting, show each sub-reviewer what others cover and keep each flag tied to the quoted sentence; decide by miss versus false-alarm cost and real defect share; and ensure a human reviewer at the gate owns the verdict.

Within Agent OS and Intelligence on rlab.works, three skill packs and three components can be read against these boundaries:

• Scoping splits in Agent OS: Plan and deliver work runs through plan the steps → approve the scope (human gate) → run and collect → review and hand over (human gate), where teams decide whether a review is split and how it is judged. It is transferable in manufacturing (software orchestrates inspection, preparation, and review records while physical work stays with personnel under permits) and a general pattern elsewhere. Persistent goals, bounded subagents and scheduled tasks (working maturity) bounds sub-agent capabilities, depth, and concurrency, while Collaborative delivery, approvals and evidence gates (working maturity) enforces review gates—expressing configured requirements, not guaranteed business success.
• Weighing evidence in Intelligence: Keep an evidence memo follows collect source evidence → resolve entities → compare claims and challenges → review and save the memo (human gate). Credit-memo checking in ^[1] is analogous, not a site case. In finance it has a source-supported association (source-tag association only); entity and timing definitions still need alignment during adaptation, and outputs support research rather than trade instructions. Investment thesis evidence and verdict tracking (working maturity) separates support, challenge, and complication; verdicts are overrideable by humans, and editorial heuristics are not financial forecasts.
• Source-grounded reporting in Intelligence: Research and write a report is transferable in legal workflows: adaptation requires reviewing legal validity and case applicability rather than automating legal conclusions.

All skill packs are proposed compositions on synthetic data; component maturity reflects static source inspection.

Pre-Deployment Checklist Before Dividing an Agent Review

Before splitting a review across agents, engineering and risk leaders should apply three checks:

• Benchmark against an equal-call single agent on known-clean files: Score file-level and cell-level false alarms on known-clean cases and miss rates on explained (masked) defects against a single agent given the same number of calls at realistic defect prevalence.
• Fix the reasoning instruction and expose out-of-scope requirements: Lock the decision instruction in the workflow contract, show sub-reviewers the items assigned to others, and require every flag to quote its source sentence.
• Prioritize the aggregator model and keep verdict authority at the human gate: In the settings tested, extra agents did little for rationale hallucination while swapping the aggregator model mattered more^[2]; invest model capability in synthesis and keep final sign-off with the human reviewer.