Confident Report, Jagged Evidence: Claim Smoothing and Evidence Balance in Research Agents

Every Citation Is Real—Why Is the Conclusion Still Overstated?

Consider an analyst reviewing a report drafted by a research agent. Every paragraph ends with links to real sources. The reviewer spot-checks a few links and approves. Yet where the cited papers showed only that a method worked in one narrow setup and failed under high load, the report states that the evidence “definitively proves” the method is superior in general.

This is neither citation fabrication nor misleading tool output. The citations are real and the tools work properly; the issue is the strength, scope, and balance of the conclusion. Checking only “is there a citation?” misses two quieter failures: agents smooth over mixed or narrow evidence to state conclusions more decisively or broadly than the cited documents warrant^[1], and factually accurate but one-sided document pools shift the answer’s stance^[2].

From Single-Paper Generalization Bias to Claim Smoothing and Stance Poisoning

Over-generalization appears even in single-paper summaries. Peters and Chin-Yee (2025) compared 4,900 model-generated summaries across 10 prominent large language models with the original scientific texts: even when explicitly prompted for accuracy, most models produced broader generalizations than the originals, with three models over-generalizing in 26–73% of cases^[3]. In a direct comparison, model summaries were nearly five times more likely than human-written summaries to contain broad generalizations (odds ratio 4.85), and newer models tended to do worse^[3].

In multi-source research agents, Roy, Wadhwa, and Durrett (2026) define claim smoothing by three conditions: (1) a claim is a conclusion drawn from aggregated evidence, (2) some evidence supports at least part of it, and (3) the conclusion as a whole is stronger or broader than the body of evidence warrants—whereas a faithful claim hedges when evidence is mixed and narrows scope when support is narrow^[1]. The authors extract conclusive claims from a report, run a “support” pass and a “skeptic” pass (evidence that qualifies, limits, or contradicts) over the report’s own cited documents, cluster evidence into aspects, and have a model judge score each claim 1–5 (3 or above = materially smoothed, 4 or above = severe)^[1]. Checked against humans on 73 claims, the judge was within one point of a human annotator on 93% of claims, similar to agreement between two human annotators^[1].

Across seven systems and four research benchmarks, materially smoothed claims appeared for every system on every dataset—at least 22% of claims in every case, and over half for most systems^[1]. Severe smoothing ranged from 1–4% of claims for the two least-smoothing systems to 22–35% for the most-smoothing one^[1]. Three patterns recurred: unsupported composition (combining individually supported pieces into a stronger conclusion), scope expansion (stating a result from one setup as general), and premature closure (phrases like “definitively proves” or “the best approach”)^[1]. Comparative questions (“which is more promising and why?”) and topic-overview questions invite smoothing^[1]. The authors link part of the smoothing to what agents actually read: one system opened only 19% of the pages it cited (working mostly from search snippets of median 585 characters), and another opened 2%^[1].

In an intervention on one benchmark (61 questions, two systems), neither calibration instructions alone nor full documents alone was sufficient; both together cut materially smoothed claims from 29% to 12% and from 61% to 25%, while severe smoothing nearly disappeared (0.5% and 2.2%)^[1]. Reports did not get shorter; claims kept their takeaway and added limits^[1]. Limits include reliance on a model judge (validated on 73 claims), research-style benchmark questions, and negligible association within a system between document count or claim length and smoothing, showing other factors matter^[1].

The evidence pool itself can also tilt an answer. Yang, Dong, Wu, Cheng, and Li (2026) tested 210 controversial questions over knowledge bases with injected documents that are factually grounded yet argue one side, across four poisoning concentrations, two directions, and four generation models from different vendors^[2]. At a mean topic-level poisoning share of 18.9%, all four models shifted stance in the poisoning direction, increasing monotonically with concentration^[2]. Checking individual claims alone cannot tell whether evidence represents competing positions—cited material can be verified yet one-sided^[2].

Their defense places a stance-balancing layer between retrieval and generation: stance detection, a fixed quota of 2 supporting + 2 opposing + 1 neutral documents, and a debiasing prompt, with no extra model calls^[2]. The full method reduced stance deviation magnitude by an estimated 0.106 to 0.350 across all eight settings, leaving small residual tilt (+0.045 to +0.063) in one direction in the cross-model test; stance detection for 20 candidate documents took about 1.5% of median generation time on one GPU^[2]. Limits include English-only data, one retriever, one question source, model-generated slanted documents, and dependence on available stance categories and detector labels; it applies to genuinely contested questions, not a rule that every factual question needs “both sides”^[2].

Engineering Mapping on rlab.works: Claim Review, Read Logs, and Stance Quotas

In our engineering view, teams should: (1) review at claim level, showing supporting and qualifying evidence side by side and asking “is this as strong and as broad as the evidence?”, not only “is it cited?”; (2) record whether each cited source was read in full (not just a snippet) and flag snippet-only claims; (3) for contested questions, fix the evidence set composition (supporting, opposing, neutral) before writing; (4) give the writer both full sources and calibration instructions; (5) watch comparative and overview questions; and (6) show counter-evidence and scope limits at human gates so reviewers can downgrade wording.

Across Intelligence and Agent OS on rlab.works, three proposed skill packs and three components map this structure:

• Research reports: In Intelligence, Research and write a report (S24) runs Plan the search scope → Collect and extract material → Organize citations and findings → Review the evidence (human gate) → Format and export, using Bounded cross-provider research tool loop (iteration-capped search-tool loop returning citations plus step evidence) and Grounded document summarization (summaries with citations, source metadata, and labeled fallbacks). The pack is transferable to legal work (replace paper sources and news topic models; review legal validity and case applicability; do not automate legal conclusions). In our view, Review the evidence (human gate) should show qualifying evidence and snippet-only flags.
• Follow-up research: In Agent OS, Continue research (S08) runs Load research context → Plan follow-up research → Approve and run (human gate) → Review the changes (human gate). The pack is transferable to finance (research execution still needs wiring to general search and enrichment queues, so the whole pack is not labeled source-supported finance). In our view, Review the changes (human gate) should check whether new evidence narrows scope on comparative or overview questions.
• Daily briefings: In Intelligence, Build a daily brief (S20) runs Collect readable material → Structure facts and entities → Link topics and events → Verify and select (human gate) → Prepare and deliver the brief, with Event, entity and narrative graph construction (linking articles into events and narratives and extracting participants and stances). The pack is source-proven in media (configure sections and subscription scope; editorial responsibility stays at the review step). In our view, the extracted stances could feed a balance check on contested topics before Verify and select (human gate).

All three packs are proposed, unintegrated compositions demonstrated with synthetic data. The three components have “working” static-inspection maturity, and none scores claim strength or scope, enforces an evidence-stance quota, or checks whether a cited page was opened—those are requirements we draw from the papers. Nothing here is legal, financial, or editorial advice.

Three Pre-Deployment Checks for Research Agents

• Verify full-text read logs and dual inputs: Log whether each cited source was opened in full or only as a snippet, flag snippet-only claims, and provide both full documents and calibration instructions.
• Enforce stance quotas on contested questions: On genuinely debated questions, balance supporting, opposing, and neutral documents before generation so a verifiable but one-sided pool cannot tilt the report.
• Show counter-evidence and downgrade controls at the human gate: Display qualifying and opposing evidence at Review the evidence (human gate), Review the changes (human gate), and Verify and select (human gate) so reviewers can downgrade wording.