[Finance I] Source Grounding in the Flood of Filings: Why Investment Research Memos Must Never Cross Over into Automated Trading Orders

The Chasm Between Headline Sentiment Scoring and Filing Verification

Consider a common situation inside an asset management firm during earnings season. Analysts face a dense wave of regulatory filings, earnings releases, and breaking news. Hoping to compress review times, the team experiments with feeding disclosures and news feeds into a large language model to draft investment summaries—and even debates wiring the model’s bullish or bearish scores directly into an order-generation interface. During internal review, however, a sharp divide emerges. While the model extracts directional sentiment from brief news headlines, the workflow breaks down when the investment committee asks for the exact footnote behind a capital expenditure figure, an off-balance-sheet contingency, or a margin forecast. Model-generated paragraphs frequently lack verifiable links to the underlying filing or misread routine accounting terms as distress signals. If unverified drafts trigger orders directly, upstream parsing errors become real portfolio exposure.

This tension highlights a foundational boundary for artificial intelligence in finance: the value of investment research resides in a traceable chain of evidence connecting every claim to its original disclosure. In system architecture, generating a research evidence memo must remain strictly separated from investment-committee decisions and trade execution.

Return Predictability, Domain Tone Misclassification, and Open-Book Financial QA Limits

Why can a model’s ability to capture market sentiment not justify bypassing human review between research memos and trading desks? Empirical studies in financial text analysis and model benchmarking clarify the mechanisms involved.

First, large language models do exhibit genuine capability in interpreting news headlines and predicting equity returns. Lopez-Lira and Tang (2023) documented that large language models can forecast stock market reactions from news headlines without direct financial training^[1]. Using headlines published after the model’s knowledge cutoff, they showed that a frontier model captures initial market responses, achieving approximately 90%^[1] portfolio-day hit rates for the non-tradable initial reaction, while its scores also significantly predict subsequent return drift, especially for small stocks and negative news^[1]. Their study further found that forecasting ability generally increases with model size—suggesting financial reasoning is an emerging capacity of complex models—whereas strategy returns decline as model adoption rises, consistent with improved price efficiency^[1]. To rationalize these findings, the authors developed a theoretical model incorporating large language model technology, information-processing capacity constraints, underreaction, and limits to arbitrage^[1].

Yet systems architects must distinguish between scoring news headlines and verifying regulatory disclosures. Headline sentiment captures rapid reactions and underreaction drift across news flows; institutional research memos interrogate structured filings where accounting and legal terms carry domain-specific meanings. Studying 10-K filings, Loughran and McDonald (2011) showed that measuring tone through negative word counts misclassifies common financial words when relying on word lists built for other disciplines^[2]. Across a large sample of filings during 1994 to 2008, almost three-fourths of the words identified as negative by a widely used general-purpose dictionary are words not typically considered negative in financial contexts—a routine balance-sheet liability does not indicate corporate distress^[2]. The authors developed an alternative negative word list along with five other word lists that better reflect financial tone, linking them to filing returns, trading volume, return volatility, fraud, material weakness, and unexpected earnings^[2].

Does pairing modern large language models with retrieval systems resolve the challenge of answering concrete questions from corporate filings? Islam et al. (2023) introduced FinanceBench, an open-book financial question-answering benchmark comprising 10,231 ecologically valid, clear-cut questions about publicly traded companies with corresponding answers and evidence strings to serve as a minimum performance standard^[3]. Testing 16 state-of-the-art model configurations—including vector stores and long-context prompts—on a sample of 150 cases and manually reviewing 2,400 answers, the authors found clear limitations: notably, a frontier model configuration used with a retrieval system incorrectly answered or refused to answer 81% of questions^[3]. While feeding relevant evidence into longer context windows improved performance, such augmentation is unrealistic for enterprise settings due to increased latency and inability to support larger financial documents; all examined models exhibited weaknesses such as hallucinations that limit enterprise suitability^[3].

Together, these findings establish a clear engineering conclusion. Large language models can help surface candidate signals, but open-book question answering over financial disclosures remains vulnerable to retrieval misses and hallucinations. An investment research platform should not issue orders on a portfolio manager’s behalf, but organize disclosures, claims, and counterevidence into an auditable ledger for human review.

Engineering Mapping: Controlled Pipelines from Evidence Memos to Multi-Week Tracking

Within the capability architecture on rlab.works, source grounding and the separation between research and trading are implemented through explicit workflow contracts rather than open-ended chat.

Under the Intelligence module, the skill pack Keep an evidence memo (S25) takes structured initial inputs—research entities, claims, and existing evidence—and organizes work into sequential stages:

• Collect source evidence (assisted): Invoking Disclosure collection and text extraction, the workflow gathers announcement metadata across US, A-share, and Hong Kong markets and cleans filing text from JSON, HTML, and text-layer PDFs, outputting a dated source bundle with provenance.
• Resolve entities and relations (assisted): The system reconciles identifiers, tickers, and reporting periods to produce an organized entity-relation map and evidence index.
• Compare claims and challenges (assisted): Using Investment thesis evidence and verdict tracking, the workflow validates multi-week thesis evidence, grades sources, separates supporting, contradicting, and complicating factors, and derives weekly verdicts open to human override—outputting a record of support, counterevidence, and uncertainty.
• Review and save the memo (human review): An assigned reviewer or requesting user inspects the evidence chain; once accepted, Investment memo generation uses the market and disclosure context alongside existing research or backtest artifacts to save the approved research memo and evidence links. Rejected outputs return for revision and do not advance.

Regarding industry boundaries, Keep an evidence memo (S25) is tagged in finance as a source-supported industry workflow: its components carry explicit financial data associations and bring disclosures, market series, and counterevidence into a shared memo so reviewers can trace each judgment’s origin. Still, entity and time conventions must be unified during implementation, and the output is strictly for research and never generates trading orders. Adapted to legal work, it becomes a transferable use: financial disclosures, market data, and corporate models must be replaced with case files, statutes, parties, and timelines, and original scores cannot be used as legal judgments.

Because institutional research spans reporting cycles across weeks, the Agent OS module offers [Continue research (S08)](/skills/continue-research-tasks/). Taking a research question, existing material, and available time as input, it runs through loading research context (assisted), planning follow-up research (assisted), approving and running bounded batches (human review), and reviewing changes (human review), where rejected outputs at human gates return for revision. In finance, Continue research (S08) is a transferable use: it carries prior evidence across weeks and saves updated conclusions after confirmation, though research execution, general search, and enrichment queues still require integration wiring.

Honesty about current status is essential. On rlab.works, both Keep an evidence memo (S25) and Continue research (S08) are proposed compositions demonstrated with synthetic data and not yet connected to live services. Component working-code ratings reflect static source inspection rather than production deployment, and a source-supported industry label means only that source data has an explicit industry association—not customer deployments or business outcomes.

Pre-Pilot Governance Checks for Research and Risk Leaders

Before funding an artificial intelligence pilot for investment research, a Head of Research, Chief Risk Officer, or Chief Technology Officer can apply the following verification checks:

• Does the memo schema mandate side-by-side counterevidence and filing provenance? Require every draft memo to retain supporting evidence, contradicting evidence, and uncertainty notes anchored to dated filing passages rather than emitting unilateral recommendations.
• Do human review gates hold blocking authority over follow-up runs and memo archiving? Confirm in the state machine that unapproved research plans and rejected drafts return for revision and cannot enter the research repository or committee packets.
• Is the research workspace architecturally isolated from order-execution interfaces? Verify at the permission and API level that the workflow ends at a traceable research memo, ensuring model scores or summaries never cross over into automated trading orders.