Intelligence · Component · news-feed-ingestion-classification
Scheduled news/feed ingestion and deduplication
What it does
Poll RSS/API feeds, retain fetch state and deduplicate articles across four source variants. Source registries, customer taxonomies and adapter-specific behavior require configuration.
Core pattern
Scheduled feed ingestion + deduplication and fetch state
- Input
- RSS/RSSHub/JSON feeds
- Output
- Normalized articles, source provenance and fetch job records
Key parameters
| Name | Type | Default | Values |
|---|---|---|---|
| sources | array | [] | — |
| poll_interval_seconds | integer | 900 | 30–86400 |
| batch_size | integer | 50 | 1–1000 |
| parallelism | integer | 4 | 1–64 |
| dedupe_strategy | string | hybrid | canonical_url · external_id · title_content · hybrid |
Maturity and limits
Source registry, adapter contracts, scheduler and fetch/admin endpoints exist. Source paths were inspected statically; tests and live integrations were not executed. This rating describes implemented code, not production validation.
No root reuse license found; publisher text/images require reuse rights; external model and crawler services require configuration. Variant-specific dependencies, limits and weaker maturity are recorded under variants; these implementations are alternatives, not one deployed system.
Used in pipelines
Industry tags
- Media & contentSource-supported industry
For work involving editorial material, programs and reader services, use scheduled feed ingestion followed by deduplication and fetch state.
- EnergyTransferable use
For equipment supplier and regulatory update briefs, use scheduled feed ingestion followed by deduplication and fetch state.
Skills pack
Contract scaffold and deterministic mock
Download · 13KBSHA-256 5d1958a1f05a0e2fc1b04ef1e81beb4a7ca32d90a7475590f7f6870fd069fb8c
Related components
-
Structured news intelligence and multilingual analysis
Source text to structured facts + uncertainty and human review
Working 1 pipeline -
Source-backed entity, company and people enrichment
Entity profile and alias reconciliation + evidenced enrichment
Working 1 pipeline -
Daily intelligence refresh and report packaging
Scheduled refresh orchestration + readiness-gated reporting
Working not yet in a pipeline -
Article and research-page extraction and enrichment
Page boilerplate removal + fallback and placeholder checks
Working 1 pipeline