Intelligence · Component · article-body-extraction
Article and research-page extraction and enrichment
What it does
Extract readable article/research text with provider fallbacks, remove boilerplate and reject placeholders. Three variants differ in crawling, refinement/translation and persistence.
Core pattern
Page boilerplate removal + fallback and placeholder checks
- Input
- Article URL and feed metadata
- Output
- Enriched full text, images and content blocks
Key parameters
| Name | Type | Default | Values |
|---|---|---|---|
| operation | string | extract | extract · clean · translate · enrich |
| urls | array | — | — |
| model_config | — | — | — |
| formats | array | [] | — |
| language | — | — | — |
Maturity and limits
No root reuse license found; publisher text/images require reuse rights; external model and crawler services require configuration.
4 sentences citing source paths or vendor names omitted for publication.
Used in pipelines
Industry tags
- Media & contentSource-supported industry
For work involving editorial material, programs and reader services, use page boilerplate removal followed by fallback and placeholder checks.
- Legal servicesTransferable use
For clause lookup in an authorized contract library, use page boilerplate removal followed by fallback and placeholder checks.
- EnergyTransferable use
Support “Collect equipment and regulatory material” with page boilerplate removal followed by fallback and placeholder checks.
Skills pack
Contract scaffold and deterministic mock
Download · 12KBSHA-256 1b16a369c89e70391de6ed06739aca915e696d4d6cc62cf4153cc2ddd9e1a2ea
Related components
-
Multi-image OTA rate extraction
Paired screenshot field recognition + structured comparison records
Working 1 pipeline -
Disclosure collection and text extraction
Document index collection + cleaned text and normalized references
Working 1 pipeline -
Legacy data and provenance migration
Legacy record mapping and import + orphan and completeness checks
Working 1 pipeline -
Scheduled news/feed ingestion and deduplication
Scheduled feed ingestion + deduplication and fetch state
Working 1 pipeline