Intelligence · Component · scrape-export-archive
Scheduled scrape exports and object archive
What it does
Run recurring collection, produce run summaries and upload exported artifacts to R2 before importing portal signals.
Core pattern
Recurring collection jobs + summaries and archive manifests
- Input
- Collection job configuration
- Output
- Run archive, upload manifest and portal import result
Key parameters
| Name | Type | Default | Values |
|---|---|---|---|
| operation | string | collect | collect · export · archive · import_signals |
| schedule | — | — | — |
| timezone | string | UTC | — |
| source_registry | — | — | — |
| output_dir | — | — | — |
Maturity and limits
Daemon/job wrappers, boto3 storage adapter and auto-import orchestration. Phase 2: source paths and implementation inspected statically; tests and external integrations were not executed. Maturity rates repository implementation, not a verified production deployment.
No root license found; source/platform terms and affiliate disclosures apply. Browser/provider credentials must be isolated; displayed returns are illustrative, not executed investments.
1 sentence citing source paths or vendor names omitted for publication.
Used in pipelines
Industry tags
- FinanceSource-supported industry
For work involving research materials and financial service operations, use recurring collection jobs followed by summaries and archive manifests.
- Government & public servicesTransferable use
For monitoring public-service notices and publicly posted resident feedback, use recurring collection jobs followed by summaries and archive manifests.
- SportsTransferable use
Support “Archive after editorial review” with recurring collection jobs followed by summaries and archive manifests.
Skills pack
Contract scaffold and deterministic mock
Download · 10KBSHA-256 9b9970f831b419288db2beca9d29552897b90eb0f8f0b2fd0bbdf731cdb9dd43
Related components
-
Structured news intelligence and multilingual analysis
Source text to structured facts + uncertainty and human review
Working 1 pipeline -
Source-backed entity, company and people enrichment
Entity profile and alias reconciliation + evidenced enrichment
Working 1 pipeline -
Daily intelligence refresh and report packaging
Scheduled refresh orchestration + readiness-gated reporting
Working not yet in a pipeline -
Reusable B2B signal dataset registration
External dataset contract registration + normalized ingestion
Working 1 pipeline