AI Gateway · Component · model-evaluation-shadow
Model comparison and shadow evaluation
What it does
Compare primary/candidate models over datasets with optional model judging; collect shadow and routing-feedback evidence.
Core pattern
Shared-sample model comparison + quality and cost records
- Input
- Evaluation samples or shadowed traffic
- Output
- Quality, latency and cost comparison reports
Key parameters
| Name | Type | Default | Values |
|---|---|---|---|
| operation | string | compare | compare · record_run · shadow_report |
| cases | array | — | — |
| primary | object | — | — |
| candidate | object | — | — |
| budget_usd | number | 0 | 0–1000000000 |
Maturity and limits
Evaluation input/report contracts, shadow service and unit tests. Phase 2: source paths and implementation inspected statically; tests and external integrations were not executed. Maturity rates repository implementation, not a verified production deployment.
config with credentials present. MIT; retain notices. Provider compatibility is configuration-dependent; metadata/privacy and optional payload plugins need review. Static inspection only.
1 sentence citing source paths or vendor names omitted for publication.
Used in pipelines
Industry tags
- EducationTransferable use
For sample acceptance checks before releasing a course question assistant, use shared-sample model comparison followed by quality and cost records.
- Business & general servicesTransferable use
Support “Compare shared question samples” with shared-sample model comparison followed by quality and cost records.
Skills pack
Contract scaffold and deterministic mock
Download · 12KBSHA-256 a4b6b089a72eea88c1fc00551ea93531aece49e61eff5daf372e427ed56d8720
Related components
-
Multi-protocol model routing, fallback and compatibility
Request protocol normalization + compatible routing and fallback
Working 1 pipeline -
Provider batch inference
Batch request submission + job state and result archives
Working 1 pipeline -
Routing recommendation review
Review stored recommendations + proposed policy conversion
Working 1 pipeline -
AI endpoint management and testing
Service configuration and connectivity tests + response inspection
Prototype 2 pipelines