Agent OS · Component · retrieval-ranking-boundary-evaluation
Retrieval ranking and authorization-boundary evaluation
What it does
Evaluate dense/hybrid retrieval with relevant and forbidden documents in a typed multilingual corpus. Report rankings, top-K hits and authorization-boundary pass rates; this is not an end-to-end factual-answer benchmark.
Core pattern
Relevant and forbidden sample tests + retrieval boundary reports
- Input
- Corpus documents/cases and embedding adapter
- Output
- Per-case ranks and aggregate pass/fail report
Key parameters
| Name | Type | Default | Values |
|---|---|---|---|
| operation | string | dense_evaluate | dense_evaluate · hybrid_evaluate · assert_pass |
| corpus | object | — | — |
| embedding_adapter | object | — | — |
| top_k | integer | 3 | 1–1000 |
| minimum_success_rate | number | 0.9 | 0–1 |
Maturity and limits
evaluateRetrievalCorpus filters allowedActorKeys before cosine ranking, checks forbidden IDs across ranked candidates and gates aggregate success; hybrid counterpart applies retrieval scoring. Parser currently fixes schemaVersion=1 and topK=3. Source inspected statically; tests and external integrations were not executed.
Authorization is fixture-declared; it does not prove production ACL correctness. Top-K and boundary thresholds require parser changes to generalize beyond current constants.
1 sentence citing source paths or vendor names omitted for publication.
Used in pipelines
Not part of a pipeline yet; available as a standalone component.
Industry tags
- EducationTransferable use
For sample acceptance checks before releasing a course question assistant, use relevant and forbidden sample tests followed by retrieval boundary reports.
Skills pack
Contract scaffold and deterministic mock
Download · 11KBSHA-256 8a7cdbbd6f0d44c139005b370894edd1f0dfd7caa91999dfa060bdd6f41c0e90
Related components
-
Local coding-agent execution governance
Pre-action policy decisions + human permission and audit
Working 1 pipeline -
Collaborative delivery, approvals and evidence gates
Draft and artifact versioning + human handoff approval
Working 4 pipelines -
Tenant identity, enterprise SSO, provisioning and access control
Organization membership sync + scoped role access
Working 1 pipeline