Scan modes
Start with a quick signal, run the complete standard evaluation, or expand into architecture-specific voice, tool, memory, security, and continuous regression evidence.
Intensity ladder
Quick, Standard, and Adversarial compose the current 60 runnable cases. Deep and Continuous require the target architecture, endpoints, permissions, and telemetry before execution.
A fast risk screen using three representative cases from every runnable failure category.
Discovery, sales qualification, and pre-release triage
The complete runnable Chat and RAG suite, executed against baseline and fixed variants.
A defensible initial audit with before-and-after evidence
Concentrates the current suite on instruction injection, unsupported claims, and conflicting knowledge.
Public agents, sensitive knowledge, and pre-production hardening
Adds channel, tool, memory, authorization, voice, handoff, and recovery tests selected from the customer architecture.
Voice agents, tool-using agents, and multi-agent workflows
Promotes proven cases into a recurring regression gate with drift, latency, and failure alerts.
Production systems that change prompts, models, data, or tools
Run locally
npm run eval:quicknpm run evalnpm run eval:adversarialHow selection works
Discovery, launch approval, remediation proof, or continuous release gate.
Add only the channel, retrieval, tool, memory, privacy, and handoff packs that apply.
Hash the selected cases, fixtures, rubrics, model settings, and expected outcomes.
Use the same manifest before and after the fix so the improvement is defensible.
The app can execute the 60-case Chat/RAG suite today. Voice and tool-using agent definitions remain blueprints until a target adapter, safe test environment, and telemetry contract are connected.