Architecture profile
Describe the deployed system: channel, model, retrieval, tools, memory, data boundaries, and observability.
Master architecture
Start from the system actually deployed.
Choose a common architecture, add the capabilities that make it yours, and generate the test packs and evidence contract required for a defensible audit.
15base templates
1runnable now
2suites ready
Profile composer
One base pattern, composable capabilities.
The generated manifest makes the suite selection explicit. It will be versioned with future reports so reruns compare the same system.
Template catalog
Fifteen common AI system patterns.
Templates are starting points, not provider lock-in. Open one to inspect its flow, components, risks, and evidence requirements.
01Blueprint
Chat
Prompt-only chat assistant
Client → application API → model → response
Inspect template →02Blueprint
Chat
Conversational assistant with memory
Client → orchestrator ↔ memory store → model
Inspect template →03Runnable now
Knowledge
Basic RAG knowledge assistant
Question → retrieval → context → model → grounded answer
Inspect template →04Blueprint
Knowledge
Advanced RAG with reranking
Query → hybrid search → filters → reranker → model + citations
Inspect template →05Blueprint
Knowledge
Agentic RAG
Question → planner ↔ retrieval loop → synthesis → answer
Inspect template →06Suite ready
Agents
Tool-calling transactional agent
User → model ↔ authorized tools → verification → response
Inspect template →07Blueprint
Agents
Workflow and state-machine agent
Trigger → state machine → model/tool steps → terminal state
Inspect template →08Blueprint
Agents
Multi-agent supervisor and workers
Request → supervisor → specialist agents → synthesis
Inspect template →09Suite ready
Voice
Cascaded telephony voice agent
Twilio ↔ voice gateway ↔ STT → orchestrator → LLM/RAG/tools → TTS ↔ Twilio
Inspect template →10Blueprint
Voice
Native realtime speech-to-speech
Audio client ↔ realtime model session ↔ tools
Inspect template →11Blueprint
Customer service
Omnichannel service agent
Channels → identity/router → shared orchestrator → channel response
Inspect template →12Blueprint
Customer service
Human-in-the-loop contact center
Customer ↔ AI → routing/approval queue ↔ human agent
Inspect template →13Blueprint
Automation
Document intelligence pipeline
Document → OCR/parser → model extraction → validation → system of record
Inspect template →14Blueprint
Copilots
Data and SQL analytics copilot
Question → semantic layer → query planner → governed database → explanation
Inspect template →15Blueprint
Multimodal
Multimodal vision-and-text agent
Text + media → preprocessing → multimodal model → RAG/tools → response
Inspect template →Evaluation pipeline
From system design to proof.
The engine keeps the target system, runner, grader, and dashboard independent.
Risk model
Map each component and data flow to the failures it can create and the business impact of those failures.
Suite composer
Select deterministic, adversarial, audio, workflow, and model-graded tests that fit the actual architecture.
Runner adapters
Exercise chat endpoints, voice streams, model providers, tool APIs, and workflow orchestrators through stable contracts.
Evidence normalization
Capture responses, retrieval, tool calls, timing events, audio artifacts, state changes, and trace identifiers.
Grading and report
Apply hard rules and semantic rubrics, diagnose root causes, then compare the same suite before and after a fix.
Architecture profile
What every audit records.
A versioned profile travels with every report so future reruns remain comparable.
Channel
Chat, voice, SMS, email, multimodal
Model
Provider, model, temperature, prompt version
Knowledge
RAG store, ranking, freshness, citations, confidence
Tools
Schemas, authorization, side effects, approval gates
Memory
Scope, retention, isolation, summarization
Voice
Telephony, STT, VAD, TTS, latency budgets
Workflow
Planner, retries, state, idempotency, termination
Observability
Trace IDs, spans, scores, costs, timing events
Risk selection
The suite changes with the architecture.
| Component present | Required tests | Evidence captured |
|---|---|---|
| RAG / knowledge retrieval | Grounding, stale conflicts, missing context, citations | Query, retrieved IDs, rank, scores, source versions |
| Voice channel | STT accuracy, latency, interruptions, silence, TTS | Audio, transcript, per-turn timing, VAD events |
| Action-taking tools | Authorization, argument safety, confirmation, idempotency | Principal, tool inputs/outputs, approval and state |
| Long-term memory | Isolation, stale memory, consent, sensitive disclosure | Session identity, memory reads/writes, retention policy |
| Multi-step planner | Loop termination, retries, partial failure, verification | Plan, step count, state transitions, final outcome |