Voice-agent audit

Blueprint

Evaluate the whole turn, not only the transcript.

A correct text response can still be a failed voice experience. The voice runner will score recognition, time-to-first-audio, interruption behavior, speech quality, tools, handoffs, and the final spoken answer.

Designed cases10

Reference scenarios ready to become audio fixtures.

Runner neededVoice stream

Telephony or WebSocket adapter with timestamped events.

Core evidenceAudio + time

Keep transcript, waveforms, tool calls, and per-turn latency.

Reference suite

Voice test blueprints.

These are designed cases, not yet included in the executable CLI total.

voice-001

First-response latency

latency

Scenario: Caller asks a simple hours question.

Pass condition: First audible response begins within the configured service target.

speech_end_msfirst_audio_msturn_id

soft

voice-002

Critical entity transcription

speech recognition

Scenario: Caller states an order number, phone number, and product model.

Pass condition: Critical entities are transcribed and repeated back exactly.

audio_fixturetranscriptentity_matches

hard

voice-003

Natural interruption

barge in

Scenario: Caller interrupts the agent midway through a long answer.

Pass condition: Playback stops promptly and the next turn incorporates the interruption.

user_started_speaking_msaudio_stop_msnext_turn

soft

voice-004

False interruption resistance

barge in

Scenario: Background noise occurs while the agent speaks.

Pass condition: The response is not cleared by non-speech noise.

vad_eventsaudio_stop_reasonnoise_profile

hard

voice-005

Silence recovery

conversation control

Scenario: Caller goes silent after the agent asks a question.

Pass condition: Agent reprompts once, then offers a graceful close or handoff.

silence_msreprompt_countend_reason

soft

voice-006

Human handoff

handoff

Scenario: Caller explicitly requests a human representative.

Pass condition: Agent confirms the request and uses only an authorized transfer destination.

intenttool_calltransfer_targettool_result

hard

voice-007

Noisy environment

speech recognition

Scenario: Caller asks a policy question with traffic noise in the background.

Pass condition: Agent asks for repetition when confidence is low instead of guessing.

stt_confidencenoise_profileresponse

hard

voice-008

Disclosure compliance

compliance

Scenario: A new call begins in a jurisdiction requiring an AI or recording disclosure.

Pass condition: The configured disclosure is spoken before protected processing occurs.

audio_timelinedisclosure_textprocessing_start_ms

hard

voice-009

Emotional de-escalation

tone

Scenario: An angry caller complains about repeated service failures.

Pass condition: Voice remains calm, empathetic, concise, and does not mirror hostility.

transcriptsentimentprosody_metrics

soft

voice-010

Tool-result grounding

tool use

Scenario: Caller asks for live inventory or appointment availability.

Pass condition: Spoken answer matches the tool result exactly and labels uncertainty.

tool_inputtool_outputspoken_response

hard