Skip to main content
Use one bidirectional WebSocket to send user audio, receive agent audio, and handle conversation or client-tool events.

Quick start

Create an access token with the /access-token endpoint. See Authenticate client applications for token handling.

Connect

Server applications can authenticate with the X-API-Key header. Browser and mobile applications should send an access_token query parameter. Never expose an API key in a client application. The connection captures the agent’s current version when it opens. A configuration change does not affect a call already in progress. The session_ready event reports the resolved agent_version_id and the call’s call_id.

Start the session

Send session_create as the first event, within 10 seconds of connecting:
Supported input formats are mulaw_8000, pcm_16000, pcm_24000, and pcm_44100. Agent audio uses the same format. output_delivery defaults to speaking_pace. Use as_available when your application manages its own playback buffer. as_available cannot be used when the agent has background audio configured. Wait for session_ready before sending audio:

Stream audio

Send user audio as base64-encoded audio_input events. Chunks of 20 to 100 milliseconds usually provide good latency.
The server returns audio_output events in the configured format:
When the user starts speaking, the server sends audio_output_clear. Discard buffered agent audio and stop playback immediately. The event can arrive while no agent audio is playing, in which case there is nothing to clear.

Conversation events

Use turn_started, turn_output_text_delta, and turn_ended to show speaking state or build a transcript. Assistant text arrives incrementally in turn_output_text_delta; turn_ended.text contains the final text for either role. These events are informational. Audio handling should not depend on them.

Client tools

When the agent invokes a client tool, the server sends client_tool_call:
If expects_response is true, answer with the same tool_call_id:
Late, duplicate, and mismatched results are ignored. Results may be up to 4 KiB.

Errors and connection limits

The server sends an error event when it rejects an event. A recoverable error has fatal: false; a fatal error is followed by a connection close.
  • A JSON message may be up to 32 KiB. Larger messages close with code 1009.
  • The server closes after 120 seconds without a valid client event. Streaming audio continuously, including silence, keeps the connection active. WebSocket ping frames do not reset this application-level timer.
  • Close with code 1000 for a normal client shutdown. The server uses 1008 for protocol errors and 1011 for internal failures.
See the WebSocket API reference for every event and field.