WebSocket API - Cartesia Docs

Stream audio between your application and your voice agent via WebSocket. Use this for web apps, mobile apps, or to bridge your own telephony provider.

Quick start

const ws = new WebSocket(
  `wss://api.cartesia.ai/agents/stream/${agentId}`,
  {
    headers: {
      Authorization: `Bearer ${accessToken}`,
      "Cartesia-Version": "2025-04-16",
    },
  }
);

// Initialize the stream
ws.onopen = () => {
  ws.send(JSON.stringify({
    event: "start",
    config: { input_format: "pcm_44100" },
  }));
};

// Handle agent audio
ws.onmessage = (msg) => {
  const data = JSON.parse(msg.data);
  if (data.event === "media_output") {
    playAudio(atob(data.media.payload));
  }
  // Conversation events (turn_started, turn_output_text_delta, turn_ended)
  // also arrive here; see "Build a live transcript" below.
};

// Send user audio
function sendAudio(audioData) {
  ws.send(JSON.stringify({
    event: "media_input",
    stream_id: streamId,
    media: { payload: btoa(audioData) },
  }));
}

Get an access token from the /access-token endpoint. See Authenticating Client Apps for details.

Connection

Connect to the WebSocket endpoint:

wss://api.cartesia.ai/agents/stream/{agent_id}

Headers:

Header	Value
`Authorization`	`Bearer {token}`
`Cartesia-Version`	`2025-04-16`

Protocol Overview

The WebSocket connection uses JSON messages for control events and base64-encoded audio for media. New server event types may be added over time. To keep your integration forward-compatible, ignore any event you do not recognize instead of treating it as an error. A session moves through four phases:

Start. Send a start event with your audio config.
Ready. The server responds with ack once the agent pipeline is ready.
Converse. Stream user audio as media_input and play agent audio from media_output. In parallel, the server pushes conversation events (turn_started, turn_output_text_delta, turn_ended) that describe the dialogue as it happens.
End. Either side can close the connection. The server’s close reason tells you why the call ended (see Connection Management).

After the call, you can fetch its full record from the calls API using the call_id from the ack.

Client events

Starting the Session (`start`)

Initializes the audio stream configuration.

config overrides your agent’s default input audio settings
stream_id is optional. If not provided, the server generates one and returns it in the ack event

This must be the first message sent.

{
  "event": "start",
  "stream_id": "unique_id",
  "config": {
    "input_format": "pcm_44100",
    "output_audio_delivery": "as_available",
    "voice_id": "a0e99841-438c-4a64-b679-ae501e7d6091"
  },
  "agent": {
    "introduction": "Hello, I'm an AI assistant",
    "system_prompt": "### Your Role \n You are a helpful assistant"
  },
  "metadata": {
    "to": "user@example.com",
    "from": "+1234567890"
  }
}

Fields:

stream_id (optional): Stream identifier. If not provided, server generates one
config.input_format: Audio format for client audio input (mulaw_8000, pcm_16000, pcm_24000, pcm_44100)
config.output_audio_delivery (optional): How the server delivers agent audio. speaking_pace (default) paces audio to playback speed; as_available sends it as fast as the model produces it, so a client that buffers locally can play it back on its own clock for lower latency. Not supported when the agent uses background audio.
config.voice_id (optional): Override the agent’s default TTS voice
agent (optional): Allows configuring individual agent calls via API and previewing changes in introduction or prompt without publishing to production
metadata (optional): Custom metadata object. These will be passed through to the agent code, but there are some special fields you can use as well:
- to (optional): Destination identifier for call routing (defaults to agent ID)
- from (optional): Source identifier for the call (defaults to “websocket”)

Streaming User Audio (`media_input`)

Audio data sent from the client to the server. payload audio data should be base64 encoded.

{
  "event": "media_input",
  "stream_id": "unique_id",
  "media": {
    "payload": "base64_encoded_audio_data"
  }
}

Fields:

stream_id: Unique identifier for the Stream from the ack response
media.payload: Base64-encoded audio data in the format specified in the start event

Sending DTMF Tones (`dtmf`)

Sends DTMF (dual-tone multi-frequency) tones.

{
  "event": "dtmf",
  "stream_id": "example_id",
  "dtmf": "1"
}

Fields:

stream_id: Stream identifier
dtmf: DTMF digit (0-9, *, #)

Sending Custom Metadata (`custom`)

Sends custom metadata to the agent.

{
  "event": "custom",
  "stream_id": "example_id",
  "metadata": {
    "user_id": "user123",
    "session_info": "custom_data"
  }
}

Fields:

stream_id: Stream identifier
metadata: Object containing key-value pairs of custom data

Server events

Session Ready (`ack`)

Sent in response to start once the agent pipeline is ready to receive audio. Confirms the stream configuration and returns the server-generated stream_id if one wasn’t provided in the start event.

{
  "event": "ack",
  "stream_id": "example_id",
  "call_id": "agent_call_abc123",
  "config": {
    "input_format": "pcm_44100",
    "output_audio_delivery": "as_available",
    "voice_id": "a0e99841-438c-4a64-b679-ae501e7d6091"
  },
  "agent": {
    "system_prompt": "### Your Role \n You are a helpful assistant",
    "introduction": "Hello, I'm an AI assistant"
  }
}

Fields:

call_id: Identifier of the call record created for this session. Use it with the calls API to fetch the recording and transcript after the call

Receiving Agent Audio (`media_output`)

The agent’s speech. payload is base64-encoded audio in the format set by config.input_format in the start event.

{
  "event": "media_output",
  "stream_id": "example_id",
  "media": {
    "payload": "base64_encoded_audio_data"
  }
}

Conversation Events

Three events describe the conversation itself: turn_started, turn_output_text_delta, and turn_ended. Every turn gets an id from a single counter shared by both roles, starting at 1 and strictly increasing over the call, so id alone identifies a turn. The events for a single turn always arrive in order: turn_started, then any turn_output_text_delta events, then turn_ended. The server pushes them as the conversation happens, independent of audio delivery timing. Use these events to drive transcripts and speaking indicators, and see Build a live transcript for a code example.

When a Turn Begins (`turn_started`)

Sent when the user starts speaking or the agent starts responding.

{
  "event": "turn_started",
  "stream_id": "example_id",
  "turn_started": {
    "id": 3,
    "role": "user",
    "start_timestamp": 12.48
  }
}

Fields:

stream_id: Stream identifier
turn_started.id: Turn index, unique across the call
turn_started.role: user or assistant
turn_started.start_timestamp: Seconds since the start of the call (approximately when the client received ack)

Streaming Agent Text (`turn_output_text_delta`)

Sent for each word the agent speaks, as it is spoken. Use it to render the agent’s side of the transcript, word by word.

{
  "event": "turn_output_text_delta",
  "stream_id": "example_id",
  "turn_output_text_delta": {
    "id": 4,
    "role": "assistant",
    "text": " world"
  }
}

Fields:

stream_id: Stream identifier
turn_output_text_delta.id: Matches the id of the turn the text belongs to
turn_output_text_delta.role: Always assistant
turn_output_text_delta.text: The exact substring to append. Separator spaces are included, so build the running text with buffer[id] += text

These deltas cover the agent’s speech only. The user’s speech is not streamed word by word. It arrives as finalized text when the user’s turn ends, in the turn_ended event.

When a Turn Ends (`turn_ended`)

Sent when a turn finishes. It carries the complete, final text for the turn, which is the version you should store and display once the turn is over.

{
  "event": "turn_ended",
  "stream_id": "example_id",
  "turn_ended": {
    "id": 3,
    "role": "user",
    "text": "I'd like to check my order status.",
    "was_interrupted": false,
    "start_timestamp": 12.48,
    "end_timestamp": 15.02,
    "tool_calls": []
  }
}

Fields:

stream_id: Stream identifier
turn_ended.id: Matches the id of the corresponding turn_started event
turn_ended.role: user or assistant
turn_ended.text: The finalized text of the turn. For a user turn, this is the transcribed speech. For an assistant turn, it is the text the agent actually spoke
turn_ended.was_interrupted: For assistant turns, true when the user interrupted the turn. For user turns, true only when the call ended mid-turn
turn_ended.start_timestamp, turn_ended.end_timestamp: Seconds since the start of the call
turn_ended.tool_calls: Tool calls made during the turn. Each entry has name, arguments, and optionally result and id

If the agent hangs up mid-turn, the connection closes without a final turn_ended. Treat the connection close as ending any open turn.

Handling Interruptions (`clear`)

Indicates the agent wants to clear/interrupt the current audio stream.

{
  "event": "clear",
  "stream_id": "example_id"
}

Call Transfers (`transfer_call`)

Indicates the agent wants to transfer the call to a phone number. The client is responsible for initiating the transfer on its telephony side.

{
  "event": "transfer_call",
  "stream_id": "example_id",
  "transfer": {
    "target_phone_number": "+1234567890"
  }
}

Fields:

stream_id: Stream identifier
transfer.target_phone_number: E.164 phone number to transfer the call to

Build a live transcript

A live transcript shows the conversation on a UI as it happens, with the agent’s words appearing one at a time like live captions.

const turns = new Map();

ws.onmessage = (msg) => {
  const data = JSON.parse(msg.data);
  switch (data.event) {
    case "turn_started": {
      const { id, role } = data.turn_started;
      turns.set(id, { role, text: "", final: false });
      break;
    }
    case "turn_output_text_delta": {
      const { id, text } = data.turn_output_text_delta;
      const turn = turns.get(id);
      if (turn) turn.text += text; // deltas include separators
      break;
    }
    case "turn_ended": {
      const { id, role, text, was_interrupted } = data.turn_ended;
      turns.set(id, { role, text, was_interrupted, final: true });
      break;
    }
  }
  render([...turns.values()]);
};

ws.onclose = () => {
  // Close out any open turn.
  for (const turn of turns.values()) turn.final = true;
  render([...turns.values()]);
};

The two roles differ in how their text arrives, so render them differently. For a user turn, show a speaking indicator when the turn starts and fill in the text once turn_ended arrives. For an agent turn, show each word as its delta arrives so the response builds up at speaking pace.

Connection Management

Inactivity Timeout

The server closes idle connections after 180 seconds. Any client message resets the timer:

Application messages (media_input, dtmf, custom events)
Standard WebSocket ping frames
Any other valid WebSocket message

When the timeout occurs, the connection is closed with:

Code: 1000 (Normal Closure)
Reason: "connection idle timeout"

Ping/Pong Keepalive

To prevent inactivity timeouts during periods of silence, use standard WebSocket ping frames for periodic keepalive:

# Client sends ping to reset inactivity timer
pong_waiter = await websocket.ping()
latency = await pong_waiter

// Requires the Node.js `ws` library — the browser WebSocket API does not expose ping()
setInterval(() => {
  if (websocket.readyState === WebSocket.OPEN) {
    websocket.ping();
  }
}, 60000); // Send ping every 60 seconds

The server automatically responds to ping frames with pong frames and resets the inactivity timer upon receiving any message.

Connection Close

The connection can be closed by either the client or server using WebSocket close frames. Client-initiated close:

await websocket.close(code=1000, reason="session completed")

Server-initiated close: When the agent ends the call, the server closes the connection with:

Code: 1000 (Normal Closure)
Reason: "call ended by agent" or "call ended by agent, reason: {specific_reason}" if additional context is available

Best Practices

Send start first — The connection closes if any other event is sent before start.
Choose the right audio format — Match the format to your source: mulaw_8000 for telephony, pcm_44100 for web clients.
Handle closes cleanly — Always capture close codes and reasons for debugging and recovery.
Keep the connection alive — Send WebSocket ping frames every 60–90 seconds to avoid the 180-second inactivity timeout.
Manage stream IDs — Provide your own stream_id values to improve observability across systems.
Recover from idle timeouts — On 1000 / connection idle timeout, reconnect and resend a start event.

​Quick start

​Connection

​Protocol Overview

​Client events

​Starting the Session (start)

​Streaming User Audio (media_input)

​Sending DTMF Tones (dtmf)

​Sending Custom Metadata (custom)

​Server events

​Session Ready (ack)

​Receiving Agent Audio (media_output)

​Conversation Events

​When a Turn Begins (turn_started)

​Streaming Agent Text (turn_output_text_delta)

​When a Turn Ends (turn_ended)

​Handling Interruptions (clear)

​Call Transfers (transfer_call)

​Build a live transcript

​Connection Management

​Inactivity Timeout

​Ping/Pong Keepalive

​Connection Close

​Best Practices

Quick start

Connection

Protocol Overview

Client events

Starting the Session (`start`)

Streaming User Audio (`media_input`)

Sending DTMF Tones (`dtmf`)

Sending Custom Metadata (`custom`)

Server events

Session Ready (`ack`)

Receiving Agent Audio (`media_output`)

Conversation Events

When a Turn Begins (`turn_started`)

Streaming Agent Text (`turn_output_text_delta`)

When a Turn Ends (`turn_ended`)

Handling Interruptions (`clear`)

Call Transfers (`transfer_call`)

Build a live transcript

Connection Management

Inactivity Timeout

Ping/Pong Keepalive

Connection Close

Best Practices