Stream Speech (WebSocket)

1
npx wscat -c "wss://api.cartesia.ai/tts/websocket?api_key=<YOUR_API_KEY>&cartesia_version=2024-06-10"

GET /tts/websocket?api_key=<YOUR_API_KEY>&cartesia_version=<API_VERSION>

Initiate a bidirectional WebSocket connection. The connection supports multiplexing, so you can send multiple requests and receive the corresponding responses in parallel. The connection times out 5 minutes after the last message you send.

WebSocket Request

Send a JSON-encoded message on the WebSocket. The schema of said message should be identical to the Server-Sent Events request body, except that you must additionally specify a context_id field containing a unique identifier for the request. (You can use a UUIDv4 or a human ID.)

1
{
2
  "context_id": "happy-monkeys-fly",
3
  "model_id": "sonic-english",
4
  "transcript": "Hello, world! I'\''m generating audio on Cartesia.",
5
  "duration": 180,
6
  "voice": {
7
    "mode": "id",
8
    "id": "a0e99841-438c-4a64-b679-ae501e7d6091",
9
    "__experimental_controls": {
10
      "speed": "normal",
11
      "emotion": ["positivity:highest", "curiosity"]
12
    }
13
  },
14
  "output_format": {
15
    "container": "raw",
16
    "encoding": "pcm_s16le",
17
    "sample_rate": 8000
18
  },
19
  "language": "en",
20
  "add_timestamps": false
21
}

You may also cancel outgoing requests through the websocket. This will only halt requests that have not begun generating a response yet.

1
{
2
  "context_id": "happy-monkeys-fly",
3
  "cancel": true,
4
}

WebSocket Responses

After you send a message body on the WebSocket, the API will respond with a series of JSON chunks with the same schema as the data in Server-Sent Events responses.

1
{
2
  "status_code": 206,
3
  "done": false,
4
  "type": "chunk",
5
  "data": "aSDinaTvuI8gbWludGxpZnk=",
6
  "step_time": 123,
7
  "context_id": "happy-monkeys-fly"
8
}

If add_timestamps is set to true, we will also return messages of the following form in addition to the audio chunks and done message:

1
{
2
  "status_code": 206,
3
  "done": false,
4
  "context_id": "happy-monkeys-fly",
5
  "type": "timestamps",
6
  "word_timestamps": {
7
    "words": ["Hello"],
8
    "start": [0.0],
9
    "end": [1.0]
10
  }
11
}

Input Streaming with Contexts

In many real time use cases, you don’t have your transcripts available upfront—like when you’re generating them using an LLM. For these cases, Sonic supports input streaming.

The context IDs you pass to the Cartesia API identify speech contexts. Contexts maintain prosody between their inputs—so you can send a transcript in multiple parts and receive seamless speech in return.

To stream in inputs on a context, just pass a continue flag (set to true) for every input that you expect will be followed by more inputs. (By default, this flag is set to false.)

To finish a context, just set continue to false. If you do not know the last transcript in advance, you can send an input with an empty transcript and continue set to false.

Input Format

1. Inputs on the same context must keep all fields except transcript, continue, and duration the same.
2. Transcripts are concatenated verbatim, so make sure they form a valid transcript when joined together. Make sure to include any spaces between words or punctuations as necessary. For example, in languages with spaces, you should include a space at the end of the preceding transcript, e.g. transcript 1 is Thanks for coming, and transcript 2 is it was great to see you.
3. It’s important to buffer the first request transcript to at least 3 or 4 words for best performance.

Example

Let’s say you’re trying to generate speech for “Hello, Sonic! I’m streaming inputs.” You should stream in the following inputs (repeated fields omitted for brevity). Note: all other fields (e.g. model_id, language) are required and should be passed unchanged between requests with input streaming.

1
{"transcript": "Hello, Sonic!", "continue": true, "context_id": "happy-monkeys-fly"}
2
{"transcript": " I'm streaming ", "continue": true, "context_id": "happy-monkeys-fly"}
3
{"transcript": "inputs.", "continue": false, "context_id": "happy-monkeys-fly"}

If you don’t know the last transcript in advance, you can send an input with an empty transcript and continue set to false:

1
{"transcript": "Hello, Sonic!", "continue": true, "context_id": "happy-monkeys-fly"}
2
{"transcript": " I'm streaming ", "continue": true, "context_id": "happy-monkeys-fly"}
3
{"transcript": "inputs.", "continue": true, "context_id": "happy-monkeys-fly"}
4
{"transcript": "", "continue": false, "context_id": "happy-monkeys-fly"}

Output

You will only receive done: true after outputs for the entire context have been returned.

Outputs for a given context will always be in order of the inputs you streamed in. (That is, if you send input A and then input B on a context, you will first receive the chunks corresponding to input A, and then the chunks corresponding to input B.)

Cancelling Requests

You may also cancel outgoing requests through the websocket.

To cancel a request, send a JSON message with the following structure:

1
{
2
  "context_id": "happy-monkeys-fly",
3
  "cancel": true
4
}

When you send a cancel request:

1. It will only halt requests that have not begun generating a response yet.
2. Any currently generating request will continue sending responses until completion.