---
name: Cartesia
description: Use when building voice AI applications, integrating text-to-speech (TTS) or speech-to-text (STT) APIs, creating voice agents with real-time audio, cloning voices, or deploying conversational AI systems with Cartesia's Sonic and Ink models.
metadata:
    mintlify-proj: cartesia
    version: "1.0"
---

# Cartesia Skill

## Product Summary

Cartesia is a voice AI platform providing state-of-the-art text-to-speech (Sonic 3.5) and speech-to-text (Ink 2) models optimized for real-time, conversational AI. Use the REST API (`/tts/bytes`, `/tts/sse`, `/tts/websocket`, `/stt/websocket`, `/stt/transcribe`) for direct model access, or the Line SDK (Python) for building full voice agents with tool calling, multi-agent handoffs, and managed deployment. Authentication uses API keys (server-side) or access tokens (browser/client). Official SDKs: Python (`cartesia` package) and JavaScript/TypeScript (`@cartesia/cartesia-js`). Primary docs: https://docs.cartesia.ai

## When to Use

Reach for this skill when:
- Building real-time voice agents or conversational AI systems
- Integrating text-to-speech into applications (notifications, dubbing, narration, voice agents)
- Transcribing audio or building speech-to-text features with turn detection
- Cloning voices (instant or professional) for custom voice experiences
- Streaming text from LLMs to audio in real time (WebSocket continuations)
- Deploying voice agents to production with the Line SDK and CLI
- Handling audio interruptions, turn-taking, or multi-turn conversations
- Calling external APIs or databases from within a voice agent

## Quick Reference

### Authentication

| Environment | Method | Header | Notes |
| --- | --- | --- | --- |
| Server/backend | API key | `Authorization: Bearer <api_key>` or `X-Api-Key: <api_key>` | Get from play.cartesia.ai/keys |
| Browser/client | Access token | `Authorization: Bearer <token>` (HTTP) or `?access_token=<token>` (WebSocket) | Generate on server, short-lived |
| Admin operations | Admin API key | `Authorization: Bearer <admin_key>` | Get from play.cartesia.ai/keys/admin (org admins only) |

### TTS Endpoints

| Endpoint | Use Case | Features | When to Choose |
| --- | --- | --- | --- |
| `POST /tts/bytes` | Batch, caching, one-shot | Streaming HTTP body, efficient | Full transcript known upfront, no timestamps needed |
| `POST /tts/sse` | HTTP with timestamps | Server-Sent Events, timestamps | Need timestamps without WebSocket |
| `WebSocket /tts/websocket` | Real-time, streaming input | Continuations, multiplexing, lowest latency | LLM streaming, long sessions, multiple utterances |

### STT Endpoints

| Endpoint | Use Case | Features |
| --- | --- | --- |
| `POST /stt/transcribe` | Batch transcription | Upload audio file, returns full transcript |
| `WebSocket /stt/websocket` | Real-time streaming | Turn detection (Ink 2), streaming audio input |

### Models

| Model | Type | Key Feature | Status |
| --- | --- | --- | --- |
| `sonic-3.5` | TTS | 42 languages, sub-90ms latency, expressive | Stable (use this) |
| `sonic-latest` | TTS | Beta features, non-snapshotted | Testing only |
| `ink-2` | STT | Native turn detection, lowest WER | Stable (use this) |

### Line SDK Core Components

| Component | Purpose |
| --- | --- |
| `LlmAgent` | Built-in agent wrapping 100+ LLM providers via LiteLLM |
| `Agent` | Custom agent with `process()` method for full control |
| `Tools` | Functions agents can call (database lookups, handoffs, web search) |
| `VoiceAgentApp` | HTTP server connecting agent to Cartesia audio infrastructure |

### CLI Commands

```bash
cartesia auth login                    # Authenticate with API key
cartesia create my-agent               # Clone example agent
cartesia init                          # Link directory to agent
cartesia deploy                        # Deploy to Cartesia cloud
cartesia chat 8000                     # Test agent locally
cartesia agents ls                     # List all agents
cartesia deployments ls                # List deployments
cartesia phone-numbers ls              # List phone numbers
cartesia env set KEY=VALUE             # Set environment variables
```

## Decision Guidance

### When to Use Each TTS Endpoint

| Scenario | Endpoint | Reason |
| --- | --- | --- |
| LLM streaming text token-by-token | WebSocket | Continuations maintain prosody; lowest latency per turn |
| Full transcript, no timestamps | Bytes | Simplest; raw audio on wire is efficient |
| Full transcript, need timestamps | SSE | Timestamps in event payload; HTTP-only |
| Many short utterances, keep connection open | WebSocket | Amortizes TCP/TLS cost across turns |

### When to Use Each STT Endpoint

| Scenario | Endpoint | Reason |
| --- | --- | --- |
| Upload pre-recorded audio file | Transcribe | Simple batch operation |
| Real-time voice agent input | WebSocket | Turn detection built-in; streaming |
| Need turn detection (when user stops talking) | WebSocket with Ink 2 | Emits turn.start, turn.end, turn.eager_end events |

### Voice Cloning: Instant vs. Pro

| Feature | Instant Clone | Pro Clone |
| --- | --- | --- |
| Audio required | 3–10 seconds | 30 minutes |
| Cost to create | Free | 1M credits |
| Cost per character | 1 credit | 1.5 credits |
| Best for | Quick prototypes, demos | Production, exact voice replica |

### Agent Deployment: Managed vs. Self-Hosted

| Option | Setup | Latency | Scaling |
| --- | --- | --- | --- |
| Managed (CLI `cartesia deploy`) | Seconds; auto-scaling | Low; Cartesia infrastructure | Automatic |
| Self-hosted | Point CLI to your URL | Depends on your infra | Your responsibility |

## Workflow

### 1. Build and Deploy a Voice Agent (Line SDK)

1. **Authenticate**: Run `cartesia auth login` with your API key from play.cartesia.ai/keys
2. **Create project**: Run `cartesia create my-agent` and choose an example (basic_chat, form_filler, etc.)
3. **Review agent code**: Open `main.py`; it defines an `LlmAgent` with system prompt, LLM model, and tools
4. **Add tools** (optional): Define custom functions with `@loopback_tool` decorator to call APIs or databases
5. **Test locally**: Run `PORT=8000 uv run python main.py` in one terminal, then `cartesia chat 8000` in another
6. **Deploy**: Run `cartesia deploy` to push to Cartesia cloud
7. **Monitor**: Check call logs and metrics in play.cartesia.ai or via CLI (`cartesia deployments ls`)

### 2. Stream Text to Speech in Real Time (WebSocket)

1. **Open WebSocket**: Connect to `wss://api.cartesia.ai/tts/websocket` with API key in header or access token in query param
2. **Create context**: Send JSON with `model_id`, `voice`, `language`, `context_id`, and initial `transcript`
3. **Stream text**: Send additional messages with same `context_id`, `continue: true` until final chunk
4. **Finalize**: Send final chunk with `continue: false` or call `no_more_inputs()` in SDK
5. **Receive audio**: Listen for `chunk` events with audio data; handle `error` events
6. **Close**: Close WebSocket when done

### 3. Transcribe Audio in Real Time (STT WebSocket)

1. **Open WebSocket**: Connect to `wss://api.cartesia.ai/stt/websocket` with authentication
2. **Send audio**: Stream audio chunks as binary messages
3. **Receive transcripts**: Listen for `transcript` events with partial and final results
4. **Monitor turns**: Watch for `turn.start`, `turn.eager_end`, `turn.end` events (Ink 2 turn detection)
5. **Finalize**: Send `finalize` command to end transcription and get final result

### 4. Clone a Voice

1. **Prepare audio**: Get 3–10 seconds of clean audio (instant) or 30 minutes (pro)
2. **Call API or SDK**: Use `client.voices.clone(clip=file, language="en", name="My Voice")`
3. **Get voice ID**: Response includes `id` field; use this in TTS requests
4. **Use in TTS**: Set `voice: { mode: "id", id: "<voice_id>" }` in TTS payload

### 5. Add Custom Pronunciations

1. **Create pronunciation dict**: Build JSON array with `text` and `pronunciation` (IPA or "sounds-like")
2. **Create via API**: POST to `/pronunciation-dicts` with dict entries
3. **Use in TTS**: Include `pronunciation_dict_id` in TTS request payload
4. **Test**: Generate speech and verify pronunciation

## Common Gotchas

- **Empty transcript**: TTS will reject empty `transcript` field; always provide text
- **Missing spaces in continuations**: When streaming text chunks, include leading spaces (e.g., `" How are you?"` not `"How are you?"`) to avoid `"world!How"` artifacts
- **Incomplete sentences without punctuation**: Model buffers text waiting for sentence-ending punctuation (`.`, `?`, `!`); missing punctuation causes latency or audio artifacts. Always end final chunks with punctuation
- **Reusing context_id**: Don't reuse old `context_id` values for unrelated utterances; create new contexts for new conversations
- **API key in browser code**: Never ship API keys in client-side code; always use access tokens generated server-side
- **WebSocket headers in browser**: Browsers can't set custom headers on WebSocket handshakes; pass access token as `?access_token=<token>` query param instead
- **Concurrency limits**: TTS concurrency is measured by unique `context_id` values; HTTP endpoints (bytes, SSE) each count as one context. Check your limit at play.cartesia.ai
- **Model ID pinning**: Use `sonic-3.5` (stable) or `sonic-3.5-YYYY-MM-DD` (pinned) in production; avoid `sonic-latest` (beta, can change without notice)
- **Turn detection only in Ink 2**: Older STT models don't have turn detection; use `ink-2` for `turn.start`, `turn.end` events
- **Admin API keys for usage**: Credit usage and agent usage endpoints require admin API keys (`sk_car_admin_...`), not standard API keys

## Verification Checklist

Before submitting voice AI work:

- [ ] Authentication: API key or access token is valid and has correct grants (tts, stt, agent)
- [ ] Model IDs: Using `sonic-3.5` (TTS) and `ink-2` (STT), not beta versions
- [ ] Transcript: Non-empty, properly formatted; final chunks end with punctuation
- [ ] Voice ID: Cloned or built-in voice ID is valid and matches language
- [ ] Continuations: If streaming text, using `context_id` consistently and `continue: false` on final chunk
- [ ] Error handling: Catching `BadRequestError`, `RateLimitError`, `AuthenticationError` from SDK
- [ ] Concurrency: Not exceeding plan limits; checked at play.cartesia.ai
- [ ] Agent deployment: Tested locally with `cartesia chat`, then deployed with `cartesia deploy`
- [ ] Tools: Custom tools return strings (not objects); handle errors gracefully
- [ ] Phone numbers: If using telephony, phone numbers are provisioned and assigned to agent

## Resources

- **Full navigation**: https://docs.cartesia.ai/llms.txt — comprehensive page-by-page listing for agent reference
- **API Reference**: https://docs.cartesia.ai/api-reference/tts/websocket — WebSocket TTS endpoint with full request/response schema
- **Line SDK Guide**: https://docs.cartesia.ai/line/sdk/overview — Python framework for voice agents with examples
- **Playground**: https://play.cartesia.ai — test voices, agents, and API calls without code

---

> For additional documentation and navigation, see: https://docs.cartesia.ai/llms.txt