diff --git a/fern/docs.yml b/fern/docs.yml index d4ce25581..50265c94f 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -123,6 +123,17 @@ navigation: icon: fa-light fa-terminal path: cli/overview.mdx + - section: GPT-Live + contents: + - page: Overview + path: gpt-live/overview.mdx + - page: Design your assistant + path: gpt-live/design.mdx + - page: Configuration + path: gpt-live/configuration.mdx + - page: Limitations and FAQs + path: gpt-live/limitations.mdx + - section: Assistants collapsed: open-by-default contents: @@ -1061,6 +1072,24 @@ navigation: - tab: changelog redirects: + - source: /providers/gpt-live + destination: /gpt-live/overview + - source: /gpt-live/quickstart + destination: /gpt-live/configuration#create-an-assistant + - source: /gpt-live/voices + destination: /gpt-live/configuration#voices + - source: /gpt-live/prompts + destination: /gpt-live/design#speaker-and-reasoner-roles + - source: /gpt-live/conversation-design + destination: /gpt-live/design + - source: /gpt-live/operations + destination: /gpt-live/configuration#review-and-monitor-calls + - source: /gpt-live/tools + destination: /gpt-live/configuration#connect-your-tools + - source: /gpt-live/testing + destination: /gpt-live/design#testing-conversations + - source: /gpt-live/faqs + destination: /gpt-live/limitations - source: /test/test-suites destination: /test/voice-testing - source: /test/chat-testing diff --git a/fern/gpt-live/configuration.mdx b/fern/gpt-live/configuration.mdx new file mode 100644 index 000000000..95f11b4b2 --- /dev/null +++ b/fern/gpt-live/configuration.mdx @@ -0,0 +1,372 @@ +--- +title: Configure GPT-Live +subtitle: Create an assistant, choose its voice, and connect your tools +description: Configure a GPT-Live assistant in Vapi using the dashboard or API, with speaker and reasoner settings, voice samples, tools, call connections, and analysis. +slug: gpt-live/configuration +--- + +Create an assistant that checks appointment availability and reads back the results. You'll configure the speaker and reasoner, connect a lookup tool, and make a test call. + +For help deciding what to put in each prompt, read [Design your assistant](/gpt-live/design). For feature restrictions, use [Limitations and FAQs](/gpt-live/limitations). + +**Jump to:** [Create an assistant](#create-an-assistant) · [Model settings](#model-settings) · [Personality](#personality-and-language) · [Voices](#voices) · [Tools](#connect-your-tools) · [Calls](#connect-a-call) · [Analysis](#review-and-monitor-calls) + +## Before you start + +- Enable GPT-Live for your Vapi organization. +- Have a browser and microphone ready for a test call. +- For API requests, use a Vapi private API key on your server or local machine. +- If you bring an OpenAI key, it needs access to GPT-Live and the reasoner you select. +- Prepare an HTTPS availability endpoint using the [tool response contract](#connect-your-tools). + +The example uses a placeholder endpoint. Replace it with your handler before testing a lookup. It returns availability only. Booking requires a separate tool. + +## Create an assistant + +Choose the dashboard or API. Both paths create an assistant with an availability lookup and an end-call tool. + + + + + +Open [Assistants](https://dashboard.vapi.ai/assistants), select **Create Assistant**, and choose **GPT-Live**. Choose Marin as the voice. + + +In **Speaker**, enter: + +```text title="Speaker instructions" +Help callers check appointment availability. Speak warmly and briefly. +Ask for an explicit date and time zone, then delegate the lookup. +If the caller changes the request, use the updated details. +Explain the returned slots. Booking isn't connected yet. +Delegate requests to end the call. +``` + +In **Reasoner**, enter: + +```text title="Reasoner instructions" +Use lookupAvailability after the caller provides an explicit date +and time zone. Ask for clarification if either is missing or ambiguous. +Return only slots from the current lookup. Explain failures without +inventing results. No booking tool is available. +Call endCall when the caller asks to finish. +``` + + +Create a function tool named `lookupAvailability` with the description “Check available appointment slots. Does not reserve or book a slot.” Add these required arguments: + +| Argument | Type | Description | +| --- | --- | --- | +| `date` | String | Requested calendar date in YYYY-MM-DD format | +| `timeZone` | String | IANA time zone, for example `America/New_York` | + +Set the tool's Server URL to your HTTPS handler. + +Attach the function tool and an `endCall` tool to the assistant. You can also create a saved tool through [Create Tool](/api-reference/tools/create) and select it in the dashboard. + + +Set the first message to “Hello. I can check appointment availability. What date and time zone should I check?” and configure the assistant to speak first. + +Select **Publish**, review the changes, and complete the publication flow. Keep the assistant ID for API calls. + + + + + + +Save the following as `availability-assistant.json`: + +```json title="availability-assistant.json" +{ + "name": "Appointment availability assistant", + "model": { + "provider": "openai", + "model": "gpt-live-1", + "speaker": { + "instructions": "You help callers check appointment availability. Speak briefly. Collect an explicit calendar date and time zone, then delegate availability checks to the reasoner. If the caller corrects the date, delegate the corrected request and do not present old results as current. Report only confirmed lookup results. Availability does not reserve a slot. Booking is not connected. Delegate requests to end the call." + }, + "reasoner": { + "model": "gpt-5.6-terra", + "reasoningEffort": "low", + "instructions": "Use lookupAvailability only after the caller supplies an explicit calendar date and time zone. Ask for clarification if either is missing or ambiguous. Report only slots returned for the current request. Ignore superseded lookup results. A successful lookup does not reserve or book anything. Explain lookup failures without inventing slots. No booking tool is available. Call endCall when the caller asks to finish." + }, + "tools": [ + { + "type": "function", + "function": { + "name": "lookupAvailability", + "description": "Check available appointment slots. Does not reserve or book a slot.", + "parameters": { + "type": "object", + "properties": { + "date": { + "type": "string", + "description": "Requested calendar date in YYYY-MM-DD format." + }, + "timeZone": { + "type": "string", + "description": "IANA time zone, such as America/New_York." + } + }, + "required": [ + "date", + "timeZone" + ] + } + }, + "server": { + "url": "https://api.example.com/vapi/tools" + } + }, + { + "type": "endCall" + } + ] + }, + "voice": { + "provider": "openai", + "voiceId": "marin" + }, + "firstMessageMode": "assistant-speaks-first", + "firstMessage": "Hello. I can check appointment availability. What date and time zone should I check?", + "maxDurationSeconds": 600 +} +``` + + + +Set `VAPI_PRIVATE_API_KEY` to your Vapi private API key, then create the assistant: + +```bash +curl --fail-with-body https://api.vapi.ai/assistant \ + -H "Authorization: Bearer $VAPI_PRIVATE_API_KEY" \ + -H "Content-Type: application/json" \ + --data-binary @availability-assistant.json +``` + +Keep the returned `id`. To change an existing assistant, use [Update Assistant](/api-reference/assistants/update), preserving its other tools and settings. + + + + + +### Make a test call + +Open the saved assistant in the dashboard and select **Talk**. Allow microphone access, then ask for availability on an explicit date in a specific time zone. + +Check that your handler receives the right values and the assistant reads back the returned slots. Ask to book a time. It should explain that booking isn't connected. Finish with “Please end the call” and confirm that the call ends. + +## Model settings + +These fields belong to the assistant object. See [Create Assistant](/api-reference/assistants/create) for the full schema. + +| Field | Values or behavior | +| --- | --- | +| `model.provider` | `openai` | +| `model.model` | `gpt-live-1` | +| `model.speaker.instructions` | Speaker prompt. See [prompt behavior](/gpt-live/configuration#prompt-defaults-and-overrides). | +| `model.speaker.personalityPacks` | Array of `eager-listener`, `idle-hummer`, `bouncy`, and `unhurried`. | +| `model.reasoner.provider` | `openai`. Defaults to OpenAI when omitted. | +| `model.reasoner.model` | `gpt-5.6-sol`, `gpt-5.6-terra`, or `gpt-5.6-luna`. Defaults to `gpt-5.6-terra`. | +| `model.reasoner.reasoningEffort` | `none`, `low`, `medium`, `high`, `xhigh`, or `max`. Defaults to `low`. | +| `model.reasoner.instructions` | Custom reasoner prompt. Omission uses Vapi's default. | +| `model.tools` | Inline supported tool definitions. | +| `model.toolIds` | IDs of saved tools to resolve for the assistant. | +| `voice.provider` | `openai` | +| `voice.voiceId` | One of the [22 supported voices](/gpt-live/configuration#voices). | + +## Prompt defaults and overrides + +Set `model.speaker.instructions` for the spoken conversation and `model.reasoner.instructions` for task procedures. + +- Explicit speaker instructions take precedence over `model.systemPrompt` and system messages in `model.messages`. If you omit the speaker instructions, Vapi uses that legacy configuration. +- Omitting reasoner instructions uses Vapi's default reasoner prompt. Custom instructions replace that prompt entirely. +- An explicit empty string stays empty. Updating one prompt doesn't update the other. + +Personality packs append guidance to the speaker prompt. Set `model.speaker.personalityPacks` to an array of the IDs above and test one pack at a time. + +## Personality and language + +Personality packs append speaking-style guidance to your speaker prompt. Add this inside the assistant's `model` object: + +```json +{ + "speaker": { + "instructions": "Speak warmly and briefly. Slow down for appointment dates and times. Follow requests to speed up or explain more. Delegate requests that need tools.", + "personalityPacks": ["unhurried"] + } +} +``` + +Available IDs are `eager-listener`, `idle-hummer`, `bouncy`, and `unhurried`. See [Speaking style](/gpt-live/design#speaking-style) for what each pack encourages and how to combine personality with pacing. These are prompt instructions, not fixed speed controls. + +Write the speaker prompt and `firstMessage` in the language you want the assistant to use. For Bossa or Tempo, you can include: + +```text +Fale em português do Brasil, com respostas curtas e claras. +Faça uma pergunta por vez. Só confirme um agendamento +quando a ferramenta confirmar que ele foi realizado. +``` + +Keep tool and delegation instructions in the prompt too. Try names, dates, and product terms in real calls, and add pronunciation guidance where needed. + +## Voices + +Choose any of these 22 voices with `voice.provider: "openai"` and `voice.voiceId`. Save the assistant and start a new call to change voices. In-call voice changes aren't supported. + +### Voices with regional tags + +Use the tags to narrow your choices, then listen to the samples. Bossa and Tempo previews are in Brazilian Portuguese. The other previews are in English. Test your preferred voice with your own prompts to hear how it handles the accent and vocabulary you need. + +| Voice | API ID | Language | Regional influence | Presentation | Source | Preview | +| --- | --- | --- | --- | --- | --- | --- | +| Quartz | `quartz` | English | Australian | Feminine | Generated | [Listen to Quartz](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/quartz.mp3) | +| Ripple | `ripple` | English | Australian | Masculine | Natural | [Listen to Ripple](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/ripple.mp3) | +| Vesper | `vesper` | English | British | Masculine | Natural | [Listen to Vesper](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/vesper.mp3) | +| Willow | `willow` | English | Irish | Feminine | Natural | [Listen to Willow](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/willow.mp3) | +| Stone | `stone` | English | Irish | Masculine | Natural | [Listen to Stone](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/stone.mp3) | +| Gleam | `gleam` | English | North American | Feminine | Natural | [Listen to Gleam](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/gleam.mp3) | +| Meridian | `meridian` | English | North American | Masculine | Natural | [Listen to Meridian](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/meridian.mp3) | +| Bossa | `bossa` | Portuguese | Brazilian | Feminine | Natural | [Listen to Bossa](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/bossa.mp3) | +| Tempo | `tempo` | Portuguese | Brazilian | Masculine | Natural | [Listen to Tempo](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/tempo.mp3) | +| Beacon | `beacon` | English | Filipino | Masculine | Generated | [Listen to Beacon](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/beacon.mp3) | +| Delta | `delta` | English | Southern U.S. | Feminine | Generated | [Listen to Delta](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/delta.mp3) | +| Cinder | `cinder` | English | Southern U.S. | Masculine | Generated | [Listen to Cinder](https://voices-storage.vapi.ai/gpt-live/gpt-live-1/cinder.mp3) | + +### More voices + +You can also choose any of these voices. Select a preview to listen. + +| Voice | API ID | Preview | +| --- | --- | --- | +| Alloy | `alloy` | [Listen to Alloy](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/alloy.mp3) | +| Ash | `ash` | [Listen to Ash](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/ash.mp3) | +| Ballad | `ballad` | [Listen to Ballad](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/ballad.mp3) | +| Cedar | `cedar` | [Listen to Cedar](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/cedar.mp3) | +| Coral | `coral` | [Listen to Coral](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/coral.mp3) | +| Echo | `echo` | [Listen to Echo](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/echo.mp3) | +| Marin | `marin` | [Listen to Marin](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/marin.mp3) | +| Sage | `sage` | [Listen to Sage](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/sage.mp3) | +| Shimmer | `shimmer` | [Listen to Shimmer](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/shimmer.mp3) | +| Verse | `verse` | [Listen to Verse](https://voices-storage.vapi.ai/gpt-live/marble-alpha-v6/verse.mp3) | + +## Greetings and duration + +Use a text greeting if the assistant should begin the conversation. + +| Field or feature | Behavior | +| --- | --- | +| `firstMessageMode: "assistant-speaks-first"` | Speaks the configured text greeting. | +| `firstMessageMode: "assistant-waits-for-user"` | Waits for the caller. | +| `firstMessage` | Text greeting. Without usable text, the assistant waits. | +| `maxDurationSeconds` | Maximum call duration in seconds. | +| Audio URL greeting | Unsupported. Not played as a greeting. | +| Generated-message first-message mode | Unsupported. The assistant waits for the caller. | + +Generated speech may vary from the supplied greeting. A text greeting is not a guarantee of exact prerecorded playback. + +## Connect your tools + +Handle Vapi's [`tool-calls` server message](/server-url/events) at your HTTPS endpoint. Read `message.toolCallList`, validate each function's arguments, and query your availability service. + +Return a result for each request using its actual `toolCallId`: + +```json +{ + "results": [ + { + "name": "lookupAvailability", + "toolCallId": "TOOL_CALL_ID_FROM_REQUEST", + "result": "There are openings at 2 PM and 4 PM on the requested date in the requested time zone. No appointment has been booked." + } + ] +} +``` + +Replace the sample result with your service's data. If the lookup fails, return the failure so the assistant can explain it. See [Server authentication](/server-url/server-authentication) to protect the endpoint. + +Supported tool types are `function`, `apiRequest`, `endCall`, `dtmf`, and `transferCall`. Give tools unique resolved function names. Save reusable tools and attach them through `model.toolIds`, or include definitions in `model.tools`. + +Use function or API request tools to connect external data sources. For phone actions, check [transfer and keypad support](/gpt-live/limitations#transfers-and-keypad-input). + +### Slow and asynchronous requests + +A synchronous function waits for your handler's result. Set `async: true` on a function tool to let the conversation continue while that request completes. Return the final result through the original request, using the matching `toolCallId`. + +If your handler returns “queued” before the work finishes, add a status lookup tool. Tell the reasoner when to check it. A later callback doesn't automatically update the conversation. + +Tools can run concurrently. Keep dependent actions in order: check availability, confirm the caller's choice, then book. Your service should validate those requirements too. + +## Connect a call + +Start with dashboard **Talk**, then connect your intended call channel. GPT-Live supports browser WebRTC, native Twilio, Vapi SIP, and raw WebSocket audio. See [connection limits](/gpt-live/limitations#calls-and-voice) before changing an existing setup. + +### Phone calls + +For inbound Twilio calls, [import a Twilio phone number](/phone-numbers/import-twilio) and assign your assistant to it. For SIP, follow the [SIP guide](/advanced/sip). + +For an outbound Twilio test, send this body to [Create Call](/api-reference/calls/create): + +```json +{ + "assistantId": "YOUR_ASSISTANT_ID", + "phoneNumberId": "YOUR_TWILIO_PHONE_NUMBER_ID", + "customer": {"number": "+14155550100"} +} +``` + +Replace the customer phone number with one you control. `phoneNumberId` is the Vapi ID of the imported Twilio phone number. + +### WebSocket configuration + +Send this body to `POST https://api.vapi.ai/call` with your Vapi private API key: + +```json +{ + "assistantId": "YOUR_ASSISTANT_ID", + "transport": { + "provider": "vapi.websocket", + "audioFormat": { + "format": "pcm_s16le", + "container": "raw", + "sampleRate": 24000 + } + } +} +``` + +Connect to `transport.websocketCallUrl` in the response. Send and receive binary audio frames containing signed 16-bit little-endian samples at 24 kHz. Convert your microphone audio to that format before sending it. + +Set `24000` explicitly. The default WebSocket rate is too low for GPT-Live. Browser WebRTC and phone connections negotiate their own audio formats. + +Close the WebSocket to end the call, or have the assistant invoke `endCall`. Incoming JSON call-control commands are not supported by this GPT-Live connection. See [WebSocket transport](/calls/websocket-transport) for the general transport, applying the restrictions above. + +## Review and monitor calls + +Completed GPT-Live calls use Vapi's shared post-call processing. Configure the features you need, then inspect the results after a test call finishes. + +| Feature | Setup and results | +| --- | --- | +| [Structured outputs](/assistants/structured-outputs-quickstart) | Create output definitions, attach their IDs through `artifactPlan.structuredOutputIds`, and read `call.artifact.structuredOutputs`. Keep existing IDs and artifact settings when updating. | +| [Call analysis](/assistants/call-analysis) | Existing `analysisPlan` configurations work. Prefer structured outputs for new setups. | +| [Scorecards](/observability/scorecard-quickstart) | Grade boolean or numeric outputs and attach the scorecard to your assistant. Read `call.artifact.scorecards`. | +| [Boards](/observability/boards-quickstart) | Track available call metrics and outcome trends for your assistant. Classic transcriber and text-to-speech timing fields don't describe GPT-Live. | +| [Monitoring](/observability/monitoring-quickstart) | Configure supported conditions, thresholds, evaluation windows, and notification destinations. Requires retained data. Unavailable with Zero Data Retention. | +| [End-of-call reports](/server-url/events) | Configure a Server URL and the `end-of-call-report` server message. [Authenticate your endpoint](/server-url/server-authentication) and handle repeated delivery by call ID. | + +Keep transcript collection enabled for transcript-based checks. Audio-based checks require a recording. See [recording and data restrictions](/gpt-live/limitations#recordings-and-post-call-data) before relying on those artifacts. + +Retrieve a completed call with your private API key: + +```bash +curl --fail-with-body "https://api.vapi.ai/call/$CALL_ID" \ + -H "Authorization: Bearer $VAPI_PRIVATE_API_KEY" +``` + +Review the transcript and recording alongside tool results. A missing analysis result isn't a pass. Check attached output IDs, processing, and available artifacts. Creating a scorecard doesn't automatically configure a monitor. + +When testing a migration, keep the previous assistant available so you can route new calls back. End test calls through the call interface, `endCall`, or by closing your WebSocket. Check any external work still running. Ending the call doesn't undo it. + +## Cost + +Review Vapi call costs for your configuration. Reasoning, tools, and telephony can contribute to the total. OpenAI's [model pricing](https://developers.openai.com/api/docs/models/gpt-live-1) describes provider charges rather than the complete cost of a Vapi call. diff --git a/fern/gpt-live/design.mdx b/fern/gpt-live/design.mdx new file mode 100644 index 000000000..52779ce8f --- /dev/null +++ b/fern/gpt-live/design.mdx @@ -0,0 +1,190 @@ +--- +title: Design your GPT-Live assistant +subtitle: Plan the conversation and choose how your assistant speaks +description: Design GPT-Live conversations with speaker and reasoner roles, outcome-based delegation, useful overlap during tools, flexible pacing, and personality packs. +slug: gpt-live/design +--- + +With GPT-Live, the conversation can continue while work happens in the background. A caller can add a preference during a lookup, ask a question halfway through scheduling, or ask the assistant to slow down. + +The design task is to make those moments feel connected. Decide what the assistant is trying to accomplish, how it should guide the caller, and which decisions need help from your services. + +This guide uses scheduling as an example. The same approach applies to support, intake, and other conversations that combine questions with actions. + +**Jump to:** [Architecture](#speaker-and-reasoner-roles) · [Squads](#start-with-one-assistant) · [Conversation flow](#questions-and-changes-of-topic) · [Voice design](#speaking-style) · [Testing](#testing-conversations) + +## Speaker and reasoner roles + +The **speaker** carries the spoken interaction: listening, responding, asking questions, and deciding when to delegate. The **reasoner** takes a delegated task, follows its instructions, and calls tools. A tool connects the assistant to information or actions in your service. + +Both models belong to the same assistant. The caller stays in one conversation while the reasoner works behind it. + +| Give the speaker | Give the reasoner | +| --- | --- | +| The assistant's role and the caller's goal | The procedure for completing each task | +| Tone, language, and pacing | Required information and confirmation rules | +| When to clarify or delegate | Which tools to use and in what order | +| How to explain results to the caller | How to interpret tool results and failures | + +For example, the speaker needs to know that it can help find an appointment. The reasoner needs to know which service returns availability, what information that service requires, and when booking is allowed. + +This separation lets you keep the speaker's instructions focused. Put detailed business procedures in the reasoner prompt. Your services remain responsible for enforcing permissions and validating actions. + +### When to delegate + +Give the speaker a clear reason to ask for help: “Delegate requests to find an appointment that fits the caller's preferences.” Give the reasoner the steps needed to produce that result. + +| Layer | Example instruction | +| --- | --- | +| Speaker | “Help the caller find a suitable appointment. Delegate availability checks and booking requests.” | +| Reasoner | “Confirm the date and time zone. Look up available slots. Book only after the caller confirms a specific slot.” | +| Service | Validate the requested slot and create the booking only when the request meets your requirements. | + +Keep the conditions for delegation concrete. A greeting or a request to repeat an already confirmed time may stay with the speaker. A fresh lookup, changed booking request, or end-call request needs the reasoner and its tools. + +The reasoner's result gives the speaker information for its next response. The speaker can explain that information in the context of what the caller just said. See [Configure GPT-Live](/gpt-live/configuration#create-an-assistant) for an assistant with both prompts and a lookup tool. + +## Start with one assistant + +Early testing suggests GPT-Live handles many of the needs that previously led builders to squads out of the box. The speaker maintains the conversation while the reasoner handles detailed instructions and task sequencing. You may be able to cover the full use case with one assistant. + +Suppose your scheduling squad has separate assistants for intake, availability, and booking. Start by carrying those responsibilities into one GPT-Live assistant: + +```mermaid +flowchart TD + A[One scheduling conversation] + A --> B[Speaker guides the caller] + B --> C[Reasoner manages the task] + C --> D[Collect required details] + D --> E[Look up availability] + E --> F[Book after confirmation] +``` + +This diagram shows responsibilities, not a fixed script for every call. The caller may supply several details at once or ask a question between stages. + +Preserve the outcomes and business rules you need. Test whether the new architecture still needs the divisions you created for the old one. Pay attention to missed steps and recovery from errors before consolidating a production flow. + +Existing squad configurations and assistant handoffs are [currently unsupported](/gpt-live/limitations#can-i-use-an-existing-squad). The early finding is that some use cases need fewer assistants, not that every squad can be converted automatically. + +## Questions and changes of topic + +Give the conversation a destination and a few meaningful stages: understand the request, find suitable options, confirm the choice, and explain the result. The caller doesn't need to answer questions in the order you wrote them. + +Consider this opening: + +> **Caller:** I need something Friday afternoon. I've been there before. + +The caller has already supplied a day, a preference, and a useful detail. Design the assistant to acknowledge those details and ask for the next missing piece, such as the exact date or time zone. Repeating questions the caller has answered makes even responsive speech feel mechanical. + +A detour can help the caller make the next decision: + +> **Caller:** Before we pick a time, how long does the appointment take? +> +> **Assistant:** Let me check that for you. Then we can find a time that works. + +If the duration is already available in trusted context, the speaker can explain it. If it needs a fresh lookup, delegate that question. Then return to scheduling with the caller's earlier preferences intact. + +A useful speaker instruction is: + +```text title="Keep the conversation connected" +Use details the caller has already provided. Ask one question for the +next missing piece. If the caller asks a related question, address it +and then return to the unfinished task. Delegate questions that need +new information from our services. +``` + +## Talking while tools run + +A lookup doesn't always need to bring the conversation to a stop. While it runs, the speaker can collect information that doesn't depend on the result. + +For example, if your service can retrieve a whole day's availability, the speaker can ask about morning or afternoon while that request is running. If the lookup requires that preference as an input, ask first. + +```mermaid +sequenceDiagram + participant C as Caller + participant S as Speaker + participant R as Reasoner and tools + C->>S: What is available on Friday? + S->>R: Look up Friday availability + S->>C: Morning or afternoon? + C->>S: Afternoon, ideally after two + R-->>S: Available times + S->>C: Explain matching options +``` + +This illustrates a conversation to design and test, not a guarantee of exact event timing. Your prompts and tool requirements determine which work can overlap. + +Keep updates proportionate. One short “I'm checking” can be enough. A forced acknowledgment before every tool call, repeated status updates, or unrelated small talk can make the wait feel longer. + +### Confirming results + +The assistant can acknowledge the request before the work finishes. Its confirmation should follow the result: + +| Moment | What the caller can hear | +| --- | --- | +| The request is understood | “I'll check Friday afternoon.” | +| The lookup finishes | “There are openings at two and four.” | +| A booking tool confirms success | “Your two o'clock appointment is booked.” | + +The [example assistant](/gpt-live/configuration#create-an-assistant) only looks up availability. Add a booking tool before teaching it to offer the final step. + +### When callers change the request + +If the caller says “Actually, Thursday” during the lookup, the speaker should acknowledge the new request. Your application should match each result to the request it answers so Friday's result doesn't drive Thursday's booking. + +Interrupting speech doesn't cancel an action already sent to a service. Keep changes and cancellations explicit in your application. The [limitations FAQ](/gpt-live/limitations#does-an-interruption-cancel-a-tool-call) explains this boundary. + +## Speaking style + +Choose both a voice and a way of speaking. The voice establishes a starting sound. Speaker instructions shape pacing, warmth, emphasis, and how much the assistant says at once. + +Write a short voice brief alongside the task brief. Describe behavior someone could hear and evaluate: + +```text title="Example voice brief" +Sound warm and conversational. Keep routine answers brief. +Move briskly through familiar information, but slow down when reading +appointment dates and times. Give one instruction at a time when +explaining a process. Follow requests to speed up, slow down, or +explain more. Leave room for the caller to respond. +``` + +These are prompted behaviors, not precise speed or pause controls. Test the spoken result. [Programmatic speech control](/gpt-live/limitations#can-i-inject-commentary-or-control-speech-during-a-call) is currently unsupported in Vapi's GPT-Live integration. + +### Pacing + +A consistent personality doesn't require a constant pace. The same assistant can move quickly through familiar information and become more measured when a caller needs help. + +| Moment | Delivery to try | +| --- | --- | +| Caller knows what they want | Short questions and concise answers | +| Caller is choosing between options | A few options at a time, with room to respond | +| Assistant reads a date or unfamiliar name | Slower delivery and clear articulation | +| Caller asks “Can you walk me through it?” | One step at a time | +| Caller says “Just the short version” | A brief summary with an offer to explain more | + +Treat those requests as part of the conversation design. Test “slow down,” “a little faster,” and “say that another way” alongside your business scenarios. + +### Personality packs + +Vapi's personality packs add speaking-style guidance to your speaker prompt: + +| Pack | What its guidance encourages | +| --- | --- | +| **Active listening** (`eager-listener`) | Brief acknowledgments at natural openings, with room for the caller to continue | +| **Humming** (`idle-hummer`) | Soft humming or wordless sounds during quiet moments while waiting for tools | +| **Bouncy** (`bouncy`) | More expressive pitch, emphasis, and rhythm, adapting to the caller's mood | +| **Unhurried** (`unhurried`) | A slower pace, clear articulation, and space between ideas | + +Choose a pack because it suits the experience. Humming might fit an informal concierge and feel out of place during a serious support issue. Expressiveness and speed are separate choices: an animated assistant can still speak slowly and clearly. + +Start with one pack, listen, and adjust the speaker prompt. Packs add guidance. They don't guarantee identical delivery on every call. See [personality configuration](/gpt-live/configuration#personality-and-language) and [voice previews](/gpt-live/configuration#voices). + +## Testing conversations + +Review a short set of conversations before expanding the prompt. Include a direct request, a detour, information supplied out of order, a slow lookup, and a request to change speaking style. + +Ask whether the assistant used what the caller had already said, delegated at the right moment, and returned naturally to unfinished work. Listen for whether its pace helps the caller make progress. + +Keep a few representative calls as a baseline. Repeat them when either prompt changes. Use [post-call analysis](/gpt-live/configuration#review-and-monitor-calls) to track outcomes, while reviewing audio for timing and delivery. Check [testing limitations](/gpt-live/limitations#can-i-run-simulations-and-evals) before choosing an automated test method. + +For additional model-level examples, see OpenAI's [GPT-Live prompting guide](https://developers.openai.com/api/docs/guides/live-prompting). Use Vapi's fields when applying that guidance to your assistant. diff --git a/fern/gpt-live/limitations.mdx b/fern/gpt-live/limitations.mdx new file mode 100644 index 000000000..7a20c68e0 --- /dev/null +++ b/fern/gpt-live/limitations.mdx @@ -0,0 +1,117 @@ +--- +title: GPT-Live limitations and FAQs +subtitle: Check compatibility before you build or migrate +description: Check GPT-Live support for squads, call control, commentary injection, transfers, simulations, voices, fallbacks, recordings, and post-call data in Vapi. +slug: gpt-live/limitations +--- + +GPT-Live has a different set of supported features from Vapi's other voice architectures. Use this page to check the requirements of your application before migrating. + +These limits describe **Vapi's GPT-Live integration**. A capability in OpenAI's direct API isn't necessarily exposed through Vapi. + +**Jump to:** [Calls and voices](#calls-and-voice) · [Transfers](#transfers-and-keypad-input) · [Squads](#can-i-use-an-existing-squad) · [Live control](#can-i-inject-commentary-or-control-speech-during-a-call) · [Testing](#can-i-run-simulations-and-evals) · [Recordings](#recordings-and-post-call-data) + +## Calls and voice + +| Feature | Support | +| --- | --- | +| Browser WebRTC, native Twilio, and Vapi SIP | Supported | +| Raw WebSocket | Supported with mono, signed 16-bit little-endian PCM at 24 kHz. Set the rate explicitly | +| Native Telnyx or Vonage | Unsupported | +| 22 preset OpenAI voices | Supported. See [voice previews](/gpt-live/configuration#voices) | +| Custom or cloned voices, or other voice providers | Unsupported | +| Changing voice during a call | Unsupported. Save the assistant and start a new call | +| Model and voice fallbacks | Unsupported. GPT-Live also cannot be a fallback model | +| Reconnecting to an existing GPT-Live session | Unsupported. Start a new call and check the status of pending work | + +See [Connect a call](/gpt-live/configuration#connect-a-call) for working connection examples. + +## Transfers and keypad input + +Function tools, API request tools, and `endCall` are supported. The other supported tool types, `transferCall` and `dtmf`, depend on the call connection: + +| Feature | Support | +| --- | --- | +| Blind transfer | Supported on native Twilio and Vapi SIP to configured phone or SIP destinations | +| Transfer on browser or raw WebSocket calls | Unsupported | +| Warm transfer or generated transfer summary | Unsupported | +| Transfer fallback plans or SIP `bye` verb | Unsupported | +| SIP transfer with extension dialing | Unsupported. Use a direct destination | +| Outgoing DTMF | Supported on SIP using RTP DTMF. Pass digits as a string, for example `"0011*#"` | +| Outgoing DTMF on Twilio, browser, or raw WebSocket calls | Unsupported | +| SIP INFO DTMF | Unsupported | +| Incoming keypad collection | Unsupported. Leave `keypadInputPlan.enabled` disabled | + +Other tool types are unsupported. Expose external lookups through a function or API request tool. Carrier acceptance of a transfer doesn't establish that the destination answered. + +## Can I use an existing squad? + +Squads and assistant handoffs are currently unsupported. GPT-Live's speaker and reasoner work within one assistant. Delegation doesn't switch the caller to another assistant. + +Early testing suggests some use cases no longer need the same divisions between assistants. See [Start with one assistant](/gpt-live/design#start-with-one-assistant) for how to rethink the architecture while keeping your business requirements. + +## Can I inject commentary or control speech during a call? + +Customer-initiated commentary injection, live message injection, and live call-control commands are currently unsupported. Incoming text commands on the raw WebSocket connection are ignored. + +Vapi uses OpenAI's commentary mechanism internally to pass reasoner results to the speaker. That doesn't expose `session.commentary.append` as a customer API. Use the supported [tool response flow](/gpt-live/configuration#connect-your-tools) to return results for requested work. + +A caller can ask the assistant to slow down or explain differently, and you can prompt it to respond to those requests. That conversational behavior is separate from an application sending a control command. + +## Can I guarantee exact speech or a fixed pause? + +GPT-Live generates speech and may paraphrase supplied text. Speaker instructions and personality packs guide delivery. They don't provide exact playback, a fixed speaking rate, or a programmatic pause while work completes. + +Audio URL greetings and the generated-message first-message mode are unsupported. Use a text `firstMessage` with `assistant-speaks-first`, or let the assistant wait for the caller. Without usable greeting text, it waits. + +If exact prerecorded wording or strict control of every spoken step is essential, choose an architecture with those controls. See [voice design](/gpt-live/design#speaking-style) for the choices available through prompting. + +## Which existing settings no longer apply? + +Classic transcriber selection, text-to-speech controls, and start/stop-speaking plans don't configure the GPT-Live conversation. General model fields such as `temperature` and `maxTokens` don't tune this integration. + +Use the speaker prompt for conversation behavior and the reasoner settings for task reasoning. The reasoner currently supports only the OpenAI models listed in [Model settings](/gpt-live/configuration#model-settings). + +## Does an interruption cancel a tool call? + +No. Interrupting speech changes the conversation, not an action already submitted to your service. Your application needs to check whether that action is pending, completed, or cancellable. + +If the caller corrects a date during a lookup, use the result that matches the updated request. For an example, see [When callers change the request](/gpt-live/design#when-callers-change-the-request). + +## Can I run Simulations and Evals? + +Voice Simulations are currently incompatible with GPT-Live. The simulation bridge uses 16 kHz audio and listen sockets. GPT-Live's raw WebSocket connection requires 24 kHz and doesn't provide listen sockets. + +Run voice scenarios through a [supported call connection](/gpt-live/configuration#connect-a-call). [Evals](/observability/evals-quickstart) can supplement testing of supported model decisions, but they don't establish GPT-Live's audio quality, timing, or interruption behavior. + +## Recordings and post-call data + +GPT-Live supports shared post-call analysis, structured outputs, scorecards, Boards, and Monitoring. Availability depends on your configuration and retained call data: + +| Setting or feature | Behavior | +| --- | --- | +| Recording disabled in the assistant | No recording is created | +| Recording-consent plan configured | Recording is disabled because GPT-Live doesn't collect that consent | +| Transcript-based analysis | Requires the relevant conversation data | +| Audio-based analysis | Requires an available recording | +| Monitoring with Zero Data Retention | Unsupported | +| Live listen or monitor sockets | Unsupported. Post-call Monitoring doesn't provide a live audio connection | + +See [Review and monitor calls](/gpt-live/configuration#review-and-monitor-calls) for setup and result fields. + +## Troubleshooting + +Use the symptom to find the setting or handler to check: + +| Symptom | Check | Fix | +| --- | --- | --- | +| GPT-Live is unavailable | Organization feature access | Contact Vapi to enable access. | +| Voice unsupported | `voice.provider` and `voice.voiceId` | Use OpenAI and a [supported voice ID](/gpt-live/configuration#voices). | +| Tool configuration unsupported | Tool type and resolved names | Use supported types and unique names. Inspect saved tools too. | +| Call fails on WebSocket startup | Audio format and sample rate | Set raw `pcm_s16le` at `24000` Hz explicitly. | +| WebSocket audio sounds distorted | Sample encoding, rate, and channels | Convert the source to mono, signed 16-bit little-endian PCM at 24 kHz. | +| Assistant never greets | First-message mode and usable text | Set `assistant-speaks-first` and a text `firstMessage`. | +| Assistant promises a lookup but no tool runs | Both prompts and attached tools | Add a concrete delegation rule and make the tool available to the reasoner. | +| Tool result never reaches the conversation | Handler response and matching ID | Return the actual `toolCallId`. Return the final async result through the original handler request. | +| Transfer fails | Transport and configured destination | Use a supported blind transfer. Check carrier requirements and destination restrictions. | +| No recording | Recording settings and consent plan | Check the documented recording behavior before the test. | diff --git a/fern/gpt-live/overview.mdx b/fern/gpt-live/overview.mdx new file mode 100644 index 000000000..0751d1241 --- /dev/null +++ b/fern/gpt-live/overview.mdx @@ -0,0 +1,59 @@ +--- +title: GPT-Live +subtitle: Voice conversations with tools and reasoning built in +description: Learn what GPT-Live brings to Vapi, how its speaker and reasoner work together, and where to start with conversation design, configuration, and compatibility. +slug: gpt-live/overview +--- + +GPT-Live is an OpenAI voice model that can listen and speak at the same time. A caller can clarify a request while the assistant is talking, ask a question during a lookup, or ask it to slow down. This is **full-duplex conversation**: audio can flow in both directions at once. + +Vapi connects that conversation to your tools and phone numbers, and provides the call records, analysis, and monitoring around it. + + +GPT-Live must be enabled for your Vapi organization. Contact Vapi if the model isn't available in your dashboard. + + +## How it works + +A GPT-Live assistant uses a **speaker** for the spoken interaction and a **reasoner** for tasks that need tools or detailed procedures. **Delegation** is the speaker asking the reasoner to handle one of those tasks. + +```mermaid +flowchart TD + C[Caller] <-->|Audio| S[Speaker] + S -->|Delegates a task| R[Reasoner] + R <-->|Requests and results| T[Tools and your services] + R -->|Result for the conversation| S +``` + +For a scheduling assistant, the speaker can guide the caller toward a suitable appointment. The reasoner checks availability through your service and returns the options. The speaker can keep listening while that work happens. + +GPT-Live handles speech directly. You don't configure a separate speech-to-text transcriber or text-to-speech provider for the conversation. + +## What changes with GPT-Live + +You can design the interaction around the caller's goal, with room for questions and changes along the way. Detailed procedures belong with the reasoner. The speaker's instructions focus on how to guide the conversation. + +You also have more to design than the words alone. Set a voice, give the speaker a delivery brief, and use personality packs to explore different styles. A concise assistant can slow down for a date, explain one step at a time, or give the short version when asked. + +[Design your assistant](/gpt-live/design) explains these choices with examples, including how to rethink an existing squad. + +## What Vapi provides + +| Capability | Where to learn more | +| --- | --- | +| Browser, phone, and WebSocket calls | [Connect a call](/gpt-live/configuration#connect-a-call) | +| Tools that look up information or take actions | [Connect your tools](/gpt-live/configuration#connect-your-tools) | +| Speaker prompts, reasoner settings, and personality packs | [Model settings](/gpt-live/configuration#model-settings) | +| 22 preset voices, with audio previews | [Choose a voice](/gpt-live/configuration#voices) | +| Transcripts, available recordings, and post-call analysis | [Review calls](/gpt-live/configuration#review-and-monitor-calls) | +| Structured outputs, scorecards, Boards, and Monitoring | [Track quality](/gpt-live/configuration#review-and-monitor-calls) | + +GPT-Live has a different set of supported features from Vapi's other voice architectures. Check [Limitations and FAQs](/gpt-live/limitations) if your application depends on squads, live call control, simulations, or a specific transfer flow. + +## Get started + +- **Explore the experience:** read [Design your assistant](/gpt-live/design) for architecture, conversation flow, and voice design. +- **Build a working assistant:** follow [Configure GPT-Live](/gpt-live/configuration#create-an-assistant) in the dashboard or API. +- **Assess a migration:** review [Limitations and FAQs](/gpt-live/limitations) against your current requirements. + +For the underlying model, see OpenAI's [GPT-Live documentation](https://developers.openai.com/api/docs/guides/live). This section describes Vapi's integration and its supported configuration.