From c450531ed1765d93db2ede235c1e8c49557de448 Mon Sep 17 00:00:00 2001 From: dev-seth-vapi Date: Tue, 22 Sep 2026 12:19:35 -0700 Subject: [PATCH] docs: clarify GPT-Live prompts, context, controls, and MCP tools --- fern/calls/call-features.mdx | 4 ++ fern/gpt-live/configuration.mdx | 80 ++++++++++++++++++++++++++++++-- fern/gpt-live/design.mdx | 82 ++++++++++++++++++++++++++++++++- fern/gpt-live/limitations.mdx | 22 +++++++-- fern/gpt-live/overview.mdx | 3 +- 5 files changed, 179 insertions(+), 12 deletions(-) diff --git a/fern/calls/call-features.mdx b/fern/calls/call-features.mdx index ea3e30e3b..02e5ce5bd 100644 --- a/fern/calls/call-features.mdx +++ b/fern/calls/call-features.mdx @@ -58,6 +58,10 @@ curl 'https://api.vapi.ai/call' ``` + +GPT-Live supports a subset of live call control: HTTP `end-call`, `append-context` for speaker commentary, thinking, or instructions, and cold `transfer` on supported phone connections. The `say`, `add-message`, and mute/unmute controls below aren't supported by GPT-Live. Use the [GPT-Live live call control guide](/gpt-live/configuration#live-call-control) for its request examples and limits. + + ## Call Control Features Once you have the `controlUrl`, you can use various control features during a live call. Here are all the available control options: diff --git a/fern/gpt-live/configuration.mdx b/fern/gpt-live/configuration.mdx index 71e72f90e..d96bcb302 100644 --- a/fern/gpt-live/configuration.mdx +++ b/fern/gpt-live/configuration.mdx @@ -9,7 +9,7 @@ Create an assistant that checks appointment availability and reads back the resu For help deciding what to put in each prompt, read [Design your assistant](/gpt-live/design). For feature restrictions, use [Limitations and FAQs](/gpt-live/limitations). -**Jump to:** [Create an assistant](#create-an-assistant) · [Model settings](#model-settings) · [Personality](#personality-and-language) · [Voices](#voices) · [Tools](#connect-your-tools) · [Calls](#connect-a-call) · [Analysis](#review-and-monitor-calls) +**Jump to:** [Create an assistant](#create-an-assistant) · [Model settings](#model-settings) · [Personality](#personality-and-language) · [Voices](#voices) · [Tools](#connect-your-tools) · [Calls](#connect-a-call) · [Live control](#live-call-control) · [Analysis](#review-and-monitor-calls) ## Before you start @@ -177,7 +177,7 @@ These fields belong to the assistant object. See [Create Assistant](/api-referen ## Prompt defaults and overrides -Set `model.speaker.instructions` for the spoken conversation and `model.reasoner.instructions` for task procedures. +Set `model.speaker.instructions` for the spoken conversation and `model.reasoner.instructions` for task procedures. See [where instructions belong](/gpt-live/design#where-should-an-instruction-go), the [paired prompt example](/gpt-live/design#example-split-a-scheduling-prompt), and [what context the reasoner receives](/gpt-live/design#what-context-does-the-reasoner-receive). - Explicit speaker instructions take precedence over `model.systemPrompt` and system messages in `model.messages`. If you omit the speaker instructions, Vapi uses that legacy configuration. - Omitting reasoner instructions uses Vapi's default reasoner prompt. Custom instructions replace that prompt entirely. @@ -285,9 +285,25 @@ Return a result for each request using its actual `toolCallId`: Replace the sample result with your service's data. If the lookup fails, return the failure so the assistant can explain it. See [Server authentication](/server-url/server-authentication) to protect the endpoint. -Supported tool types are `function`, `apiRequest`, `endCall`, `dtmf`, and `transferCall`. Give tools unique resolved function names. Save reusable tools and attach them through `model.toolIds`, or include definitions in `model.tools`. +Supported tool types are `function`, `apiRequest`, `mcp`, `endCall`, `dtmf`, and `transferCall`. Give tools unique resolved function names. Save reusable tools and attach them through `model.toolIds`, or include definitions in `model.tools`. -Use function or API request tools to connect external data sources. For phone actions, check [transfer and keypad support](/gpt-live/limitations#transfers-and-keypad-input). +Use function, API request, or MCP tools to connect external data sources. For phone actions, check [transfer and keypad support](/gpt-live/limitations#transfers-and-keypad-input). + +### MCP tools + +You can keep the MCP tools you already use with an existing Vapi assistant when migrating to GPT-Live. Attach the same saved tool IDs through `model.toolIds`, or keep your inline `type: "mcp"` definitions in `model.tools`. Existing server URLs, headers, credentials, and protocol settings are reused. See [MCP tools](/tools/mcp) for setup. + +For example, add your saved MCP tool ID to the assistant's `model` configuration, preserving any other tool IDs: + +```json +{ + "toolIds": ["YOUR_EXISTING_MCP_TOOL_ID"] +} +``` + +At call setup, Vapi connects to the MCP server and discovers its tools. Those tools are available to the **reasoner**. The speaker delegates work to the reasoner, which calls the tool and returns the result for the conversation. Put tool-use procedures in `model.reasoner.instructions` and delegation guidance in `model.speaker.instructions`. + +The discovered tool list stays fixed for that call. Each tool invocation opens a separate MCP connection; it doesn't rediscover the tool list. Start a new call to pick up changes to the server's available tools. GPT-Live doesn't play classic per-tool speech messages; use speaker instructions to guide how it talks while work runs. ### Slow and asynchronous requests @@ -339,7 +355,61 @@ Connect to `transport.websocketCallUrl` in the response. Send and receive binary Set `sampleRate` to `24000` for GPT-Live. Browser WebRTC and phone connections negotiate their own audio formats. -Close the WebSocket to end the call, or have the assistant invoke `endCall`. Incoming JSON call-control commands are not supported by this GPT-Live connection. See [WebSocket transport](/calls/websocket-transport) for the general transport, applying the restrictions above. +Close the WebSocket to end the call, have the assistant invoke `endCall`, or send an HTTP `end-call` request to the [live control URL](#live-call-control). Send control commands over HTTP, not as JSON frames on the audio WebSocket. See [WebSocket transport](/calls/websocket-transport) for the general transport, applying the restrictions above. + +## Live call control + +Your application can end an active GPT-Live call, append speaker context, or initiate a cold transfer. Send an HTTP POST to that call's `monitor.controlUrl`, returned in the call object. These commands aren't supported through audio WebSocket text frames or SDK data channels. + +Set `CONTROL_URL` to the returned URL. Call control must be enabled in the assistant's `monitorPlan`: setting `controlEnabled` to `false` disables it. If `controlAuthenticationEnabled` is `true`, include `Authorization: Bearer YOUR_VAPI_PUBLIC_API_KEY` from the same organization on each request. The examples below assume control authentication is disabled. Keep the control URL private. + +### End the call + +```bash +curl --fail-with-body -X POST "$CONTROL_URL" \ + -H "Content-Type: application/json" \ + -d '{"type":"end-call"}' +``` + +### Append speaker context + +Use `append-context` with a `kind` and nonblank `content` of up to 16,000 characters: + +| Kind | Purpose | Example content | +| --- | --- | --- | +| `commentary` | Give the speaker information to convey in its own words | `Your appointment is confirmed for Tuesday at 2 PM.` | +| `thinking` | Add silent context for the speaker to use | `The caller has already verified their account.` | +| `instructions` | Guide the speaker's behavior or ask it to try saying something | `Tell the caller their order is ready, then ask whether they need anything else.` | + +```bash +curl --fail-with-body -X POST "$CONTROL_URL" \ + -H "Content-Type: application/json" \ + -d '{"type":"append-context","kind":"instructions","content":"Tell the caller their order is ready, then ask whether they need anything else."}' +``` + +These messages go to the speaker. They don't add context to the reasoner's history or directly start reasoning or tool work. `thinking` doesn't request speech, but it isn't a separate private context store: the speaker can use that information in subsequent responses. + +A successful append response means the context was submitted. It doesn't guarantee exact wording, immediate speech, or completed playback. + +### Cold transfer + +On native Twilio and Vapi SIP calls, send a phone or SIP destination. For example: + +```bash +curl --fail-with-body -X POST "$CONTROL_URL" \ + -H "Content-Type: application/json" \ + -d '{"type":"transfer","destination":{"type":"number","number":"+14155550100","transferPlan":{"mode":"blind-transfer"}}}' +``` + +Replace the number with your transfer destination. For SIP, use a destination such as `{"type":"sip","sipUri":"sip:agent@example.com"}`. An application supplies the destination in this request; a reasoner using a `transferCall` tool selects from that tool's configured destinations. + +Cold transfer isn't supported on browser or raw WebSocket calls. Don't include pre-transfer `content`, warm-transfer plans, generated summaries, or fallback plans. See [transfer restrictions](/gpt-live/limitations#transfers-and-keypad-input). Carrier acceptance doesn't establish that the destination answered. + +### Responses and unsupported controls + +Successful commands return HTTP `200` with `{"status":"ok"}`. Rejected requests return an error, including when the call isn't active or is already transferring. If a `502` response reports an unconfirmed outcome, don't automatically retry: the action may have reached the provider. A `503` response stating that no action was taken is safe to retry. + +GPT-Live doesn't support `say`, `add-message`, or mute/unmute controls. Use `append-context` for speaker guidance. See [live control compatibility](/gpt-live/limitations#can-i-inject-commentary-or-control-speech-during-a-call). ## Review and monitor calls diff --git a/fern/gpt-live/design.mdx b/fern/gpt-live/design.mdx index 52779ce8f..e54e77061 100644 --- a/fern/gpt-live/design.mdx +++ b/fern/gpt-live/design.mdx @@ -11,7 +11,7 @@ The design task is to make those moments feel connected. Decide what the assista This guide uses scheduling as an example. The same approach applies to support, intake, and other conversations that combine questions with actions. -**Jump to:** [Architecture](#speaker-and-reasoner-roles) · [Squads](#start-with-one-assistant) · [Conversation flow](#questions-and-changes-of-topic) · [Voice design](#speaking-style) · [Testing](#testing-conversations) +**Jump to:** [Architecture](#speaker-and-reasoner-roles) · [Prompt example](#example-split-a-scheduling-prompt) · [Context](#what-context-does-the-reasoner-receive) · [Squads](#start-with-one-assistant) · [Conversation flow](#questions-and-changes-of-topic) · [Voice design](#speaking-style) · [Testing](#testing-conversations) ## Speaker and reasoner roles @@ -30,6 +30,84 @@ For example, the speaker needs to know that it can help find an appointment. The This separation lets you keep the speaker's instructions focused. Put detailed business procedures in the reasoner prompt. Your services remain responsible for enforcing permissions and validating actions. +### Where should an instruction go? + +Use `model.speaker.instructions` for **how to conduct the conversation and when to ask for help**. Use `model.reasoner.instructions` for **how to decide and act once asked**. + +| Instruction | Where it belongs | Why | +| --- | --- | --- | +| “Ask one question at a time and speak briefly.” | Speaker | Controls the interaction the caller hears. | +| “Delegate availability checks and booking requests.” | Speaker | Tells it when to involve the reasoner. | +| “Use `lookupAvailability` with an explicit date and time zone.” | Reasoner | Defines how to use a tool correctly. | +| “Call `bookAppointment` only after the caller confirms a specific slot.” | Reasoner | Defines the condition for taking an action. | +| “Never tell the caller a booking succeeded before the tool confirms it.” | Both, tailored to each role | The reasoner must verify success; the speaker must describe it accurately. | + +The prompts are separate. Instructions written only in the speaker prompt aren't automatically included in the reasoner prompt. Put rules that affect both conversation and actions in both prompts, using wording appropriate to each role. Keep detailed tool procedures in the reasoner prompt rather than copying the entire prompt into both fields. + +### Example: split a scheduling prompt + +Suppose the assistant can look up availability and book appointments. The example below assumes you have attached tools named `lookupAvailability` and `bookAppointment`. Replace those names and requirements with your actual tools. The [configuration walkthrough](/gpt-live/configuration#create-an-assistant) provides a lookup-only example if you haven't connected booking yet. + +```text title="model.speaker.instructions" +You help callers find and book appointments. Speak warmly and briefly. +Ask one question at a time. Use details the caller has already supplied. + +Collect an explicit date and time zone, then delegate availability checks. +Read back the returned options. Ask the caller to confirm one specific +slot before delegating a booking request. + +While a lookup runs, you may ask about preferences that don't change +its inputs. If the caller changes the date or time zone, acknowledge +the correction and delegate the updated request. Don't present an old +lookup result as an answer to the new request. + +Only say an appointment is booked after the reasoner returns a confirmed +booking. If information is missing or a tool fails, explain the issue +and ask for the next detail needed. +``` + +```text title="model.reasoner.instructions" +Help the speaker complete the caller's scheduling request using the +conversation transcript and the available tools. + +For availability, require an explicit date and time zone before calling +lookupAvailability. If either is missing or ambiguous, return a concise +clarification for the speaker to ask. Don't invent missing arguments. + +For booking, require the caller's explicit confirmation of a specific +slot returned by lookupAvailability. Call bookAppointment only when +those conditions and the tool's required arguments are satisfied. + +Use the latest request in the supplied transcript. Before taking a new +action, check prior tool results so you don't repeat a completed booking. +Don't retry an uncertain booking outcome without checking its status. + +Return the relevant date, time zone, slots, or confirmed booking details +for the speaker to explain. If a tool fails, report the failure and the +next useful step. Don't invent availability or claim success without a +successful tool result. +``` + +The speaker asks the caller for clarification; the reasoner returns what needs clarification. The reasoner doesn't speak to the caller directly. Your service should also enforce booking requirements: prompt instructions don't replace validation in the tool handler. + +### What context does the reasoner receive? + +**At each delegation, the reasoner receives the available spoken conversation transcript so far**, including caller and assistant turns. It also receives its own instructions, available tool definitions, and retained history from prior reasoner work and tool results within that call. It doesn't rely on a short summary or a few selected lines supplied by the speaker. + +The speaker and reasoner still have different context: + +| Context | Speaker | Reasoner | +| --- | --- | --- | +| Live audio | Listens and speaks continuously | Receives text, not the audio stream | +| Spoken conversation | Part of the live conversation | Receives the transcript available when delegation starts | +| Instructions | Speaker prompt and personality guidance | Reasoner prompt; the speaker prompt isn't automatically copied | +| Tool work | Receives results returned by the reasoner | Has tool definitions and retained reasoner/tool history | +| HTTP `append-context` | Receives the injected speaker context | Doesn't receive it directly in its history | + +The transcript is a snapshot of what Vapi has received at that point. Speech that arrives while the reasoner is already working doesn't automatically refresh that delegation's transcript. A subsequent delegation receives the updated transcript. Completed work and tool results can inform later reasoning, but the reasoner isn't continuously listening alongside the speaker. + +For example, if the caller says “Actually, Thursday” while a Friday lookup is running, the speaker can hear and acknowledge the correction. The ongoing lookup may still return Friday's availability. Delegate the changed request, match results to the date they answer, and don't assume an interruption cancels an action already sent to a tool. + ### When to delegate Give the speaker a clear reason to ask for help: “Delegate requests to find an appointment that fits the caller's preferences.” Give the reasoner the steps needed to produce that result. @@ -148,7 +226,7 @@ explaining a process. Follow requests to speed up, slow down, or explain more. Leave room for the caller to respond. ``` -These are prompted behaviors, not precise speed or pause controls. Test the spoken result. [Programmatic speech control](/gpt-live/limitations#can-i-inject-commentary-or-control-speech-during-a-call) is currently unsupported in Vapi's GPT-Live integration. +These are prompted behaviors, not precise speed or pause controls. Test the spoken result. Your application can also send [live speaker guidance](/gpt-live/configuration#live-call-control) with `append-context`; it doesn't guarantee exact wording, speed, or pauses. ### Pacing diff --git a/fern/gpt-live/limitations.mdx b/fern/gpt-live/limitations.mdx index c9b77f0fd..42c58dc77 100644 --- a/fern/gpt-live/limitations.mdx +++ b/fern/gpt-live/limitations.mdx @@ -28,11 +28,11 @@ See [Connect a call](/gpt-live/configuration#connect-a-call) for working connect ## Transfers and keypad input -Function tools, API request tools, and `endCall` are supported. The other supported tool types, `transferCall` and `dtmf`, depend on the call connection: +Function tools, API request tools, MCP tools, and `endCall` are supported. MCP tools run through the reasoner; see [MCP configuration](/gpt-live/configuration#mcp-tools). The other supported tool types, `transferCall` and `dtmf`, depend on the call connection: | Feature | Support | | --- | --- | -| Blind transfer | Supported on native Twilio and Vapi SIP to configured phone or SIP destinations | +| Blind transfer | Supported on native Twilio and Vapi SIP to phone or SIP destinations. Configure destinations for a `transferCall` tool, or supply one in an HTTP `transfer` request | | Transfer on browser or raw WebSocket calls | Unsupported | | Warm transfer or generated transfer summary | Unsupported | | Transfer fallback plans or SIP `bye` verb | Unsupported | @@ -42,7 +42,7 @@ Function tools, API request tools, and `endCall` are supported. The other suppor | SIP INFO DTMF | Unsupported | | Incoming keypad collection | Unsupported. Leave `keypadInputPlan.enabled` disabled | -Other tool types are unsupported. Expose external lookups through a function or API request tool. Carrier acceptance of a transfer doesn't establish that the destination answered. +Other tool types are unsupported. Expose external lookups through a function, API request, or MCP tool. Carrier acceptance of a transfer doesn't establish that the destination answered. ## Can I use an existing squad? @@ -52,7 +52,15 @@ Early testing suggests some use cases no longer need the same divisions between ## Can I inject commentary or control speech during a call? -Commentary injection, live message injection, and live call-control commands are currently unsupported. The raw WebSocket connection supports audio streaming only. Use the supported [tool response flow](/gpt-live/configuration#connect-your-tools) to return results for requested work. +Yes. Send HTTP requests to the call's `monitor.controlUrl` to: + +- End the call with `end-call`. +- Append `commentary`, `thinking`, or `instructions` to the speaker with `append-context`. +- Initiate a cold transfer with `transfer` on native Twilio or Vapi SIP calls. + +See [Live call control](/gpt-live/configuration#live-call-control) for request examples, authentication, and response behavior. Appended context affects the speaker; it doesn't update reasoner history or directly trigger tools. Submission doesn't guarantee exact wording or completed speech. + +The raw WebSocket connection supports audio streaming only. Send controls over HTTP, not WebSocket text frames or SDK data channels. Classic `say`, `add-message`, and mute/unmute commands remain unsupported. A caller can ask the assistant to slow down or explain differently, and you can prompt it to respond to those requests. That conversational behavior is separate from an application sending a control command. @@ -64,6 +72,12 @@ Audio URL greetings and the generated-message first-message mode are unsupported If exact prerecorded wording or strict control of every spoken step is essential, choose an architecture with those controls. See [voice design](/gpt-live/design#speaking-style) for the choices available through prompting. +## Does the reasoner get the conversation transcript? + +Yes. Each delegation receives the available caller and assistant transcript up to that point, along with the reasoner's instructions, tool definitions, and retained reasoner/tool history. It isn't limited to a short snippet chosen by the speaker. + +It doesn't receive raw audio or automatically inherit the speaker prompt. The transcript is a snapshot at delegation time; later speech doesn't automatically update a delegation already in progress. See [reasoner context](/gpt-live/design#what-context-does-the-reasoner-receive) for the context comparison and correction example, and [prompt guidance](/gpt-live/design#where-should-an-instruction-go) for what to put in each prompt. + ## Which existing settings no longer apply? Classic transcriber selection, text-to-speech controls, and start/stop-speaking plans don't configure the GPT-Live conversation. General model fields such as `temperature` and `maxTokens` don't tune this integration. diff --git a/fern/gpt-live/overview.mdx b/fern/gpt-live/overview.mdx index 143d368fe..d3c0f75b9 100644 --- a/fern/gpt-live/overview.mdx +++ b/fern/gpt-live/overview.mdx @@ -58,7 +58,8 @@ You also have more to design than the words alone. Set a voice, give the speaker | Capability | Where to learn more | | --- | --- | | Browser, phone, and WebSocket calls | [Connect a call](/gpt-live/configuration#connect-a-call) | -| Tools that look up information or take actions | [Connect your tools](/gpt-live/configuration#connect-your-tools) | +| Function, API request, and MCP tools that look up information or take actions | [Connect your tools](/gpt-live/configuration#connect-your-tools) | +| HTTP live call control: end call, speaker context, and cold transfer | [Control an active call](/gpt-live/configuration#live-call-control) | | Speaker prompts, reasoner settings, and personality packs | [Model settings](/gpt-live/configuration#model-settings) | | 22 preset voices, with audio previews | [Choose a voice](/gpt-live/configuration#voices) | | Transcripts, available recordings, and post-call analysis | [Review calls](/gpt-live/configuration#review-and-monitor-calls) |