Skip to content

Commit ff2d035

Browse files
stephenvapiaiclaude
andcommitted
Add AssemblyAI Universal 3.6 Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent b702666 commit ff2d035

1 file changed

Lines changed: 84 additions & 47 deletions

File tree

‎fern/providers/transcriber/assembly-ai.mdx‎

Lines changed: 84 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -5,21 +5,18 @@ description: Configure AssemblyAI speech models, multilingual transcription, lan
55
slug: providers/transcriber/assembly-ai
66
---
77

8-
## Universal-Streaming
9-
10-
Universal-Streaming is AssemblyAI's purpose-built speech-to-text model that delivers ultra-fast, immutable transcripts in ~300ms with intelligent endpointing and superior accuracy for voice agents. It eliminates common pain points like misheard account numbers, awkward pauses, and premature cutoffs, enabling more natural and successful voice interactions.
11-
128
AssemblyAI provides real-time speech-to-text and turn detection for Vapi voice agents.
139

14-
## Supported capability
10+
## Universal Pro models
1511

16-
| Capability | Provider value |
17-
| --- | --- |
18-
| Speech-to-text | `assembly-ai` |
12+
Universal Pro is AssemblyAI's flagship streaming model, purpose-built for voice agents and harsh audio environments. It has broad language coverage with native code-switching across up to 32 languages, plus conversational context features like prompting, keyterms, and agent context (passing spoken agent turns back to the model to steer transcription of the next user reply).
1913

20-
## Speech-to-text
14+
| Model | Speech model | Languages |
15+
| --- | --- | --- |
16+
| Universal 3.6 Pro | `universal-3-6-pro` | 32 |
17+
| Universal 3.5 Pro | `universal-3-5-pro` | 18 |
2118

22-
Set `transcriber.provider` to `assembly-ai`. By default the `universal-streaming-english` speech model is used; set `transcriber.speechModel` to choose a different model.
19+
Universal 3.6 Pro (`universal-3-6-pro`) is the newest and most accurate model. It supports every Universal 3.5 Pro option and adds 14 languages.
2320

2421
```bash
2522
curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \
@@ -28,16 +25,47 @@ curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \
2825
-d '{
2926
"transcriber": {
3027
"provider": "assembly-ai",
31-
"language": "en"
28+
"speechModel": "universal-3-6-pro",
29+
"mode": "balanced",
30+
"prompt": "Customer support call for a bank. Callers ask about account balances, card activation, and recent transactions.",
31+
"languageCodes": ["en", "es"],
32+
"keytermsPrompt": ["Acme Bank", "overdraft", "routing number"],
33+
"agentContext": "Hi, thanks for calling Acme Bank. Can I get your account number?",
34+
"agentContextAutoUpdateEnabled": true
3235
}
3336
}'
3437
```
3538

36-
For additional API configuration options, review the [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) in the Create Assistant API reference.
39+
Both Universal Pro models support additional configuration:
40+
41+
- `mode`: Trade off accuracy and latency with `max_accuracy`, `balanced`, or `min_latency`. Defaults to `balanced`.
42+
- `prompt`: Provide additional context for the transcription model, up to 1,750 characters.
43+
- `agentContext`: Provide context about the assistant that helps the transcription model interpret caller speech, up to 1,750 characters.
44+
- `agentContextAutoUpdateEnabled`: Automatically send what the assistant just said as agent context after every assistant turn. Defaults to `false`.
45+
- `languageCodes`: Steer automatic language detection with language codes such as `["en", "es"]`.
46+
- `keytermsPrompt`: Boost recognition of domain-specific terms. Included at no extra cost on the Universal Pro models.
3747

38-
### Universal 3.5 Pro
48+
### Automatic agent context
49+
50+
When `agentContextAutoUpdateEnabled` is `true`, Vapi sends the assistant's last spoken reply to AssemblyAI as agent context after every assistant turn, replacing the previous value. The caller's answer is then transcribed in the context of the question it responds to. For example, after "What's your email address?", the model expects an email and formats it as one.
51+
52+
- `agentContext` still seeds the first turn, so set it to your assistant's first message.
53+
- Replies longer than 1,750 characters are trimmed to their last 1,750 characters.
54+
- Assistant turns the caller interrupts are not sent when the interruption is detected by voice activity (the default, `stopSpeakingPlan.numWords: 0`).
55+
56+
## Universal-Streaming
3957

40-
Universal 3.5 Pro (`universal-3-5-pro`) is AssemblyAI's most accurate voice-agent model, with automatic language detection and code-switching across 18 languages.
58+
Universal-Streaming is a transcription model offering a good balance of speed and cost-effectiveness. It is available in English, or multilingual across 6 languages: English, Spanish, German, French, Portuguese, and Italian.
59+
60+
## Supported capability
61+
62+
| Capability | Provider value |
63+
| --- | --- |
64+
| Speech-to-text | `assembly-ai` |
65+
66+
## Speech-to-text
67+
68+
Set `transcriber.provider` to `assembly-ai`. By default the `universal-streaming-english` speech model is used; set `transcriber.speechModel` to choose a different model.
4169

4270
```bash
4371
curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \
@@ -46,18 +74,12 @@ curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \
4674
-d '{
4775
"transcriber": {
4876
"provider": "assembly-ai",
49-
"speechModel": "universal-3-5-pro"
77+
"language": "en"
5078
}
5179
}'
5280
```
5381

54-
Universal 3.5 Pro supports additional configuration:
55-
56-
- `mode`: Trade off accuracy and latency with `max_accuracy`, `balanced`, or `min_latency`.
57-
- `prompt`: Provide additional context for the transcription model, up to 1,750 characters.
58-
- `agentContext`: Provide context about the assistant that helps the transcription model interpret caller speech, up to 1,750 characters.
59-
- `languageCodes`: Steer automatic language detection with language codes such as `["en", "es"]`.
60-
- `keytermsPrompt`: Boost recognition of domain-specific terms.
82+
For additional API configuration options, review the [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) in the Create Assistant API reference.
6183

6284
### Configure in the dashboard
6385

@@ -86,38 +108,53 @@ This guide details how to setup AssemblyAI as a transcriber for your assistant.
86108

87109
### Supported languages
88110

89-
Vapi supports AssemblyAI's English and multilingual Universal-Streaming modes, plus the Universal 3.5 Pro model.
111+
Vapi supports AssemblyAI's English and multilingual Universal-Streaming modes, plus the Universal 3.5 Pro and Universal 3.6 Pro models.
90112

91113
| Mode | Speech model | Language value | Supported languages |
92114
| --- | --- | --- | --- |
93115
| English | `universal-streaming-english` | `en` | English |
94116
| Multilingual | `universal-streaming-multilingual` | `multi` | English, French, German, Italian, Portuguese, and Spanish |
117+
| Universal 3.6 Pro | `universal-3-6-pro` | Automatic | 32 languages with automatic language detection and code-switching |
95118
| Universal 3.5 Pro | `universal-3-5-pro` | Automatic | 18 languages with automatic language detection and code-switching |
96119

97-
Universal 3.5 Pro accepts the following `languageCodes` values for language steering. Omit `languageCodes` to keep automatic detection and code-switching across all supported languages. See [AssemblyAI's language steering guide](https://www.assemblyai.com/blog/fix-language-steering-streaming-transcription) for provider guidance.
98-
99-
| Language | Language code |
100-
| --- | --- |
101-
| Arabic | `ar` |
102-
| Danish | `da` |
103-
| Dutch | `nl` |
104-
| English | `en` |
105-
| Finnish | `fi` |
106-
| French | `fr` |
107-
| German | `de` |
108-
| Hebrew | `he` |
109-
| Hindi | `hi` |
110-
| Italian | `it` |
111-
| Japanese | `ja` |
112-
| Mandarin Chinese | `zh` |
113-
| Norwegian | `no` |
114-
| Portuguese | `pt` |
115-
| Spanish | `es` |
116-
| Swedish | `sv` |
117-
| Turkish | `tr` |
118-
| Vietnamese | `vi` |
119-
120-
Keyterms prompting is supported with the `universal-streaming-english`, `universal-streaming-multilingual` and `universal-3-5-pro` models.
120+
The Universal Pro models accept the following `languageCodes` values for language steering. Omit `languageCodes` to keep automatic detection and code-switching across all languages the selected model supports. See [AssemblyAI's language steering guide](https://www.assemblyai.com/blog/fix-language-steering-streaming-transcription) for provider guidance.
121+
122+
| Language | Language code | Supported models |
123+
| --- | --- | --- |
124+
| Afrikaans | `af` | Universal 3.6 Pro |
125+
| Arabic | `ar` | Universal 3.5 Pro, Universal 3.6 Pro |
126+
| Cantonese | `yue` | Universal 3.6 Pro |
127+
| Catalan | `ca` | Universal 3.6 Pro |
128+
| Danish | `da` | Universal 3.5 Pro, Universal 3.6 Pro |
129+
| Dutch | `nl` | Universal 3.5 Pro, Universal 3.6 Pro |
130+
| English | `en` | Universal 3.5 Pro, Universal 3.6 Pro |
131+
| Estonian | `et` | Universal 3.6 Pro |
132+
| Finnish | `fi` | Universal 3.5 Pro, Universal 3.6 Pro |
133+
| French | `fr` | Universal 3.5 Pro, Universal 3.6 Pro |
134+
| Galician | `gl` | Universal 3.6 Pro |
135+
| German | `de` | Universal 3.5 Pro, Universal 3.6 Pro |
136+
| Hebrew | `he` | Universal 3.5 Pro, Universal 3.6 Pro |
137+
| Hindi | `hi` | Universal 3.5 Pro, Universal 3.6 Pro |
138+
| Italian | `it` | Universal 3.5 Pro, Universal 3.6 Pro |
139+
| Japanese | `ja` | Universal 3.5 Pro, Universal 3.6 Pro |
140+
| Korean | `ko` | Universal 3.6 Pro |
141+
| Mandarin Chinese | `zh` | Universal 3.5 Pro, Universal 3.6 Pro |
142+
| Marathi | `mr` | Universal 3.6 Pro |
143+
| Norwegian | `no` | Universal 3.5 Pro, Universal 3.6 Pro |
144+
| Norwegian Nynorsk | `nn` | Universal 3.6 Pro |
145+
| Persian | `fa` | Universal 3.6 Pro |
146+
| Portuguese | `pt` | Universal 3.5 Pro, Universal 3.6 Pro |
147+
| Romanian | `ro` | Universal 3.6 Pro |
148+
| Russian | `ru` | Universal 3.6 Pro |
149+
| Spanish | `es` | Universal 3.5 Pro, Universal 3.6 Pro |
150+
| Swedish | `sv` | Universal 3.5 Pro, Universal 3.6 Pro |
151+
| Turkish | `tr` | Universal 3.5 Pro, Universal 3.6 Pro |
152+
| Urdu | `ur` | Universal 3.6 Pro |
153+
| Vietnamese | `vi` | Universal 3.5 Pro, Universal 3.6 Pro |
154+
| Xhosa | `xh` | Universal 3.6 Pro |
155+
| Zulu | `zu` | Universal 3.6 Pro |
156+
157+
Keyterms prompting is supported with the `universal-streaming-english`, `universal-3-5-pro`, and `universal-3-6-pro` models.
121158

122159
## Related
123160

0 commit comments

Comments
 (0)