diff --git a/fern/docs.yml b/fern/docs.yml index 1a027d6e8..b200d9747 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -698,6 +698,9 @@ navigation: - page: Quickstart path: observability/simulations-quickstart.mdx icon: fa-light fa-rocket + - page: Test an assistant + path: observability/simulations-test-tab.mdx + icon: fa-light fa-vial - page: Advanced path: observability/simulations-advanced.mdx icon: fa-light fa-flask-vial diff --git a/fern/observability/simulations-quickstart.mdx b/fern/observability/simulations-quickstart.mdx index 436a8b0d3..926ede871 100644 --- a/fern/observability/simulations-quickstart.mdx +++ b/fern/observability/simulations-quickstart.mdx @@ -7,6 +7,8 @@ slug: observability/simulations-quickstart In about 10 minutes, you'll run an AI tester through an appointment-booking conversation with Riley and get a pass or fail result. You'll also have a reusable simulation suite that you can run again after any change. +Testing a single assistant? Start from its [**Test tab**](/observability/simulations-test-tab), where Composer can generate a suite for you. + ## What you'll build A reusable **simulation suite** that asks an AI tester to book an appointment with Riley, Vapi's appointment-scheduler template assistant. The first criterion checks whether Riley reports the requested date and time with a confirmation number. This is a conversation-only demonstration, not proof that an appointment exists in a calendar. diff --git a/fern/observability/simulations-test-tab.mdx b/fern/observability/simulations-test-tab.mdx new file mode 100644 index 000000000..ab62b1b67 --- /dev/null +++ b/fern/observability/simulations-test-tab.mdx @@ -0,0 +1,94 @@ +--- +title: Test an assistant +subtitle: Generate, run, and track simulation suites from an assistant's Test tab +description: "Use an assistant's Test tab to generate a simulation suite with Composer, run it, and follow its latest results and run history without leaving the assistant." +slug: observability/simulations-test-tab +--- + +The **Test** tab lists [**simulation suites**](/observability/simulations-overview) that include this assistant in their target assignments. This includes suites Composer generates for the assistant and suites you assign to it yourself. From the tab, you can generate a suite with Composer, run or rerun a suite, and open its results and run history without leaving the assistant. + +## Prerequisites + +- An [**assistant**](/assistants) in your Vapi account. +- Permission to read test suites and test runs. Without both, you won't see the **Test** tab. +- To generate a suite, permission to create test suites. +- To run a suite, permission to create test runs. + + + Runs use the published version of the assistant. If you're editing a draft, the tab shows **Runs use the published version**. Publish your changes to test them. If [**traffic splitting**](/assistants/versioning/traffic-splitting) is enabled, publish the new version with 0% traffic so it handles no live calls. + + +## Generate a suite with Composer + +Open the assistant and select the **Test** tab. If no suites are assigned to the assistant, the tab shows **No test suites yet** with **Generate with Composer** and an **About simulations** link. Once the assistant has suites, select **Generate suite** above the table to create another. + + + Assistant Test tab empty state showing No test suites yet, Generate with Composer, and About simulations. + + +Both buttons open [**Composer**](/composer#generate-tests-for-an-assistant) in a new thread for this assistant: + + + + Composer reads the assistant's prompt and tools and drafts **Test requirements**. Select **Generate coverage** to accept them, or type the changes you want in the chat. + + + + Composer generates simulations from the requirements and saves them as a suite assigned to the assistant. The suite card shows **Saved to the Test tab**. Tools that reach your systems are mocked by default, and each simulation lists any tools it mocks. + + + + Select **Run suite** on the card to start a chat run with one iteration. The suite also appears in the **Test** tab. + + + +## Read the suite table + +The table shows up to 100 suites assigned to the assistant, one row per suite, with the latest run of that suite against this assistant. + + + Assistant Test tab suite table showing two assigned suites and the Simulations, Status, Progress, and Last run columns, with Generate suite and View run history actions. + + +| Column | What it shows | +| -- | -- | +| **Test suites assigned** | The suite name. Once the suite has run, select the row to open its latest run. | +| **Simulations** | How many simulations the latest run executed, with its chat or voice mode. Before the first run, the number of simulations in the suite. | +| **Status** | The latest run's status: **Queued**, **Running**, **Passed**, **Failed**, **Errored**, or **Canceled**. A suite that has never run for this assistant shows **Not run yet**. | +| **Progress** | A bar of the latest run's outcomes. During a run, it shows how many simulations are complete, such as `2 of 5 complete`. After the run, it summarizes the outcomes, such as `4 passed ยท 1 failed or errored`. | +| **Last run** | When the latest run ended, such as `5 minutes ago`. It updates every minute while the tab is open. | + +Each row ends with a **Run** or **Rerun** button. Once a suite has run, the row also shows a run history button that opens **Simulations** > **Runs** filtered to that suite and this assistant. To see every run for the assistant, select **View run history** above the table. + +## Run a suite + +Select **Run** on a suite that has never run for this assistant. If the suite can run in chat, it starts immediately as a chat run with one iteration, and the row updates as it progresses. + +In these cases, the **Run suite** dialog opens instead, so you can choose the mode and iterations: + +- The assistant needs voice, such as one that uses a GPT-Live model. **Voice** is the only mode offered. +- A simulation in the suite, its scenario, or its AI tester personality couldn't be loaded, for example because it was deleted. +- A scenario has hooks configured. + +To run a suite again, select **Rerun**. The dialog always opens so you can choose **Chat** or **Voice** and the number of **Iterations**. The mode defaults to the previous run's. Each rerun creates a new run and picks up any changes made to the suite. + + + A suite with no simulations can't run. Add at least one simulation before running it. **Run** and **Rerun** stay disabled while a run of the suite is queued or running. + + +## Next steps + + + + Build and run a simulation suite step by step. + + + Edit suites, review results, and maintain coverage. + + + Control what tools return during a simulation. + + + Generate tests for an assistant in conversation. + + diff --git a/fern/static/images/assistant-test-tab-empty.png b/fern/static/images/assistant-test-tab-empty.png new file mode 100644 index 000000000..314873feb Binary files /dev/null and b/fern/static/images/assistant-test-tab-empty.png differ diff --git a/fern/static/images/assistant-test-tab-suites.png b/fern/static/images/assistant-test-tab-suites.png new file mode 100644 index 000000000..42b6f85b1 Binary files /dev/null and b/fern/static/images/assistant-test-tab-suites.png differ