From 2e5fc9e6f031d151f59f93eecdd76100ccbf3356 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:28:31 -0700 Subject: [PATCH 01/30] feat(release-autopilot): add onboarding wizard skill Add launchdarkly-release-autopilot-onboarding: a step-by-step setup that asks which capabilities to enable, hands off the GitHub App link, maps chosen repos, guides managed or bring-your-own-key inference, and creates environment release plans for auto-releasing. One confirmed plan lists every write; plans are created before repos are mapped and account gates are flipped last. Route first-time setup to it from the settings skill. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude-plugin/plugin.json | 3 +- CHANGELOG.md | 2 + README.md | 1 + skills.json | 22 +- .../README.md | 42 +++ .../SKILL.md | 258 ++++++++++++++++++ .../marketplace.json | 21 ++ .../README.md | 1 + .../SKILL.md | 7 +- .../marketplace.json | 2 +- 10 files changed, 352 insertions(+), 7 deletions(-) create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index cf847c65..92a9ffbd 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -36,6 +36,7 @@ "./skills/agentcontrol/online-evals", "./skills/agentcontrol/projects", "./skills/agentcontrol/tools", - "./skills/release-autopilot/launchdarkly-release-autopilot-settings" + "./skills/release-autopilot/launchdarkly-release-autopilot-settings", + "./skills/release-autopilot/launchdarkly-release-autopilot-onboarding" ] } diff --git a/CHANGELOG.md b/CHANGELOG.md index 2903208b..4c499880 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ All notable changes to this repository will be documented in this file. ## Unreleased +- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.1.0-experimental: a step-by-step wizard that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and creates environment release plans for auto-releasing. One confirmed plan lists every write before any change. +- `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. - Rename `factory/launchdarkly-factory-settings` to `release-autopilot/launchdarkly-release-autopilot-settings` and switch it to the `*-release-autopilot-*` MCP tools (Claude Code plugin 1.0.2, Cursor plugin 1.0.3). Requires a hosted MCP server that exposes the new tool names. diff --git a/README.md b/README.md index 8895a6db..6ada9423 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor | Skill | Description | |-------|-------------| +| `release-autopilot/launchdarkly-release-autopilot-onboarding` | Step-by-step setup: pick capabilities, connect the GitHub App, map repos, choose inference, set up release plans | | `release-autopilot/launchdarkly-release-autopilot-settings` | Turn on GitHub App auto-flagging / auto-releasing, map `owner/name` repos, or diagnose why a PR was not classified (confirm before every write) | ### AgentControl diff --git a/skills.json b/skills.json index 32ac8986..618b8e52 100644 --- a/skills.json +++ b/skills.json @@ -419,11 +419,29 @@ "license": "Apache-2.0", "compatibility": "Requires the remotely hosted LaunchDarkly MCP server" }, + { + "name": "launchdarkly-release-autopilot-onboarding", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed or bring your own key), set up environment release plans, then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", + "version": "0.1.0-experimental", + "license": "Apache-2.0", + "compatibility": "Requires the remotely hosted LaunchDarkly MCP server", + "tags": [ + "launchdarkly", + "release-autopilot", + "onboarding", + "github", + "auto-flagging", + "auto-releasing", + "release-plans", + "mcp" + ] + }, { "name": "launchdarkly-release-autopilot-settings", - "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR.", + "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, set up release plans), use launchdarkly-release-autopilot-onboarding instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-settings", - "version": "1.3.0-experimental", + "version": "1.4.0-experimental", "license": "Apache-2.0", "compatibility": "Requires the remotely hosted LaunchDarkly MCP server", "tags": [ diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md new file mode 100644 index 00000000..d48b9161 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -0,0 +1,42 @@ +# LaunchDarkly Release Autopilot Onboarding Skill + +An Agent Skill that walks a user through setting up Release Autopilot: pick capabilities, connect the GitHub App, map repositories, choose Vega inference, and set up environment release plans. + +## Overview + +This skill teaches agents how to: + +- Ask which capabilities the user wants (auto-flagging, auto-releasing) and set up only what those need +- Hand the user the GitHub App install or authorize link, and continue after they finish +- Map chosen `owner/name` repos to LaunchDarkly projects +- Guide LaunchDarkly managed inference or bring-your-own-key setup +- Create environment release plans, with metrics, for auto-releasing +- Confirm one plan that lists every write before making any change, then verify + +## Installation (Local) + +- **Generic**: copy `skills/release-autopilot/launchdarkly-release-autopilot-onboarding/` into your client's skills path + +## Prerequisites + +The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools, and `list-release-plans`, `create-release-plan`, and `list-metrics` for auto-releasing. Release Autopilot is in alpha, so accounts outside the alpha get 404s. + +## Usage + +``` +Set up Release Autopilot for this repo +``` + +``` +Onboard us to Release Autopilot with auto-flagging and auto-releasing +``` + +## Related + +- [LaunchDarkly Release Autopilot Settings](../launchdarkly-release-autopilot-settings/): single settings changes and "why didn't my PR get classified?" +- [LaunchDarkly Metric Choose](../../metrics/launchdarkly-metric-choose/): pick metrics for guarded release plans +- [LaunchDarkly MCP Server](https://github.com/launchdarkly/mcp-server) + +## License + +Apache-2.0 diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md new file mode 100644 index 00000000..c7914bfd --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -0,0 +1,258 @@ +--- +name: launchdarkly-release-autopilot-onboarding +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed or bring your own key), set up environment release plans, then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +license: Apache-2.0 +compatibility: Requires the remotely hosted LaunchDarkly MCP server +metadata: + author: launchdarkly + version: "0.1.0-experimental" +--- + +# LaunchDarkly Release Autopilot Onboarding + +You're using a skill that walks a user through setting up Release Autopilot from nothing to a working setup. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and environment release plans. + +**This is a step-by-step wizard.** Ask one question at a time and wait for each answer. Gather every decision first, confirm one plan, and only then write. Release Autopilot opens and changes pull requests, and auto-releasing can ship code, so every write is production automation. + +This skill reuses the rules of [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) for every settings read and write. If the two disagree, the settings skill wins. + +## Prerequisites + +This skill requires the remotely hosted LaunchDarkly MCP server. + +**Required MCP tools:** +- `get-release-autopilot-settings` / `update-release-autopilot-settings` +- `list-release-autopilot-github-repos` +- `list-release-autopilot-repo-settings` / `get-release-autopilot-repo-settings` / `update-release-autopilot-repo-settings` + +**Required for auto-releasing:** +- `list-release-plans` / `create-release-plan` +- `list-metrics` (release plans can only attach metrics that exist) + +**Optional MCP tools:** +- `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text + +If a required tool is missing, stop and tell the user which one. Do not invent REST calls. + +Release Autopilot is in alpha. If every Release Autopilot tool returns 404, the account is not in the alpha. Tell the user and stop. + +## Core Principles + +1. **Only set up what they picked.** Release plans exist for auto-releasing. If the user picks auto-flagging only, skip the release plan step. +2. **Least privilege by default.** Recommend auto-flagging with approval required. Auto-releasing is opt-in and you must say it can ship code. +3. **Gather, confirm, then write.** No write happens before [Step 7](#step-7-confirm-the-plan). The confirmation lists every write with its current and proposed state. +4. **Never act in the user's browser.** GitHub install, GitHub authorization, and bring-your-own-key setup are browser steps. Give the exact link and wait for the user to say they finished. +5. **Only touch what the user named.** Never map every repo on the install. Never create release plans in environments the user did not pick. + +## Workflow + +**Progression rule:** When a step is complete, go to the next step. Do not stop between steps unless the step tells you to ask or wait. + +**Response scoping:** During this flow, treat user messages as answers to the current question. Do not take unrelated actions until the flow is finished. + +**Asking questions:** When a question has options, use `AskUserQuestion` with the labels and descriptions given below. Ask free-text questions as a normal message. + +**State tracking:** Track every decision. Step 7 re-states all of them. + +### Step 0: Welcome the user + +Present this intro (you may lightly adapt the wording, but keep the structure): + +> **Release Autopilot Setup** +> +> I'll set up Release Autopilot in a few short steps: +> +> 1. **Check your account**: I'll read your current Release Autopilot settings. +> 2. **Pick capabilities**: choose auto-flagging, auto-releasing, or both. +> 3. **Connect GitHub**: install or authorize the LaunchDarkly GitHub App. +> 4. **Map repositories**: choose which repos Release Autopilot watches, and which LaunchDarkly project each one uses. +> 5. **Choose inference**: run the agents on LaunchDarkly managed inference or on your own model provider key. +> 6. **Set up release plans** (auto-releasing only): define how each environment rolls out a change. +> 7. **Confirm the plan**: I'll show every change before I make it. +> 8. **Apply and verify**: I'll make the changes and check the result. +> +> Nothing changes until you approve the plan in step 7. + +Then go to Step 1. + +### Step 1: Read current state + +Run these reads. Do not write anything. + +1. `get-release-autopilot-settings`: the account gates. +2. `list-release-autopilot-repo-settings`: repos that are already mapped. +3. `list-release-autopilot-github-repos`: repos on the GitHub App install, or a `githubConnectUrl` if the App is not installed or the user has not authorized it. + +Release Autopilot's agents need generative AI turned on for the organization. There is no MCP tool to read that setting. If a later step fails with a generative AI or Vega eligibility error, tell the user that an Admin or Owner must select **Enable generative AI use cases** in Organization settings > Security, then stop. + +Summarize what is already set up in two or three sentences. If the account already has the capabilities on and repos mapped, ask: + +- label: "Add more", description: "Keep the current setup and add capabilities, repos, or release plans" +- label: "Change one setting", description: "Hand off to the settings skill for a single change" +- label: "Diagnose a PR", description: "A PR was not classified or auto-flagged. Hand off to the settings skill" + +For "Change one setting" or "Diagnose a PR", stop this skill and follow [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). + +### Step 2: Pick capabilities + +Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: + +- label: "Auto-flagging (Recommended)", description: "Release Autopilot reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" +- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with your environment release plans. This can ship code to users" + +If the user picks auto-flagging, ask header "Approval": + +- label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before Release Autopilot changes code" +- label: "No approval", description: "Release Autopilot changes PRs without an approval step" + +Do not offer auto-cleanup. It is not part of the Release Autopilot settings API yet. + +If the user picks neither capability, stop and tell them there is nothing to set up. + +### Step 3: Connect GitHub + +If Step 1 returned a `githubConnectUrl`, read its `flow`: + +- `flow=install`: the LaunchDarkly GitHub App is not installed on their GitHub organization. Release Autopilot cannot see any pull request until it is. +- `flow=authorize`: the App is installed, but this member has not authorized GitHub, so repos cannot be listed. + +Give the user the exact URL. Tell them to open it in a browser where they are logged in to LaunchDarkly. To install, they must be a GitHub organization owner, or they must request the install from one. Then ask header "GitHub": + +- label: "Done", description: "I finished the GitHub step" +- label: "Need an org owner", description: "I can't install the App myself. Pause setup here" + +On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, say so and repeat this step. On "Need an org owner", summarize the decisions so far and stop. The user can run this skill again after the App is installed. + +If Step 1 returned repos and no `githubConnectUrl`, skip this step. + +### Step 4: Map repositories + +1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. If a repo the user wants is not on the list, tell them to add it to the App's repository access in GitHub, then repeat Step 3's "Done" check. +2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. +3. **Skip repos that are already mapped** to the chosen project. Tell the user they are already set up. + +### Step 5: Choose inference + +Release Autopilot runs its agents on Vega. Each project has Vega inference settings. Use `AskUserQuestion` with header "Inference": + +- label: "LaunchDarkly managed", description: "The agents run on a model that LaunchDarkly hosts. No setup" +- label: "Bring your own key", description: "The agents run on your own Anthropic or OpenAI account. Your provider bills you for usage" + +For "LaunchDarkly managed", go to Step 6. + +For "Bring your own key", there is no MCP tool for inference settings yet, so this is a browser step: + +1. The key lives in the account's integration settings. Tell the user to add an Anthropic or OpenAI key there. +2. The project must then use it. For each project from Step 4, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL) and tell them to select their own key and save. LaunchDarkly checks the key with the provider when they save, and shows the cause if the check fails. + +Then ask header "Inference": + +- label: "Done", description: "I saved the inference settings for every project" +- label: "Use managed for now", description: "Skip this. I'll switch to my own key later" + +You cannot read inference settings, so record the answer as the user's statement, not as verified. + +### Step 6: Set up release plans (auto-releasing only) + +Skip this step if the user did not pick auto-releasing. + +Auto-releasing releases each change with a **release plan**: a reusable recipe for one environment, made of phases, rollout steps, and metrics. A plan with metrics makes the release guarded, so a regression on a metric can roll it back. Release plans replace project-level release policies. + +For each project from Step 4: + +1. **Choose environments.** Use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Auto-releasing works in the environments that have a plan. +2. **Check existing plans.** For each chosen environment, call `list-release-plans`. If a complete plan exists, show its name, phases, and metrics, and ask header "Plan": + - label: "Use this plan", description: "Keep the existing plan for this environment" + - label: "Create another", description: "Add a new plan for this environment" +3. **Create a plan where needed.** Follow the `create-release-plan` tool guidance. Ask each setting as its own question, never bundled: + 1. Randomization unit: the context kind to roll out by, such as `user` or `account`. + 2. Rollout steps: offer header "Rollout" with options such as "1% → 10% → 50% → 100%, 4h holds", or "10% → 100%, 24h hold". The user can type another schedule in Other. + 3. Advancement: header "Advance", options "Automatic", description "Move to the next step when the hold ends and metrics are healthy", and "Manual", description "A person moves each step". + 4. Metrics: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, follow [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) and [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md) and come back. A plan without metrics is progressive, not guarded: say so. + 5. Cleanup: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. + + Every phase needs at least one clause. For a phase that targets everyone, use `{ attribute: "kind", op: "in", values: [""], contextKind: "" }`. + + Record the plan. Do not call `create-release-plan` yet. + +`list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so it needs a new complete plan in that environment. + +### Step 7: Confirm the plan + +**STOP. Do not write anything until the user explicitly confirms.** A detailed request at the start is not approval. + +List every write in the order you will make it. For each write, state the current and the proposed state and the blast radius, using the wording rules from the settings skill's [Confirm before write](../launchdarkly-release-autopilot-settings/SKILL.md#confirm-before-write): + +1. **Release plans**: one line per environment, with the plan name, steps, advancement, and metrics. +2. **Repo mappings**: one line per repo, naming the project and **both** inherited capabilities. +3. **Account settings**: for example, "Auto-flagging: OFF → ON, approval required. Mapped repos that inherit this setting can start getting auto-flagging PRs." If auto-releasing turns on, say: "Auto-releasing: OFF → ON. This can ship code." +4. **Browser steps the user reported done**: GitHub and inference, marked as reported, not verified. + +Then use `AskUserQuestion` with header "Confirm": + +- label: "Looks good", description: "Make every change in this plan" +- label: "Change something", description: "Adjust the plan before you start" + +On "Change something", make the change and show the full plan again. Do not go to Step 8 until the user selects "Looks good". + +### Step 8: Apply + +Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. + +1. `create-release-plan` for each new plan. +2. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. +3. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. + +The order matters. Release plans come first so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. + +### Step 9: Verify + +1. `get-release-autopilot-settings`: the account gates match the plan. +2. `get-release-autopilot-repo-settings` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. +3. `list-release-plans` for each environment: the new plans are listed. + +Report a short checklist with each item marked verified or reported by the user. Then tell the user how to see it work: + +- Open a pull request in a mapped repo. Release Autopilot reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. +- If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. + +## Edge Cases + +| Situation | Action | +|-----------|--------| +| Every Release Autopilot tool returns 404 | The account is not in the alpha. **STOP** and tell the user | +| `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | +| Repo not on the install list | The App does not have access to that repo. The user adds it in GitHub, then you list again | +| Repo already mapped to a different project | Show the current mapping. Remapping changes where flags are created, so it needs its own line in Step 7 | +| Auto-releasing picked but no metrics exist | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | +| Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | +| User lacks permission for a write | Report the failed write and the missing permission. Do not retry with other settings | +| Bring your own key chosen | Browser step only. Record it as reported, never as verified | + +## What NOT to Do + +- Don't write before Step 7's "Looks good". +- Don't turn on auto-releasing unless the user picked it. +- Don't turn off approval for auto-flagging unless the user picked "No approval". +- Don't map repos the user did not pick, or iterate over the install list. +- Don't install or authorize GitHub, or set up a provider key, on the user's behalf. +- Don't send `autoCleanup`. +- Don't report a browser step as verified. + +## References + +- [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md): settings rules, write confirmations, and PR diagnosis +- [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) / [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md): metrics for guarded release plans +- [Vega inference settings](https://launchdarkly.com/docs/home/getting-started/vega#vega-inference-settings) +- [Guarded rollouts](https://launchdarkly.com/docs/home/releases/guarded-rollouts) + +## Example Flows + +**Happy path: auto-flagging only, managed inference.** +Step 1 → nothing set up, install list has repos. Step 2 → auto-flagging, approval required. Step 3 → skipped. Step 4 → maps `acme/web` to project `web`. Step 5 → managed. Step 6 → skipped. Step 7 → two writes: map `acme/web` inheriting auto-flagging ON and auto-releasing OFF, then account auto-flagging ON with approval. Step 8 → applies. Step 9 → verified. + +**Both capabilities, no GitHub App yet.** +Step 1 → `githubConnectUrl` with `flow=install`. Step 2 → both capabilities. Step 3 → user installs the App, selects "Done", repos appear. Step 4 → maps `acme/api` to `api`. Step 5 → bring your own key, user saves settings in the browser. Step 6 → `production` has no plan; creates `prod-guarded` with 1% → 10% → 50% → 100%, automatic advancement, metric `api-error-rate`. Step 7 → plan, mapping, and account writes, plus two reported browser steps. Steps 8–9 → applied and verified. + +**Org owner needed: pause.** +Step 3 → `flow=install`, user selects "Need an org owner". Summarize decisions so far and stop. Nothing was written. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json new file mode 100644 index 00000000..6fd07789 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json @@ -0,0 +1,21 @@ +{ + "name": "launchdarkly-release-autopilot-onboarding", + "description": "Step-by-step Release Autopilot onboarding: pick capabilities, connect GitHub, map repos, choose inference, set up release plans", + "version": "0.1.0-experimental", + "author": "LaunchDarkly", + "repository": "https://github.com/launchdarkly/ai-tooling", + "skills": ["./"], + "tags": [ + "launchdarkly", + "release-autopilot", + "onboarding", + "github", + "auto-flagging", + "auto-releasing", + "release-plans", + "mcp" + ], + "requirements": { + "mcp-servers": ["@launchdarkly/mcp-server"] + } +} diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md index ddfb6b47..3e4d98e0 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md @@ -37,6 +37,7 @@ Which GitHub repos are mapped for Release Autopilot, and which can I map? ## Related +- [LaunchDarkly Release Autopilot Onboarding](../launchdarkly-release-autopilot-onboarding/): step-by-step first-time setup - [LaunchDarkly Flag Create](../../feature-flags/launchdarkly-flag-create/): Create flags after Release Autopilot is mapped - [LaunchDarkly MCP Server](https://github.com/launchdarkly/mcp-server) - [LaunchDarkly Docs](https://docs.launchdarkly.com) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md index 5c9c8ab9..8c6c1662 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md @@ -1,11 +1,11 @@ --- name: launchdarkly-release-autopilot-settings -description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR." +description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, set up release plans), use launchdarkly-release-autopilot-onboarding instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly - version: "1.3.0-experimental" + version: "1.4.0-experimental" --- # LaunchDarkly Release Autopilot Settings @@ -73,6 +73,7 @@ After they confirm, apply **only** the fields in the proposal. Then verify (do n ## Choose a workflow +- **First-time setup** ("set up Release Autopilot", "onboard us", or no capability on and no repo mapped) → [`launchdarkly-release-autopilot-onboarding`](../launchdarkly-release-autopilot-onboarding/SKILL.md). It gathers every decision first and confirms one plan. - **"Why didn't Release Autopilot classify / auto-flag my PR?"** (or similar) → [Why didn't my PR get classified?](#why-didnt-my-pr-get-classified) first. Do not write settings in that workflow. - Configure / map / turn on / unmap → [Configure settings](#configure-settings). @@ -139,6 +140,6 @@ To unmap: [Confirm before write](#confirm-before-write), then `delete-release-au - Auto-cleanup - Completing GitHub OAuth inside the tool. Hand the user `githubConnectUrl` instead. -- Vega BYOK +- Vega inference and bring-your-own-key setup, and environment release plans. The [onboarding skill](../launchdarkly-release-autopilot-onboarding/SKILL.md) guides both. - Observability MCP (`github_repositories`) — do not require a second MCP server just to map a repo - Changing classification rules, GitHub App permissions, or org install scope via this skill diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json index 818cb907..af89cebf 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json @@ -1,7 +1,7 @@ { "name": "launchdarkly-release-autopilot-settings", "description": "Configure LaunchDarkly Release Autopilot settings via MCP, or diagnose why a PR was not classified", - "version": "1.3.0-experimental", + "version": "1.4.0-experimental", "author": "LaunchDarkly", "repository": "https://github.com/launchdarkly/ai-tooling", "skills": ["./"], From 6180eb93bf06fd53c31e62832173806457e1486a Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:37:53 -0700 Subject: [PATCH 02/30] fix(release-autopilot-onboarding): address dry-run and review findings Handle 403 plan gating on release plans, join mapped repos to the install list by githubRepoId, override unpicked capabilities off on mapped repos, defer metric creation to Step 8, clarify the batched confirmation versus the settings skill, and point deprecated MCP servers at the latest one. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../SKILL.md | 35 ++++++++++++------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index c7914bfd..1a73a8b8 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -14,7 +14,7 @@ You're using a skill that walks a user through setting up Release Autopilot from **This is a step-by-step wizard.** Ask one question at a time and wait for each answer. Gather every decision first, confirm one plan, and only then write. Release Autopilot opens and changes pull requests, and auto-releasing can ship code, so every write is production automation. -This skill reuses the rules of [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) for every settings read and write. If the two disagree, the settings skill wins. +This skill reuses the rules of [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) for every settings read and write: least privilege, `owner/name` only, never `autoCleanup`, and the current → proposed wording with blast radius. One difference is deliberate: the settings skill confirms each write on its own, and this skill confirms every write in one itemized plan at [Step 7](#step-7-confirm-the-plan). That itemized confirmation satisfies the settings skill's confirm-before-write rule. For anything else, the settings skill wins. ## Prerequisites @@ -32,7 +32,9 @@ This skill requires the remotely hosted LaunchDarkly MCP server. **Optional MCP tools:** - `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text -If a required tool is missing, stop and tell the user which one. Do not invent REST calls. +If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. + +If only the auto-releasing tools are missing, the server predates environment release plans. Run the rest of the setup, and tell the user that auto-releasing setup needs the latest server. Release Autopilot is in alpha. If every Release Autopilot tool returns 404, the account is not in the alpha. Tell the user and stop. @@ -81,11 +83,13 @@ Run these reads. Do not write anything. 1. `get-release-autopilot-settings`: the account gates. 2. `list-release-autopilot-repo-settings`: repos that are already mapped. -3. `list-release-autopilot-github-repos`: repos on the GitHub App install, or a `githubConnectUrl` if the App is not installed or the user has not authorized it. +3. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. + +`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. Release Autopilot's agents need generative AI turned on for the organization. There is no MCP tool to read that setting. If a later step fails with a generative AI or Vega eligibility error, tell the user that an Admin or Owner must select **Enable generative AI use cases** in Organization settings > Security, then stop. -Summarize what is already set up in two or three sentences. If the account already has the capabilities on and repos mapped, ask: +Summarize what is already set up in two or three sentences. If the account already has a capability on or any repo mapped, ask: - label: "Add more", description: "Keep the current setup and add capabilities, repos, or release plans" - label: "Change one setting", description: "Hand off to the settings skill for a single change" @@ -102,8 +106,8 @@ Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: If the user picks auto-flagging, ask header "Approval": -- label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before Release Autopilot changes code" -- label: "No approval", description: "Release Autopilot changes PRs without an approval step" +- label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before Release Autopilot changes code. This is an account setting, so it applies to every mapped repo" +- label: "No approval", description: "Release Autopilot changes PRs without an approval step, in every mapped repo" Do not offer auto-cleanup. It is not part of the Release Autopilot settings API yet. @@ -129,7 +133,8 @@ If Step 1 returned repos and no `githubConnectUrl`, skip this step. 1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. If a repo the user wants is not on the list, tell them to add it to the App's repository access in GitHub, then repeat Step 3's "Done" check. 2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. -3. **Skip repos that are already mapped** to the chosen project. Tell the user they are already set up. +3. **Check existing mappings** with the `githubRepoId` join from Step 1. Skip a repo that is already mapped to the chosen project, and tell the user it is already set up. A repo mapped to a different project needs its own line in Step 7, because remapping changes where its flags are created. +4. **Say what starts when.** If the account already has a capability on, mapping a repo starts that automation in the repo as soon as Step 8 maps it. ### Step 5: Choose inference @@ -168,13 +173,15 @@ For each project from Step 4: 1. Randomization unit: the context kind to roll out by, such as `user` or `account`. 2. Rollout steps: offer header "Rollout" with options such as "1% → 10% → 50% → 100%, 4h holds", or "10% → 100%, 24h hold". The user can type another schedule in Other. 3. Advancement: header "Advance", options "Automatic", description "Move to the next step when the hold ends and metrics are healthy", and "Manual", description "A person moves each step". - 4. Metrics: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, follow [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) and [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md) and come back. A plan without metrics is progressive, not guarded: say so. + 4. Metrics: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create. Do not create them yet: Step 8 creates them with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md) before the release plans that use them. A plan without metrics is progressive, not guarded: say so. 5. Cleanup: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. Every phase needs at least one clause. For a phase that targets everyone, use `{ attribute: "kind", op: "in", values: [""], contextKind: "" }`. Record the plan. Do not call `create-release-plan` yet. +If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans. Tell the user that auto-releasing needs a plan that includes them, drop auto-releasing from this setup, and continue with the other steps. Do not retry. + `list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so it needs a new complete plan in that environment. ### Step 7: Confirm the plan @@ -183,8 +190,8 @@ For each project from Step 4: List every write in the order you will make it. For each write, state the current and the proposed state and the blast radius, using the wording rules from the settings skill's [Confirm before write](../launchdarkly-release-autopilot-settings/SKILL.md#confirm-before-write): -1. **Release plans**: one line per environment, with the plan name, steps, advancement, and metrics. -2. **Repo mappings**: one line per repo, naming the project and **both** inherited capabilities. +1. **Metrics and release plans**: one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. +2. **Repo mappings**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, the mapping sends a repo override that turns it off, so the repo cannot inherit it from the account. Say so: "Auto-releasing: OFF (repo override; the account has it ON)." 3. **Account settings**: for example, "Auto-flagging: OFF → ON, approval required. Mapped repos that inherit this setting can start getting auto-flagging PRs." If auto-releasing turns on, say: "Auto-releasing: OFF → ON. This can ship code." 4. **Browser steps the user reported done**: GitHub and inference, marked as reported, not verified. @@ -199,9 +206,10 @@ On "Change something", make the change and show the full plan again. Do not go t Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. -1. `create-release-plan` for each new plan. -2. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. -3. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. +1. Metrics recorded in Step 6, with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). +2. `create-release-plan` for each new plan. +3. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. For each capability the user did not pick, send `{ "enabled": false }` for it, so the repo does not inherit it from the account. +4. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. The order matters. Release plans come first so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. @@ -225,6 +233,7 @@ Report a short checklist with each item marked verified or reported by the user. | Repo not on the install list | The App does not have access to that repo. The user adds it in GitHub, then you list again | | Repo already mapped to a different project | Show the current mapping. Remapping changes where flags are created, so it needs its own line in Step 7 | | Auto-releasing picked but no metrics exist | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | +| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Drop auto-releasing from this setup and tell the user | | Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | | User lacks permission for a write | Report the failed write and the missing permission. Do not retry with other settings | | Bring your own key chosen | Browser step only. Record it as reported, never as verified | From 139505b20d08e65db08a4dad816e78b8c846a040 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:39:06 -0700 Subject: [PATCH 03/30] fix(release-autopilot-onboarding): handle uninstalled GitHub orgs and accounts without release plans Found in a catamorphic dry run: a repo in an org without the App install gets no githubConnectUrl, and an account can run auto-releasing on release policies while release plans return 403. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../launchdarkly-release-autopilot-onboarding/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 1a73a8b8..35fca5ff 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -131,7 +131,7 @@ If Step 1 returned repos and no `githubConnectUrl`, skip this step. ### Step 4: Map repositories -1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. If a repo the user wants is not on the list, tell them to add it to the App's repository access in GitHub, then repeat Step 3's "Done" check. +1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. If a repo the user wants is not on the list, compare its owner to the owners on the list. If the owner is on the list, the App lacks access to that repo: tell the user to add it to the App's repository access in GitHub. If the owner is not on the list, the App is not installed on that GitHub organization, and `githubConnectUrl` will not appear because the account already has an install elsewhere: tell the user that an owner of that organization must install the LaunchDarkly GitHub App on it. Either way, wait for "Done" and list again. 2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. 3. **Check existing mappings** with the `githubRepoId` join from Step 1. Skip a repo that is already mapped to the chosen project, and tell the user it is already set up. A repo mapped to a different project needs its own line in Step 7, because remapping changes where its flags are created. 4. **Say what starts when.** If the account already has a capability on, mapping a repo starts that automation in the repo as soon as Step 8 maps it. @@ -180,7 +180,7 @@ For each project from Step 4: Record the plan. Do not call `create-release-plan` yet. -If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans. Tell the user that auto-releasing needs a plan that includes them, drop auto-releasing from this setup, and continue with the other steps. Do not retry. +If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans, so this skill cannot create them. Do not retry. Auto-releasing can still be on for such an account, using the project's release policies instead. Call `list-release-policies` for the project and show the user what it returns, including any `warnings` such as a missing or incomplete policy. Record no plan writes, and in Step 7 say that auto-releasing in this project depends on its release policies. `list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so it needs a new complete plan in that environment. @@ -230,10 +230,10 @@ Report a short checklist with each item marked verified or reported by the user. |-----------|--------| | Every Release Autopilot tool returns 404 | The account is not in the alpha. **STOP** and tell the user | | `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | -| Repo not on the install list | The App does not have access to that repo. The user adds it in GitHub, then you list again | +| Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | | Repo already mapped to a different project | Show the current mapping. Remapping changes where flags are created, so it needs its own line in Step 7 | | Auto-releasing picked but no metrics exist | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | -| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Drop auto-releasing from this setup and tell the user | +| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Show `list-release-policies` and its warnings instead, and create no plans | | Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | | User lacks permission for a write | Report the failed write and the missing permission. Do not retry with other settings | | Bring your own key chosen | Browser step only. Record it as reported, never as verified | From 060567eeeb2b620559e8b3a38cb5f69c1a66ebc8 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:41:37 -0700 Subject: [PATCH 04/30] fix(release-autopilot-onboarding): BYOK is optional; managed Vega inference is the default Co-Authored-By: Claude Opus 5.5 (1M context) --- .../launchdarkly-release-autopilot-onboarding/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 35fca5ff..6947d172 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -138,9 +138,9 @@ If Step 1 returned repos and no `githubConnectUrl`, skip this step. ### Step 5: Choose inference -Release Autopilot runs its agents on Vega. Each project has Vega inference settings. Use `AskUserQuestion` with header "Inference": +Release Autopilot runs its agents on Vega. Bring your own key is optional: a project that does not set it up uses LaunchDarkly managed Vega inference. Use `AskUserQuestion` with header "Inference": -- label: "LaunchDarkly managed", description: "The agents run on a model that LaunchDarkly hosts. No setup" +- label: "LaunchDarkly managed (Recommended)", description: "The default. The agents run on Vega inference that LaunchDarkly hosts. No setup" - label: "Bring your own key", description: "The agents run on your own Anthropic or OpenAI account. Your provider bills you for usage" For "LaunchDarkly managed", go to Step 6. From f6b1be681bf070a607c0271975177b1bdfbc5d3a Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:43:07 -0700 Subject: [PATCH 05/30] fix(release-autopilot-onboarding): address adversarial review #3 Treat release plans as optional with a release policy fallback; keep auto-releasing off where a policy is missing unless the user accepts the fallback; hand remaps to the settings skill; override unpicked capabilities on already-mapped repos; limit metric creation to LaunchDarkly; present managed inference and BYOK as equal options. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- README.md | 2 +- skills.json | 2 +- .../README.md | 4 +- .../SKILL.md | 55 ++++++++++++------- .../marketplace.json | 2 +- 6 files changed, 40 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c499880..ac831d99 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this repository will be documented in this file. ## Unreleased -- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.1.0-experimental: a step-by-step wizard that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and creates environment release plans for auto-releasing. One confirmed plan lists every write before any change. +- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.1.0-experimental: a step-by-step wizard that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, falling back to release policies where plans are not available. One confirmed plan lists every write before any change. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. diff --git a/README.md b/README.md index 6ada9423..252b5168 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor | Skill | Description | |-------|-------------| -| `release-autopilot/launchdarkly-release-autopilot-onboarding` | Step-by-step setup: pick capabilities, connect the GitHub App, map repos, choose inference, set up release plans | +| `release-autopilot/launchdarkly-release-autopilot-onboarding` | Step-by-step setup: pick capabilities, connect the GitHub App, map repos, choose inference, review release setup | | `release-autopilot/launchdarkly-release-autopilot-settings` | Turn on GitHub App auto-flagging / auto-releasing, map `owner/name` repos, or diagnose why a PR was not classified (confirm before every write) | ### AgentControl diff --git a/skills.json b/skills.json index 618b8e52..1272d824 100644 --- a/skills.json +++ b/skills.json @@ -421,7 +421,7 @@ }, { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed or bring your own key), set up environment release plans, then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review or set up how each environment releases (environment release plans, or release policies where plans are not available), then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", "version": "0.1.0-experimental", "license": "Apache-2.0", diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md index d48b9161..5406bc6e 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -10,7 +10,7 @@ This skill teaches agents how to: - Hand the user the GitHub App install or authorize link, and continue after they finish - Map chosen `owner/name` repos to LaunchDarkly projects - Guide LaunchDarkly managed inference or bring-your-own-key setup -- Create environment release plans, with metrics, for auto-releasing +- Review or create environment release plans for auto-releasing, or show release policies where plans are not available - Confirm one plan that lists every write before making any change, then verify ## Installation (Local) @@ -19,7 +19,7 @@ This skill teaches agents how to: ## Prerequisites -The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools, and `list-release-plans`, `create-release-plan`, and `list-metrics` for auto-releasing. Release Autopilot is in alpha, so accounts outside the alpha get 404s. +The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools, and `list-release-policies` and `list-metrics` for auto-releasing (`list-release-plans` and `create-release-plan` when the account has environment release plans). Release Autopilot is in alpha, so accounts outside the alpha get 404s. ## Usage diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 6947d172..ce107913 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -1,6 +1,6 @@ --- name: launchdarkly-release-autopilot-onboarding -description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed or bring your own key), set up environment release plans, then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review or set up how each environment releases (environment release plans, or release policies where plans are not available), then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: @@ -10,7 +10,7 @@ metadata: # LaunchDarkly Release Autopilot Onboarding -You're using a skill that walks a user through setting up Release Autopilot from nothing to a working setup. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and environment release plans. +You're using a skill that walks a user through setting up Release Autopilot from nothing to a working setup. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and how each environment releases. **This is a step-by-step wizard.** Ask one question at a time and wait for each answer. Gather every decision first, confirm one plan, and only then write. Release Autopilot opens and changes pull requests, and auto-releasing can ship code, so every write is production automation. @@ -26,21 +26,24 @@ This skill requires the remotely hosted LaunchDarkly MCP server. - `list-release-autopilot-repo-settings` / `get-release-autopilot-repo-settings` / `update-release-autopilot-repo-settings` **Required for auto-releasing:** +- `list-release-policies` +- `list-metrics` (plans and policies can only attach metrics that exist) + +**Used for auto-releasing when available:** - `list-release-plans` / `create-release-plan` -- `list-metrics` (release plans can only attach metrics that exist) **Optional MCP tools:** - `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. -If only the auto-releasing tools are missing, the server predates environment release plans. Run the rest of the setup, and tell the user that auto-releasing setup needs the latest server. +If `list-release-plans` or `create-release-plan` is missing, use the release policy path in Step 6, the same as for a 403. Release Autopilot is in alpha. If every Release Autopilot tool returns 404, the account is not in the alpha. Tell the user and stop. ## Core Principles -1. **Only set up what they picked.** Release plans exist for auto-releasing. If the user picks auto-flagging only, skip the release plan step. +1. **Only set up what they picked.** Release plans and policies matter only for auto-releasing. If the user picks auto-flagging only, skip Step 6. 2. **Least privilege by default.** Recommend auto-flagging with approval required. Auto-releasing is opt-in and you must say it can ship code. 3. **Gather, confirm, then write.** No write happens before [Step 7](#step-7-confirm-the-plan). The confirmation lists every write with its current and proposed state. 4. **Never act in the user's browser.** GitHub install, GitHub authorization, and bring-your-own-key setup are browser steps. Give the exact link and wait for the user to say they finished. @@ -69,7 +72,7 @@ Present this intro (you may lightly adapt the wording, but keep the structure): > 3. **Connect GitHub**: install or authorize the LaunchDarkly GitHub App. > 4. **Map repositories**: choose which repos Release Autopilot watches, and which LaunchDarkly project each one uses. > 5. **Choose inference**: run the agents on LaunchDarkly managed inference or on your own model provider key. -> 6. **Set up release plans** (auto-releasing only): define how each environment rolls out a change. +> 6. **Review release setup** (auto-releasing only): check or define how each environment rolls out a change. > 7. **Confirm the plan**: I'll show every change before I make it. > 8. **Apply and verify**: I'll make the changes and check the result. > @@ -102,7 +105,7 @@ For "Change one setting" or "Diagnose a PR", stop this skill and follow [`launch Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: - label: "Auto-flagging (Recommended)", description: "Release Autopilot reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" -- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with your environment release plans. This can ship code to users" +- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with the environment's release plan or release policy. This can ship code to users" If the user picks auto-flagging, ask header "Approval": @@ -133,14 +136,17 @@ If Step 1 returned repos and no `githubConnectUrl`, skip this step. 1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. If a repo the user wants is not on the list, compare its owner to the owners on the list. If the owner is on the list, the App lacks access to that repo: tell the user to add it to the App's repository access in GitHub. If the owner is not on the list, the App is not installed on that GitHub organization, and `githubConnectUrl` will not appear because the account already has an install elsewhere: tell the user that an owner of that organization must install the LaunchDarkly GitHub App on it. Either way, wait for "Done" and list again. 2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. -3. **Check existing mappings** with the `githubRepoId` join from Step 1. Skip a repo that is already mapped to the chosen project, and tell the user it is already set up. A repo mapped to a different project needs its own line in Step 7, because remapping changes where its flags are created. +3. **Check existing mappings** with the `githubRepoId` join from Step 1. + - Mapped to the chosen project, and its effective capabilities match the user's picks: tell the user it is already set up, and write nothing. + - Mapped to the chosen project, but a capability the user did not pick is effectively on: record a repo override that turns it off, as its own line in Step 7. + - Mapped to a different project: do not remap it in this flow. Remapping changes where its flags are created. Tell the user to use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) to unmap it first, then run this step again. 4. **Say what starts when.** If the account already has a capability on, mapping a repo starts that automation in the repo as soon as Step 8 maps it. ### Step 5: Choose inference Release Autopilot runs its agents on Vega. Bring your own key is optional: a project that does not set it up uses LaunchDarkly managed Vega inference. Use `AskUserQuestion` with header "Inference": -- label: "LaunchDarkly managed (Recommended)", description: "The default. The agents run on Vega inference that LaunchDarkly hosts. No setup" +- label: "LaunchDarkly managed", description: "The default if you choose nothing. The agents run on Vega inference that LaunchDarkly hosts. No setup" - label: "Bring your own key", description: "The agents run on your own Anthropic or OpenAI account. Your provider bills you for usage" For "LaunchDarkly managed", go to Step 6. @@ -157,15 +163,15 @@ Then ask header "Inference": You cannot read inference settings, so record the answer as the user's statement, not as verified. -### Step 6: Set up release plans (auto-releasing only) +### Step 6: Review release setup (auto-releasing only) Skip this step if the user did not pick auto-releasing. -Auto-releasing releases each change with a **release plan**: a reusable recipe for one environment, made of phases, rollout steps, and metrics. A plan with metrics makes the release guarded, so a regression on a metric can roll it back. Release plans replace project-level release policies. +Auto-releasing releases each change the way the environment is set up to release. On accounts with **environment release plans**, that is a plan: a reusable recipe for one environment, made of phases, rollout steps, and metrics. On accounts without them, it is the project's **release policies**. Either way, metrics make the release guarded, so a regression on a metric can roll it back. With neither, a release can fall back to defaults, which are often an immediate release. For each project from Step 4: -1. **Choose environments.** Use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Auto-releasing works in the environments that have a plan. +1. **Choose environments.** Use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Check each environment you pick. 2. **Check existing plans.** For each chosen environment, call `list-release-plans`. If a complete plan exists, show its name, phases, and metrics, and ask header "Plan": - label: "Use this plan", description: "Keep the existing plan for this environment" - label: "Create another", description: "Add a new plan for this environment" @@ -173,14 +179,21 @@ For each project from Step 4: 1. Randomization unit: the context kind to roll out by, such as `user` or `account`. 2. Rollout steps: offer header "Rollout" with options such as "1% → 10% → 50% → 100%, 4h holds", or "10% → 100%, 24h hold". The user can type another schedule in Other. 3. Advancement: header "Advance", options "Automatic", description "Move to the next step when the hold ends and metrics are healthy", and "Manual", description "A person moves each step". - 4. Metrics: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create. Do not create them yet: Step 8 creates them with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md) before the release plans that use them. A plan without metrics is progressive, not guarded: say so. + 4. Metrics: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create. Do not create them yet: Step 8 creates them in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md) before the release plans that use them. This skill creates the metric only. If the event is not sent by the app yet, list instrumenting it as a follow-up for the user; do not edit code or `.env` files in this flow. A plan without metrics is progressive, not guarded: say so. 5. Cleanup: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. Every phase needs at least one clause. For a phase that targets everyone, use `{ attribute: "kind", op: "in", values: [""], contextKind: "" }`. Record the plan. Do not call `create-release-plan` yet. -If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans, so this skill cannot create them. Do not retry. Auto-releasing can still be on for such an account, using the project's release policies instead. Call `list-release-policies` for the project and show the user what it returns, including any `warnings` such as a missing or incomplete policy. Record no plan writes, and in Step 7 say that auto-releasing in this project depends on its release policies. +If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans, so this skill cannot create them. Do not retry. Auto-releasing can still be on for such an account, using the project's release policies instead. Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. + +If the policies have no warnings for the chosen environments, say that auto-releasing in this project uses those policies. If there is any warning, such as `missing_policy` or `incomplete_policy`, a release there can fall back to defaults, which are often immediate. Ask header "Release defaults": + +- label: "Keep auto-releasing off here (Recommended)", description: "Map the repo with auto-releasing off until the project has a complete release policy" +- label: "Accept the fallback", description: "Auto-release with the default behavior, which can be an immediate release to everyone" + +For "Keep auto-releasing off here", the repo mapping in Step 8 sends an auto-releasing override that turns it off. Release policies are edited in the project's release settings in the LaunchDarkly UI; this skill does not change them. `list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so it needs a new complete plan in that environment. @@ -191,7 +204,7 @@ If `list-release-plans` returns 403 with `Plan does not allow this operation`, t List every write in the order you will make it. For each write, state the current and the proposed state and the blast radius, using the wording rules from the settings skill's [Confirm before write](../launchdarkly-release-autopilot-settings/SKILL.md#confirm-before-write): 1. **Metrics and release plans**: one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. -2. **Repo mappings**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, the mapping sends a repo override that turns it off, so the repo cannot inherit it from the account. Say so: "Auto-releasing: OFF (repo override; the account has it ON)." +2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, the mapping sends a repo override that turns it off, so the repo cannot inherit it from the account. Say so: "Auto-releasing: OFF (repo override; the account has it ON)." 3. **Account settings**: for example, "Auto-flagging: OFF → ON, approval required. Mapped repos that inherit this setting can start getting auto-flagging PRs." If auto-releasing turns on, say: "Auto-releasing: OFF → ON. This can ship code." 4. **Browser steps the user reported done**: GitHub and inference, marked as reported, not verified. @@ -206,18 +219,18 @@ On "Change something", make the change and show the full plan again. Do not go t Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. -1. Metrics recorded in Step 6, with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). +1. Metrics recorded in Step 6, created in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). Skip its instrumentation steps. 2. `create-release-plan` for each new plan. 3. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. For each capability the user did not pick, send `{ "enabled": false }` for it, so the repo does not inherit it from the account. 4. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. -The order matters. Release plans come first so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. +The order matters. When the plan creates release plans, they come first, so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. ### Step 9: Verify 1. `get-release-autopilot-settings`: the account gates match the plan. 2. `get-release-autopilot-repo-settings` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. -3. `list-release-plans` for each environment: the new plans are listed. +3. If plans were created, `list-release-plans` for each environment: the new plans are listed. On the release policy path, report the policy state from Step 6 instead. Report a short checklist with each item marked verified or reported by the user. Then tell the user how to see it work: @@ -231,9 +244,9 @@ Report a short checklist with each item marked verified or reported by the user. | Every Release Autopilot tool returns 404 | The account is not in the alpha. **STOP** and tell the user | | `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | | Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | -| Repo already mapped to a different project | Show the current mapping. Remapping changes where flags are created, so it needs its own line in Step 7 | +| Repo already mapped to a different project | Do not remap here. The user unmaps it with the settings skill, then maps it in this flow | | Auto-releasing picked but no metrics exist | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | -| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Show `list-release-policies` and its warnings instead, and create no plans | +| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Show `list-release-policies` and its warnings, create no plans, and keep auto-releasing off where a policy is missing unless the user accepts the fallback | | Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | | User lacks permission for a write | Report the failed write and the missing permission. Do not retry with other settings | | Bring your own key chosen | Browser step only. Record it as reported, never as verified | @@ -258,7 +271,7 @@ Report a short checklist with each item marked verified or reported by the user. ## Example Flows **Happy path: auto-flagging only, managed inference.** -Step 1 → nothing set up, install list has repos. Step 2 → auto-flagging, approval required. Step 3 → skipped. Step 4 → maps `acme/web` to project `web`. Step 5 → managed. Step 6 → skipped. Step 7 → two writes: map `acme/web` inheriting auto-flagging ON and auto-releasing OFF, then account auto-flagging ON with approval. Step 8 → applies. Step 9 → verified. +Step 1 → nothing set up, install list has repos. Step 2 → auto-flagging, approval required. Step 3 → skipped. Step 4 → maps `acme/web` to project `web`. Step 5 → managed. Step 6 → skipped. Step 7 → two writes: map `acme/web` with auto-releasing OFF (repo override), then account auto-flagging ON with approval. Step 8 → applies. Step 9 → verified. **Both capabilities, no GitHub App yet.** Step 1 → `githubConnectUrl` with `flow=install`. Step 2 → both capabilities. Step 3 → user installs the App, selects "Done", repos appear. Step 4 → maps `acme/api` to `api`. Step 5 → bring your own key, user saves settings in the browser. Step 6 → `production` has no plan; creates `prod-guarded` with 1% → 10% → 50% → 100%, automatic advancement, metric `api-error-rate`. Step 7 → plan, mapping, and account writes, plus two reported browser steps. Steps 8–9 → applied and verified. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json index 6fd07789..c752b68b 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json @@ -1,6 +1,6 @@ { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step Release Autopilot onboarding: pick capabilities, connect GitHub, map repos, choose inference, set up release plans", + "description": "Step-by-step Release Autopilot onboarding: pick capabilities, connect GitHub, map repos, choose inference, review release setup", "version": "0.1.0-experimental", "author": "LaunchDarkly", "repository": "https://github.com/launchdarkly/ai-tooling", From d777602e0fb0b8d6c567f1028df04dcbb0340620 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:45:12 -0700 Subject: [PATCH 06/30] fix(release-autopilot-onboarding): release policies required today; handle unmatched mapped repos Release policies gate auto-releasing until release plans replace them by GA, so a missing or incomplete policy keeps auto-releasing off for the repo. Found in a catamorphic dry run: a mapped repo can be missing from the install list, and the release-plans 403 is account-wide. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../SKILL.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index ce107913..43bd72b9 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -88,7 +88,9 @@ Run these reads. Do not write anything. 2. `list-release-autopilot-repo-settings`: repos that are already mapped. 3. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. -`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. +`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. A mapped `githubRepoId` with no match on the install list is a repo the App can no longer see, or one this member cannot see: show it as `repo , not on the install` with its project, and do not change it in this flow. + +An unmapped repo returns 404 from `get-release-autopilot-repo-settings`. That is the expected answer for a new repo, not an error. Release Autopilot's agents need generative AI turned on for the organization. There is no MCP tool to read that setting. If a later step fails with a generative AI or Vega eligibility error, tell the user that an Admin or Owner must select **Enable generative AI use cases** in Organization settings > Security, then stop. @@ -167,7 +169,7 @@ You cannot read inference settings, so record the answer as the user's statement Skip this step if the user did not pick auto-releasing. -Auto-releasing releases each change the way the environment is set up to release. On accounts with **environment release plans**, that is a plan: a reusable recipe for one environment, made of phases, rollout steps, and metrics. On accounts without them, it is the project's **release policies**. Either way, metrics make the release guarded, so a regression on a metric can roll it back. With neither, a release can fall back to defaults, which are often an immediate release. +Auto-releasing releases each change the way the environment is set up to release. Today that is the project's **release policies**, and auto-releasing requires a complete one for each environment. Accounts that have **environment release plans** use a plan instead: a reusable recipe for one environment, made of phases, rollout steps, and metrics. Release plans replace release policies by general availability. Either way, metrics make the release guarded, so a regression on a metric can roll it back. For each project from Step 4: @@ -186,14 +188,9 @@ For each project from Step 4: Record the plan. Do not call `create-release-plan` yet. -If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans, so this skill cannot create them. Do not retry. Auto-releasing can still be on for such an account, using the project's release policies instead. Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. - -If the policies have no warnings for the chosen environments, say that auto-releasing in this project uses those policies. If there is any warning, such as `missing_policy` or `incomplete_policy`, a release there can fall back to defaults, which are often immediate. Ask header "Release defaults": - -- label: "Keep auto-releasing off here (Recommended)", description: "Map the repo with auto-releasing off until the project has a complete release policy" -- label: "Accept the fallback", description: "Auto-release with the default behavior, which can be an immediate release to everyone" +If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans, so this skill cannot create them. The 403 is account-wide: do not retry, and do not call `list-release-plans` for the other environments. Auto-releasing can still be on for such an account, using the project's release policies instead. Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. -For "Keep auto-releasing off here", the repo mapping in Step 8 sends an auto-releasing override that turns it off. Release policies are edited in the project's release settings in the LaunchDarkly UI; this skill does not change them. +If the policies have no warnings for the chosen environments, say that auto-releasing in this project uses those policies. If there is any warning, such as `missing_policy` or `incomplete_policy`, auto-releasing cannot work there yet. Tell the user to add or complete a release policy in the project's release settings (`/projects/{projectKey}/settings/releases` on their LaunchDarkly instance); this skill does not change policies. Record a repo override that keeps auto-releasing off for that repo, and say in Step 7 that they can turn it on with the settings skill once the policy is complete. `list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so it needs a new complete plan in that environment. @@ -246,7 +243,7 @@ Report a short checklist with each item marked verified or reported by the user. | Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | | Repo already mapped to a different project | Do not remap here. The user unmaps it with the settings skill, then maps it in this flow | | Auto-releasing picked but no metrics exist | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | -| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Show `list-release-policies` and its warnings, create no plans, and keep auto-releasing off where a policy is missing unless the user accepts the fallback | +| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Show `list-release-policies` and its warnings, create no plans, and keep auto-releasing off for the repo where a policy is missing or incomplete | | Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | | User lacks permission for a write | Report the failed write and the missing permission. Do not retry with other settings | | Bring your own key chosen | Browser step only. Record it as reported, never as verified | From 56fc0154e7cb9aa6285219456d79caad59e73d65 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:48:02 -0700 Subject: [PATCH 07/30] docs(changelog): note release policies are required until release plans replace them by GA Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac831d99..49e5c7a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this repository will be documented in this file. ## Unreleased -- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.1.0-experimental: a step-by-step wizard that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, falling back to release policies where plans are not available. One confirmed plan lists every write before any change. +- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.1.0-experimental: a step-by-step wizard that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. From ea1b629f84963ae6933786af3379703e3dc10c05 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 16:48:13 -0700 Subject: [PATCH 08/30] fix(release-autopilot-onboarding): send BYOK users to the project Vega settings page only Co-Authored-By: Claude Opus 5.5 (1M context) --- .../launchdarkly-release-autopilot-onboarding/SKILL.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 43bd72b9..020842f6 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -155,8 +155,11 @@ For "LaunchDarkly managed", go to Step 6. For "Bring your own key", there is no MCP tool for inference settings yet, so this is a browser step: -1. The key lives in the account's integration settings. Tell the user to add an Anthropic or OpenAI key there. -2. The project must then use it. For each project from Step 4, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL) and tell them to select their own key and save. LaunchDarkly checks the key with the provider when they save, and shows the cause if the check fails. +For each project from Step 4, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). On that page they: + +1. Select bring your own key. If the account has no Anthropic or OpenAI key yet, the page links to the integration form to add one. Keys are account-wide, so later projects can reuse them. +2. Keep or change the suggested default and small models. +3. Save. LaunchDarkly checks the key with the provider before it stores the settings, and shows the cause if the check fails. Then ask header "Inference": From eaa38a7fd7d0ca3fcfd01b6ee92e5a159d49e55f Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 17:03:13 -0700 Subject: [PATCH 09/30] feat(release-autopilot-onboarding): preview and resume modes, permission check, fewer questions Add Preview mode (no writes; ends with an admin checklist) and Resume mode (re-read live state, continue at the first unfinished step). Check the member role up front and fall back to Preview for readers. Skip questions the account already answers. Detect the release mode once (policy today, plans by GA). Add voice rules, and move plan creation, edge cases, and examples to references/. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- skills.json | 4 +- .../SKILL.md | 180 +++++++++--------- .../marketplace.json | 2 +- .../references/edge-cases.md | 32 ++++ .../references/release-plans.md | 21 ++ 6 files changed, 145 insertions(+), 96 deletions(-) create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 49e5c7a6..1c81a95b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this repository will be documented in this file. ## Unreleased -- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.1.0-experimental: a step-by-step wizard that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. +- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. diff --git a/skills.json b/skills.json index 1272d824..fc5a6980 100644 --- a/skills.json +++ b/skills.json @@ -421,9 +421,9 @@ }, { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review or set up how each environment releases (environment release plans, or release policies where plans are not available), then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review how each environment releases (release policies today, environment release plans by GA), then verify. Also previews the changes without making them, and resumes a setup that paused. Use when the user wants to set up, onboard to, get started with, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", - "version": "0.1.0-experimental", + "version": "0.2.0-experimental", "license": "Apache-2.0", "compatibility": "Requires the remotely hosted LaunchDarkly MCP server", "tags": [ diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 020842f6..75dad429 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -1,16 +1,16 @@ --- name: launchdarkly-release-autopilot-onboarding -description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review or set up how each environment releases (environment release plans, or release policies where plans are not available), then verify. Use when the user wants to set up, onboard to, get started with, or turn on Release Autopilot for the first time. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review how each environment releases (release policies today, environment release plans by GA), then verify. Also previews the changes without making them, and resumes a setup that paused. Use when the user wants to set up, onboard to, get started with, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly - version: "0.1.0-experimental" + version: "0.2.0-experimental" --- # LaunchDarkly Release Autopilot Onboarding -You're using a skill that walks a user through setting up Release Autopilot from nothing to a working setup. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and how each environment releases. +You're using a skill that walks a user through setting up Release Autopilot. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and the environment release setup for auto-releasing. **This is a step-by-step wizard.** Ask one question at a time and wait for each answer. Gather every decision first, confirm one plan, and only then write. Release Autopilot opens and changes pull requests, and auto-releasing can ship code, so every write is production automation. @@ -24,30 +24,46 @@ This skill requires the remotely hosted LaunchDarkly MCP server. - `get-release-autopilot-settings` / `update-release-autopilot-settings` - `list-release-autopilot-github-repos` - `list-release-autopilot-repo-settings` / `get-release-autopilot-repo-settings` / `update-release-autopilot-repo-settings` +- `get-member-self` **Required for auto-releasing:** - `list-release-policies` -- `list-metrics` (plans and policies can only attach metrics that exist) +- `list-metrics` -**Used for auto-releasing when available:** -- `list-release-plans` / `create-release-plan` - -**Optional MCP tools:** +**Used when available:** +- `list-release-plans` / `create-release-plan`: environment release plans (see [Detect the release mode](#detect-the-release-mode)) - `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. -If `list-release-plans` or `create-release-plan` is missing, use the release policy path in Step 6, the same as for a 403. - Release Autopilot is in alpha. If every Release Autopilot tool returns 404, the account is not in the alpha. Tell the user and stop. +## Modes + +Pick the mode from the user's request before Step 0: + +| Mode | Trigger | What changes | +|------|---------|--------------| +| **Setup** | "set up", "onboard", "turn on" | The full flow | +| **Preview** | "preview", "what would you change", "dry run", or a member who cannot write (see Step 1) | Run Steps 1–7, show the plan, and stop. Make no writes. End with the plan as a checklist the user can hand to an admin | +| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Re-read live state in Step 1, say in one sentence what is already done, and continue at the first step that is not done. Never re-ask a decision the live state already answers | + +Every mode is safe to run again. A re-run reads live state and proposes only the writes that are still needed. + ## Core Principles -1. **Only set up what they picked.** Release plans and policies matter only for auto-releasing. If the user picks auto-flagging only, skip Step 6. +1. **Only set up what they picked.** Release setup matters only for auto-releasing. If the user picks auto-flagging only, skip Step 6. 2. **Least privilege by default.** Recommend auto-flagging with approval required. Auto-releasing is opt-in and you must say it can ship code. 3. **Gather, confirm, then write.** No write happens before [Step 7](#step-7-confirm-the-plan). The confirmation lists every write with its current and proposed state. -4. **Never act in the user's browser.** GitHub install, GitHub authorization, and bring-your-own-key setup are browser steps. Give the exact link and wait for the user to say they finished. -5. **Only touch what the user named.** Never map every repo on the install. Never create release plans in environments the user did not pick. +4. **Don't ask what the account already answers.** If live state already settles a question, state the answer and move on. +5. **Never act in the user's browser.** GitHub install, GitHub authorization, release policy edits, and bring-your-own-key setup are browser steps. Give the exact link and wait for the user to say they finished. +6. **Only touch what the user named.** Never map every repo on the install. Never create release plans in environments the user did not pick. + +## Speaking to the user + +- Use product words: "auto-flagging", "auto-releasing", "repository", "project", "release policy", "release plan", "approval". +- Never show tool names or field names such as `githubRepoId`, `enabledOverride`, or `autoReleasing.enabled`. Say "repository ID", "turned off for this repository", or "auto-releasing is on". +- Keep each message to the current step. Do not narrate tool calls. ## Workflow @@ -61,7 +77,7 @@ Release Autopilot is in alpha. If every Release Autopilot tool returns 404, the ### Step 0: Welcome the user -Present this intro (you may lightly adapt the wording, but keep the structure): +In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." In Resume mode, skip the intro. > **Release Autopilot Setup** > @@ -72,31 +88,30 @@ Present this intro (you may lightly adapt the wording, but keep the structure): > 3. **Connect GitHub**: install or authorize the LaunchDarkly GitHub App. > 4. **Map repositories**: choose which repos Release Autopilot watches, and which LaunchDarkly project each one uses. > 5. **Choose inference**: run the agents on LaunchDarkly managed inference or on your own model provider key. -> 6. **Review release setup** (auto-releasing only): check or define how each environment rolls out a change. +> 6. **Review release setup** (auto-releasing only): check how each environment rolls out a change. > 7. **Confirm the plan**: I'll show every change before I make it. > 8. **Apply and verify**: I'll make the changes and check the result. > > Nothing changes until you approve the plan in step 7. -Then go to Step 1. +Then go to Step 1. Skip any step that the account state already completes, and say why in one line. ### Step 1: Read current state Run these reads. Do not write anything. -1. `get-release-autopilot-settings`: the account gates. -2. `list-release-autopilot-repo-settings`: repos that are already mapped. -3. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. - -`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. A mapped `githubRepoId` with no match on the install list is a repo the App can no longer see, or one this member cannot see: show it as `repo , not on the install` with its project, and do not change it in this flow. +1. `get-member-self`: the member's `role`. If it is `reader` or `no_access`, switch to Preview mode and say why. Any other role may still lack a specific permission; Step 8 reports that if it happens. +2. `get-release-autopilot-settings`: the account gates. +3. `list-release-autopilot-repo-settings`: repos that are already mapped. +4. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. -An unmapped repo returns 404 from `get-release-autopilot-repo-settings`. That is the expected answer for a new repo, not an error. +`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. A mapped `githubRepoId` with no match on the install list is a repo the App can no longer see, or one this member cannot see: show it as "repository ID , not on the GitHub install" with its project, and do not change it in this flow. Release Autopilot's agents need generative AI turned on for the organization. There is no MCP tool to read that setting. If a later step fails with a generative AI or Vega eligibility error, tell the user that an Admin or Owner must select **Enable generative AI use cases** in Organization settings > Security, then stop. Summarize what is already set up in two or three sentences. If the account already has a capability on or any repo mapped, ask: -- label: "Add more", description: "Keep the current setup and add capabilities, repos, or release plans" +- label: "Add more", description: "Keep the current setup and add capabilities, repos, or release setup" - label: "Change one setting", description: "Hand off to the settings skill for a single change" - label: "Diagnose a PR", description: "A PR was not classified or auto-flagged. Hand off to the settings skill" @@ -107,9 +122,9 @@ For "Change one setting" or "Diagnose a PR", stop this skill and follow [`launch Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: - label: "Auto-flagging (Recommended)", description: "Release Autopilot reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" -- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with the environment's release plan or release policy. This can ship code to users" +- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with the environment's release policy. This can ship code to users" -If the user picks auto-flagging, ask header "Approval": +Ask about approval only if the user picked auto-flagging **and** the account does not already have auto-flagging on with approval required. If it already does, say "Auto-flagging already requires approval for every mapped repo" and move on. Otherwise ask header "Approval": - label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before Release Autopilot changes code. This is an account setting, so it applies to every mapped repo" - label: "No approval", description: "Release Autopilot changes PRs without an approval step, in every mapped repo" @@ -120,7 +135,9 @@ If the user picks neither capability, stop and tell them there is nothing to set ### Step 3: Connect GitHub -If Step 1 returned a `githubConnectUrl`, read its `flow`: +Skip this step without mentioning GitHub if Step 1 returned `installed: true`, `userAuthorized: true`, and no `githubConnectUrl`. + +Otherwise read the URL's `flow`: - `flow=install`: the LaunchDarkly GitHub App is not installed on their GitHub organization. Release Autopilot cannot see any pull request until it is. - `flow=authorize`: the App is installed, but this member has not authorized GitHub, so repos cannot be listed. @@ -130,15 +147,15 @@ Give the user the exact URL. Tell them to open it in a browser where they are lo - label: "Done", description: "I finished the GitHub step" - label: "Need an org owner", description: "I can't install the App myself. Pause setup here" -On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, say so and repeat this step. On "Need an org owner", summarize the decisions so far and stop. The user can run this skill again after the App is installed. - -If Step 1 returned repos and no `githubConnectUrl`, skip this step. +On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, say so and repeat this step. On "Need an org owner", summarize the decisions so far, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. ### Step 4: Map repositories -1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. If a repo the user wants is not on the list, compare its owner to the owners on the list. If the owner is on the list, the App lacks access to that repo: tell the user to add it to the App's repository access in GitHub. If the owner is not on the list, the App is not installed on that GitHub organization, and `githubConnectUrl` will not appear because the account already has an install elsewhere: tell the user that an owner of that organization must install the LaunchDarkly GitHub App on it. Either way, wait for "Done" and list again. +1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. + + If a repo the user wants is not on the list, compare its owner to the owners on the list. Same owner: the App lacks access to that repo, so the user adds it to the App's repository access in GitHub. New owner: the App is not installed on that GitHub organization, and no `githubConnectUrl` appears because the account already has an install elsewhere, so an owner of that organization must install the LaunchDarkly GitHub App on it. Either way, wait for "Done" and list again. 2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. -3. **Check existing mappings** with the `githubRepoId` join from Step 1. +3. **Check existing mappings** with the join from Step 1. An unmapped repo returns 404 from `get-release-autopilot-repo-settings`; that is the expected answer for a new repo, not an error. - Mapped to the chosen project, and its effective capabilities match the user's picks: tell the user it is already set up, and write nothing. - Mapped to the chosen project, but a capability the user did not pick is effectively on: record a repo override that turns it off, as its own line in Step 7. - Mapped to a different project: do not remap it in this flow. Remapping changes where its flags are created. Tell the user to use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) to unmap it first, then run this step again. @@ -151,11 +168,7 @@ Release Autopilot runs its agents on Vega. Bring your own key is optional: a pro - label: "LaunchDarkly managed", description: "The default if you choose nothing. The agents run on Vega inference that LaunchDarkly hosts. No setup" - label: "Bring your own key", description: "The agents run on your own Anthropic or OpenAI account. Your provider bills you for usage" -For "LaunchDarkly managed", go to Step 6. - -For "Bring your own key", there is no MCP tool for inference settings yet, so this is a browser step: - -For each project from Step 4, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). On that page they: +For "Bring your own key", there is no MCP tool for inference settings yet, so this is a browser step. For each project from Step 4, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). On that page they: 1. Select bring your own key. If the account has no Anthropic or OpenAI key yet, the page links to the integration form to add one. Keys are account-wide, so later projects can reuse them. 2. Keep or change the suggested default and small models. @@ -172,30 +185,30 @@ You cannot read inference settings, so record the answer as the user's statement Skip this step if the user did not pick auto-releasing. -Auto-releasing releases each change the way the environment is set up to release. Today that is the project's **release policies**, and auto-releasing requires a complete one for each environment. Accounts that have **environment release plans** use a plan instead: a reusable recipe for one environment, made of phases, rollout steps, and metrics. Release plans replace release policies by general availability. Either way, metrics make the release guarded, so a regression on a metric can roll it back. +For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then detect the release mode once per account and follow that mode. + +#### Detect the release mode + +Call `list-release-plans` for the first chosen environment. + +- **Policy mode** if the call returns 403 `Plan does not allow this operation`, or if `list-release-plans` is not available. The 403 is account-wide, so do not call it for other environments. This is the mode for every account today: auto-releasing requires a complete release policy for each environment. +- **Plan mode** if the call succeeds. Environment release plans replace release policies by general availability. -For each project from Step 4: +#### Policy mode -1. **Choose environments.** Use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Check each environment you pick. -2. **Check existing plans.** For each chosen environment, call `list-release-plans`. If a complete plan exists, show its name, phases, and metrics, and ask header "Plan": - - label: "Use this plan", description: "Keep the existing plan for this environment" - - label: "Create another", description: "Add a new plan for this environment" -3. **Create a plan where needed.** Follow the `create-release-plan` tool guidance. Ask each setting as its own question, never bundled: - 1. Randomization unit: the context kind to roll out by, such as `user` or `account`. - 2. Rollout steps: offer header "Rollout" with options such as "1% → 10% → 50% → 100%, 4h holds", or "10% → 100%, 24h hold". The user can type another schedule in Other. - 3. Advancement: header "Advance", options "Automatic", description "Move to the next step when the hold ends and metrics are healthy", and "Manual", description "A person moves each step". - 4. Metrics: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create. Do not create them yet: Step 8 creates them in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md) before the release plans that use them. This skill creates the metric only. If the event is not sent by the app yet, list instrumenting it as a follow-up for the user; do not edit code or `.env` files in this flow. A plan without metrics is progressive, not guarded: say so. - 5. Cleanup: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. +Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. This skill does not change policies. - Every phase needs at least one clause. For a phase that targets everyone, use `{ attribute: "kind", op: "in", values: [""], contextKind: "" }`. +- No warnings for the chosen environments: say that auto-releasing in this project uses those policies. +- Any warning, such as `missing_policy` or `incomplete_policy`: auto-releasing cannot work there yet. Tell the user to add or complete a release policy in the project's release settings (`/projects/{projectKey}/settings/releases` on their LaunchDarkly instance). Record a repo override that keeps auto-releasing off for each repo in that project, and say in Step 7 that they can turn it on with the settings skill once the policy is complete. - Record the plan. Do not call `create-release-plan` yet. +#### Plan mode -If `list-release-plans` returns 403 with `Plan does not allow this operation`, the account's LaunchDarkly plan does not include environment release plans, so this skill cannot create them. The 403 is account-wide: do not retry, and do not call `list-release-plans` for the other environments. Auto-releasing can still be on for such an account, using the project's release policies instead. Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. +For each chosen environment, call `list-release-plans`. If a complete plan exists, show its name, phases, and metrics, and ask header "Plan": -If the policies have no warnings for the chosen environments, say that auto-releasing in this project uses those policies. If there is any warning, such as `missing_policy` or `incomplete_policy`, auto-releasing cannot work there yet. Tell the user to add or complete a release policy in the project's release settings (`/projects/{projectKey}/settings/releases` on their LaunchDarkly instance); this skill does not change policies. Record a repo override that keeps auto-releasing off for that repo, and say in Step 7 that they can turn it on with the settings skill once the policy is complete. +- label: "Use this plan", description: "Keep the existing plan for this environment" +- label: "Create another", description: "Add a new plan for this environment" -`list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so it needs a new complete plan in that environment. +To create a plan, follow [references/release-plans.md](references/release-plans.md). Record the plan and any metrics to create. Do not call `create-release-plan` or create metrics yet. ### Step 7: Confirm the plan @@ -203,12 +216,17 @@ If the policies have no warnings for the chosen environments, say that auto-rele List every write in the order you will make it. For each write, state the current and the proposed state and the blast radius, using the wording rules from the settings skill's [Confirm before write](../launchdarkly-release-autopilot-settings/SKILL.md#confirm-before-write): -1. **Metrics and release plans**: one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. -2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, the mapping sends a repo override that turns it off, so the repo cannot inherit it from the account. Say so: "Auto-releasing: OFF (repo override; the account has it ON)." -3. **Account settings**: for example, "Auto-flagging: OFF → ON, approval required. Mapped repos that inherit this setting can start getting auto-flagging PRs." If auto-releasing turns on, say: "Auto-releasing: OFF → ON. This can ship code." +1. **Metrics and release plans** (plan mode only): one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. +2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." +3. **Account settings**: only fields that change. For example, "Auto-flagging: off → on, approval required. Mapped repos that inherit this setting can start getting auto-flagging PRs." If auto-releasing turns on, say: "Auto-releasing: off → on. This can ship code." 4. **Browser steps the user reported done**: GitHub and inference, marked as reported, not verified. +5. **Follow-ups for the user**: release policies to add, metric events to instrument. -Then use `AskUserQuestion` with header "Confirm": +If there are no writes, say so, list the follow-ups, and stop. + +**Preview mode:** present the list as a checklist under the heading "Changes an admin would make", then stop. Do not ask for confirmation. + +**Setup mode:** use `AskUserQuestion` with header "Confirm": - label: "Looks good", description: "Make every change in this plan" - label: "Change something", description: "Adjust the plan before you start" @@ -219,62 +237,40 @@ On "Change something", make the change and show the full plan again. Do not go t Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. -1. Metrics recorded in Step 6, created in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). Skip its instrumentation steps. -2. `create-release-plan` for each new plan. -3. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. For each capability the user did not pick, send `{ "enabled": false }` for it, so the repo does not inherit it from the account. +1. Plan mode only: metrics recorded in Step 6, created in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). Skip its instrumentation steps. +2. Plan mode only: `create-release-plan` for each new plan. +3. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. For each capability the user did not pick, or that cannot work yet, send `{ "enabled": false }` for it, so the repo does not inherit it from the account. 4. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. -The order matters. When the plan creates release plans, they come first, so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. +The order matters. Release plans come first so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. + +If a write fails with 403, the member lacks that permission. Report which write failed, list the remaining writes as a checklist for an admin, and stop. ### Step 9: Verify 1. `get-release-autopilot-settings`: the account gates match the plan. 2. `get-release-autopilot-repo-settings` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. -3. If plans were created, `list-release-plans` for each environment: the new plans are listed. On the release policy path, report the policy state from Step 6 instead. +3. Plan mode: `list-release-plans` for each environment lists the new plans. Policy mode: report the policy state from Step 6. -Report a short checklist with each item marked verified or reported by the user. Then tell the user how to see it work: +Report a short checklist with each item marked verified or reported by the user, then the follow-ups. Then tell the user how to see it work: - Open a pull request in a mapped repo. Release Autopilot reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. - If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. -## Edge Cases - -| Situation | Action | -|-----------|--------| -| Every Release Autopilot tool returns 404 | The account is not in the alpha. **STOP** and tell the user | -| `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | -| Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | -| Repo already mapped to a different project | Do not remap here. The user unmaps it with the settings skill, then maps it in this flow | -| Auto-releasing picked but no metrics exist | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | -| `list-release-plans` returns 403 `Plan does not allow this operation` | The account's plan lacks environment release plans. Show `list-release-policies` and its warnings, create no plans, and keep auto-releasing off for the repo where a policy is missing or incomplete | -| Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | -| User lacks permission for a write | Report the failed write and the missing permission. Do not retry with other settings | -| Bring your own key chosen | Browser step only. Record it as reported, never as verified | - ## What NOT to Do -- Don't write before Step 7's "Looks good". +- Don't write before Step 7's "Looks good", and never write in Preview mode. - Don't turn on auto-releasing unless the user picked it. - Don't turn off approval for auto-flagging unless the user picked "No approval". -- Don't map repos the user did not pick, or iterate over the install list. -- Don't install or authorize GitHub, or set up a provider key, on the user's behalf. +- Don't map repos the user did not pick, iterate over the install list, or remap a repo to another project. +- Don't install or authorize GitHub, change release policies, or set up a provider key on the user's behalf. - Don't send `autoCleanup`. - Don't report a browser step as verified. ## References +- [references/release-plans.md](references/release-plans.md): creating environment release plans (plan mode) +- [references/edge-cases.md](references/edge-cases.md): edge cases and example flows - [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md): settings rules, write confirmations, and PR diagnosis -- [`launchdarkly-metric-choose`](../../metrics/launchdarkly-metric-choose/SKILL.md) / [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md): metrics for guarded release plans - [Vega inference settings](https://launchdarkly.com/docs/home/getting-started/vega#vega-inference-settings) - [Guarded rollouts](https://launchdarkly.com/docs/home/releases/guarded-rollouts) - -## Example Flows - -**Happy path: auto-flagging only, managed inference.** -Step 1 → nothing set up, install list has repos. Step 2 → auto-flagging, approval required. Step 3 → skipped. Step 4 → maps `acme/web` to project `web`. Step 5 → managed. Step 6 → skipped. Step 7 → two writes: map `acme/web` with auto-releasing OFF (repo override), then account auto-flagging ON with approval. Step 8 → applies. Step 9 → verified. - -**Both capabilities, no GitHub App yet.** -Step 1 → `githubConnectUrl` with `flow=install`. Step 2 → both capabilities. Step 3 → user installs the App, selects "Done", repos appear. Step 4 → maps `acme/api` to `api`. Step 5 → bring your own key, user saves settings in the browser. Step 6 → `production` has no plan; creates `prod-guarded` with 1% → 10% → 50% → 100%, automatic advancement, metric `api-error-rate`. Step 7 → plan, mapping, and account writes, plus two reported browser steps. Steps 8–9 → applied and verified. - -**Org owner needed: pause.** -Step 3 → `flow=install`, user selects "Need an org owner". Summarize decisions so far and stop. Nothing was written. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json index c752b68b..cec8016f 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json @@ -1,7 +1,7 @@ { "name": "launchdarkly-release-autopilot-onboarding", "description": "Step-by-step Release Autopilot onboarding: pick capabilities, connect GitHub, map repos, choose inference, review release setup", - "version": "0.1.0-experimental", + "version": "0.2.0-experimental", "author": "LaunchDarkly", "repository": "https://github.com/launchdarkly/ai-tooling", "skills": ["./"], diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md new file mode 100644 index 00000000..266e99f0 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -0,0 +1,32 @@ +# Edge cases and example flows + +## Edge cases + +| Situation | Action | +|-----------|--------| +| Every Release Autopilot tool returns 404 | The account is not in the alpha. Stop and tell the user | +| Member role is `reader` or `no_access` | Preview mode. End with the plan as a checklist for an admin | +| A write returns 403 | Report which write failed, list the remaining writes for an admin, and stop | +| `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | +| Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | +| Mapped repo ID not on the install list | Show it by ID with its project. Do not change it | +| Repo already mapped to a different project | Do not remap here. The user unmaps it with the settings skill, then maps it in this flow | +| `list-release-plans` returns 403 `Plan does not allow this operation` | Policy mode. Show release policies and their warnings, create no plans, and keep auto-releasing off for the repo where a policy is missing or incomplete | +| Auto-releasing picked but no metrics exist (plan mode) | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | +| Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | +| Bring your own key chosen | Browser step only. Record it as reported, never as verified | +| No writes needed | Say so, list follow-ups, and stop | + +## Example flows + +**Fresh account: auto-flagging only, managed inference.** +Step 1 → admin, nothing set up, install list has repos. Step 2 → auto-flagging, approval required. Step 3 → skipped silently. Step 4 → maps `acme/web` to project `web`. Step 5 → managed. Step 6 → skipped. Step 7 → two writes: map `acme/web` with auto-releasing off for the repo, then account auto-flagging on with approval. Steps 8–9 → applied and verified. + +**Account already on, policy mode, missing policy.** +Step 1 → both capabilities on, approval required, 3 repos mapped; "Add more". Step 2 → both capabilities; approval question skipped. Step 3 → skipped silently. Step 4 → maps `acme/demo` to `demo`. Step 5 → managed. Step 6 → release plans return 403, so policy mode; `missing_policy`. Step 7 → one write: map `acme/demo` with auto-flagging on and auto-releasing off for the repo; follow-up: add a release policy. Steps 8–9 → applied and verified. + +**Reader: preview only.** +Step 1 → role `reader`, so Preview mode. Steps 2–6 as usual. Step 7 → "Changes an admin would make" checklist. No writes. + +**Org owner needed: pause, then resume.** +Step 3 → `flow=install`, user selects "Need an org owner". Summarize and stop. Later, "continue Release Autopilot setup" → Resume mode: Step 1 shows the App installed, and the flow continues at Step 4 with the earlier picks. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md new file mode 100644 index 00000000..25c6a2d6 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md @@ -0,0 +1,21 @@ +# Create an environment release plan (plan mode) + +Use this from Step 6 of the onboarding skill when the account has environment release plans. A release plan is a reusable recipe for one environment, made of phases, rollout steps, and metrics. A plan with metrics makes a release guarded, so a regression on a metric can roll it back. A plan without metrics is progressive: say so. + +Follow the `create-release-plan` tool guidance. Ask each setting as its own question, never bundled: + +1. **Randomization unit**: the context kind to roll out by, such as `user` or `account`. +2. **Rollout steps**: header "Rollout", with options such as "1% → 10% → 50% → 100%, 4h holds" or "10% → 100%, 24h hold". The user can type another schedule in Other. +3. **Advancement**: header "Advance", options "Automatic", description "Move to the next step when the hold ends and metrics are healthy", and "Manual", description "A person moves each step". +4. **Metrics**: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create. Step 8 creates them in LaunchDarkly before the plans that use them. This flow creates the metric only: if the app does not send the event yet, list instrumenting it as a follow-up, and do not edit code or `.env` files. +5. **Cleanup**: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. + +Every phase needs at least one clause. For a phase that targets everyone, use: + +```json +{ "attribute": "kind", "op": "in", "values": [""], "contextKind": "" } +``` + +`list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so that environment needs a new complete plan. + +Always list before creating, so the new key does not collide with an active plan. From 14c72b2a4562cadd4da2a93cea6140e879ac436d Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 17:07:28 -0700 Subject: [PATCH 10/30] fix(release-autopilot-onboarding): address improvement review #2 Let users change approval even when the account already requires it; leave account writes to admins and owners; keep the mode and rebuild picks on Resume; drop the writing hand-off in Preview; restore the conditional plan-ordering wording; shorten the uninstalled-org guidance. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../SKILL.md | 18 +++++++++++------- .../references/edge-cases.md | 1 + 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 75dad429..83cf2949 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -46,7 +46,7 @@ Pick the mode from the user's request before Step 0: |------|---------|--------------| | **Setup** | "set up", "onboard", "turn on" | The full flow | | **Preview** | "preview", "what would you change", "dry run", or a member who cannot write (see Step 1) | Run Steps 1–7, show the plan, and stop. Make no writes. End with the plan as a checklist the user can hand to an admin | -| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Re-read live state in Step 1, say in one sentence what is already done, and continue at the first step that is not done. Never re-ask a decision the live state already answers | +| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Keep the earlier mode: a resumed Preview stays Preview. Re-read live state in Step 1 and say in one sentence what is already done. Use the earlier picks if they are still in the conversation; if not, ask them again with live state as the default. Writes that landed before a Step 8 failure show as already set up, so the new plan lists only what is left | Every mode is safe to run again. A re-run reads live state and proposes only the writes that are still needed. @@ -100,7 +100,10 @@ Then go to Step 1. Skip any step that the account state already completes, and s Run these reads. Do not write anything. -1. `get-member-self`: the member's `role`. If it is `reader` or `no_access`, switch to Preview mode and say why. Any other role may still lack a specific permission; Step 8 reports that if it happens. +1. `get-member-self`: the member's `role`. + - `reader` or `no_access`: switch to Preview mode and say why. + - `admin` or `owner`: can make every write in this flow. + - Any other role, including `writer` and custom roles: may map repos in projects they can edit, but account settings need an admin or owner. In Step 7, mark account writes as "needs an admin" and leave them out of Step 8. 2. `get-release-autopilot-settings`: the account gates. 3. `list-release-autopilot-repo-settings`: repos that are already mapped. 4. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. @@ -115,16 +118,16 @@ Summarize what is already set up in two or three sentences. If the account alrea - label: "Change one setting", description: "Hand off to the settings skill for a single change" - label: "Diagnose a PR", description: "A PR was not classified or auto-flagged. Hand off to the settings skill" -For "Change one setting" or "Diagnose a PR", stop this skill and follow [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). +In Preview mode, leave out "Change one setting", because that hand-off writes. For "Change one setting" or "Diagnose a PR", stop this skill and follow [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). ### Step 2: Pick capabilities Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: - label: "Auto-flagging (Recommended)", description: "Release Autopilot reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" -- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with the environment's release policy. This can ship code to users" +- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with the environment's release policy or release plan. This can ship code to users" -Ask about approval only if the user picked auto-flagging **and** the account does not already have auto-flagging on with approval required. If it already does, say "Auto-flagging already requires approval for every mapped repo" and move on. Otherwise ask header "Approval": +Ask about approval only if the user picked auto-flagging **and** the account does not already have auto-flagging on with approval required. If it already does, say "Auto-flagging already requires approval for every mapped repo. Say 'change approval' if you want to turn it off," and move on unless they say so. Otherwise, or if they ask to change it, ask header "Approval": - label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before Release Autopilot changes code. This is an account setting, so it applies to every mapped repo" - label: "No approval", description: "Release Autopilot changes PRs without an approval step, in every mapped repo" @@ -153,7 +156,7 @@ On "Done", call `list-release-autopilot-github-repos` again. If it still returns 1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. - If a repo the user wants is not on the list, compare its owner to the owners on the list. Same owner: the App lacks access to that repo, so the user adds it to the App's repository access in GitHub. New owner: the App is not installed on that GitHub organization, and no `githubConnectUrl` appears because the account already has an install elsewhere, so an owner of that organization must install the LaunchDarkly GitHub App on it. Either way, wait for "Done" and list again. + If a repo the user wants is not on the list, its owner decides the fix: a listed owner means the App needs access to that repo, and a new owner means an org owner must install the App there (no `githubConnectUrl` appears for this case). See [references/edge-cases.md](references/edge-cases.md). Wait for "Done" and list again. 2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. 3. **Check existing mappings** with the join from Step 1. An unmapped repo returns 404 from `get-release-autopilot-repo-settings`; that is the expected answer for a new repo, not an error. - Mapped to the chosen project, and its effective capabilities match the user's picks: tell the user it is already set up, and write nothing. @@ -219,6 +222,7 @@ List every write in the order you will make it. For each write, state the curren 1. **Metrics and release plans** (plan mode only): one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." 3. **Account settings**: only fields that change. For example, "Auto-flagging: off → on, approval required. Mapped repos that inherit this setting can start getting auto-flagging PRs." If auto-releasing turns on, say: "Auto-releasing: off → on. This can ship code." + For a member who is not an admin or owner, list account writes under "Needs an admin" instead. 4. **Browser steps the user reported done**: GitHub and inference, marked as reported, not verified. 5. **Follow-ups for the user**: release policies to add, metric events to instrument. @@ -242,7 +246,7 @@ Make only the writes in the confirmed plan, in this order. Stop at the first fai 3. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. For each capability the user did not pick, or that cannot work yet, send `{ "enabled": false }` for it, so the repo does not inherit it from the account. 4. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. -The order matters. Release plans come first so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. +The order matters. When the plan creates release plans, they come first, so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. If a write fails with 403, the member lacks that permission. Report which write failed, list the remaining writes as a checklist for an admin, and stop. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md index 266e99f0..ddafc444 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -6,6 +6,7 @@ |-----------|--------| | Every Release Autopilot tool returns 404 | The account is not in the alpha. Stop and tell the user | | Member role is `reader` or `no_access` | Preview mode. End with the plan as a checklist for an admin | +| Member role is `writer` or a custom role | Account writes are marked "needs an admin" and left out of Step 8; repo writes still run | | A write returns 403 | Report which write failed, list the remaining writes for an admin, and stop | | `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | | Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | From 7028caeb59faa14090dc3f6154df615761d66a12 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 17:07:28 -0700 Subject: [PATCH 11/30] test(evals): add release-autopilot-onboarding suite Six cases lock in the onboarding write rules found in dry runs: confirm before writing, repo overrides for unpicked capabilities, account write last, one release-plans call in policy mode, no writes for readers or previews, and no account write for writers. The provider gains per-test mock_tool_overrides and label matching for scripted ask-question answers. Co-Authored-By: Claude Opus 5.5 (1M context) --- evals/README.md | 2 + evals/mocks/tool-responses.json | 12 +- evals/package.json | 2 + evals/providers/claude-skill-agent-sdk.js | 32 ++- .../assertions.js | 77 ++++++ .../promptfooconfig.yaml | 238 ++++++++++++++++++ evals/scripts/_manifest.js | 6 + evals/tests/provider-match-option.test.js | 27 ++ evals/tools/definitions.json | 12 +- 9 files changed, 403 insertions(+), 5 deletions(-) create mode 100644 evals/release-autopilot-onboarding/assertions.js create mode 100644 evals/release-autopilot-onboarding/promptfooconfig.yaml create mode 100644 evals/tests/provider-match-option.test.js diff --git a/evals/README.md b/evals/README.md index 603fc282..687c7006 100644 --- a/evals/README.md +++ b/evals/README.md @@ -34,6 +34,7 @@ npm run eval:agentcontrol-tools # agentcontrol/tools npm run eval:configs-variations # agentcontrol/configs-variations npm run eval:flag-create # feature-flags/launchdarkly-flag-create npm run eval:onboarding # onboarding +npm run eval:release-autopilot-onboarding # release-autopilot/launchdarkly-release-autopilot-onboarding # Quick smoke check — first test case only (~15-20s, ~$0.05) npm run eval:configs-create:single @@ -42,6 +43,7 @@ npm run eval:agentcontrol-tools:single npm run eval:configs-variations:single npm run eval:flag-create:single npm run eval:onboarding:single +npm run eval:release-autopilot-onboarding:single # Aggregate and CI operations npm run eval:all # Run every suite and rebuild ../eval-scores.json diff --git a/evals/mocks/tool-responses.json b/evals/mocks/tool-responses.json index b0669541..d616a93c 100644 --- a/evals/mocks/tool-responses.json +++ b/evals/mocks/tool-responses.json @@ -590,5 +590,15 @@ { "_id": "treat-treatment", "name": "Treatment", "baseline": false, "allocationPercent": 50 } ] } - } + }, + "get-member-self": {"id": "member-1", "email": "admin@acme.example", "role": "admin", "teams": []}, + "get-release-autopilot-settings": {"autoFlagging": {"enabled": true, "approvalRequired": true}, "autoReleasing": {"enabled": true}, "version": 9}, + "update-release-autopilot-settings": {"autoFlagging": {"enabled": true, "approvalRequired": true}, "autoReleasing": {"enabled": false}, "version": 10}, + "list-release-autopilot-github-repos": {"installed": true, "userAuthorized": true, "repositories": [{"githubRepoId": 101, "name": "acme/storefront", "isPublic": false}, {"githubRepoId": 102, "name": "acme/api", "isPublic": false}, {"githubRepoId": 103, "name": "acme/demo", "isPublic": true}], "totalCount": 3}, + "list-release-autopilot-repo-settings": {"repos": [{"githubRepoId": 101, "projectKey": "default", "autoFlagging": {"enabled": true, "approvalRequired": true, "enabledOverride": false}, "autoReleasing": {"enabled": true, "enabledOverride": false}}, {"githubRepoId": 999, "projectKey": "default", "autoFlagging": {"enabled": true, "approvalRequired": true, "enabledOverride": false}, "autoReleasing": {"enabled": true, "enabledOverride": false}}], "totalCount": 2}, + "get-release-autopilot-repo-settings": {"error": "LaunchDarkly API error: 404: Factory settings for this repository were not found"}, + "update-release-autopilot-repo-settings": {"projectKey": "{{projectKey}}", "autoFlagging": {"enabled": true, "approvalRequired": true}, "autoReleasing": {"enabled": false, "enabledOverride": true}, "version": 1}, + "delete-release-autopilot-repo-settings": {"deleted": true}, + "list-release-plans": {"error": "LaunchDarkly API error on GET /projects/{{projectKey}}/environments/{{environmentKey}}/release-plans: 403: {\"code\":\"forbidden\",\"message\":\"Plan does not allow this operation\"}"}, + "create-release-plan": {"key": "new-plan", "environmentKey": "{{environmentKey}}", "created": true} } diff --git a/evals/package.json b/evals/package.json index 5b16f174..5b1cbf79 100644 --- a/evals/package.json +++ b/evals/package.json @@ -25,6 +25,8 @@ "eval:flag-and-release-change:single": "promptfoo eval -c shared/defaults.yaml -c flag-and-release-change/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:onboarding": "promptfoo eval -c shared/defaults.yaml -c onboarding/promptfooconfig.yaml --env-file .env --no-cache -o onboarding/results.json", "eval:onboarding:single": "promptfoo eval -c shared/defaults.yaml -c onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", + "eval:release-autopilot-onboarding": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache -o release-autopilot-onboarding/results.json", + "eval:release-autopilot-onboarding:single": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:experiment-guide": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache -o experiment-guide/results.json", "eval:experiment-guide:single": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "fixture:from-pr": "node scripts/build-pr-fixture.js", diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 3adabd65..1e76385e 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -45,7 +45,13 @@ * (true/false), e.g. to run one test with no * LaunchDarkly tools * mock_ask_question_answers - optional array of `selected` arrays returned by - * successive `ask-question` calls + * successive `ask-question` calls. An entry that is + * not one of the call's option ids is matched + * against option labels (case-insensitive substring), + * so tests can steer by label when the agent picks ids + * mock_tool_overrides - optional object mapping tool name to a mock template + * that replaces mocks/tool-responses.json for this test, + * e.g. to return a 403 or a different account state * mock_files - optional object mapping relative file paths to * string content, written into the isolated cwd * before the agent runs @@ -75,6 +81,19 @@ const toolDefs = JSON.parse( fs.readFileSync(path.join(__dirname, "../tools/definitions.json"), "utf-8"), ); +/** + * Resolve a scripted answer to an option id. Exact id wins; otherwise the + * first option whose label contains the answer (case-insensitive). Unmatched + * answers pass through unchanged. + */ +function matchOption(want, options) { + const list = Array.isArray(options) ? options : []; + if (list.some((o) => o.id === want)) return want; + const needle = String(want).toLowerCase(); + const hit = list.find((o) => String(o.label || "").toLowerCase().includes(needle)); + return hit ? hit.id : want; +} + const mockTemplates = JSON.parse( fs.readFileSync(path.join(__dirname, "../mocks/tool-responses.json"), "utf-8"), ); @@ -233,6 +252,7 @@ class ClaudeSkillAgentSdk { ? context.vars.mock_ask_question_answers : []; const mockFiles = context?.vars?.mock_files || {}; + const mockToolOverrides = context?.vars?.mock_tool_overrides || {}; let userMessage = userRequest; if (codebaseContext) { @@ -262,7 +282,12 @@ class ClaudeSkillAgentSdk { def.description, inputSchemaToZodShape(def.input_schema), async (args) => { - const template = mockTemplates[def.name]; + const template = Object.prototype.hasOwnProperty.call( + mockToolOverrides, + def.name, + ) + ? mockToolOverrides[def.name] + : mockTemplates[def.name]; const mock = template ? renderMockResponse(template, args, def.name, mockState) : { error: `No mock configured for tool: ${def.name}` }; @@ -313,7 +338,7 @@ class ClaudeSkillAgentSdk { const override = askQuestionAnswers[idx]; let selected; if (Array.isArray(override) && override.length > 0) { - selected = override; + selected = override.map((want) => matchOption(want, args.options)); } else if (Array.isArray(args.options) && args.options.length > 0) { selected = [args.options[0].id]; } else { @@ -587,3 +612,4 @@ module.exports = ClaudeSkillAgentSdk; // Exposed for unit tests; the promptfoo loader uses the class default export. module.exports.stripRawWrapper = stripRawWrapper; module.exports.resolveExposeMcpTools = resolveExposeMcpTools; +module.exports.matchOption = matchOption; diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js new file mode 100644 index 00000000..8c8960bd --- /dev/null +++ b/evals/release-autopilot-onboarding/assertions.js @@ -0,0 +1,77 @@ +// Assertions for the Release Autopilot onboarding suite. Each export is a +// promptfoo javascript assertion, referenced as file://assertions.js:. + +const WRITE_TOOLS = new Set([ + "update-release-autopilot-settings", + "update-release-autopilot-repo-settings", + "delete-release-autopilot-repo-settings", + "create-release-plan", + "create-metric", +]); + +const traj = (output) => (output && output.trajectory) || []; +const writes = (output) => traj(output).filter((t) => WRITE_TOOLS.has(t.tool)); +const result = (pass, reason) => ({ pass, score: pass ? 1 : 0, reason }); + +function isConfirmQuestion(step) { + if (step.tool !== "ask-question") return false; + const labels = ((step.arguments && step.arguments.options) || []).map((o) => + String(o.label || "").toLowerCase(), + ); + return labels.some((l) => l.includes("looks good")); +} + +module.exports.noWrites = (output) => { + const w = writes(output); + return result(w.length === 0, w.length ? `wrote: ${w.map((t) => t.tool).join(", ")}` : "no writes"); +}; + +module.exports.confirmsBeforeWriting = (output) => { + const steps = traj(output); + const firstWrite = steps.findIndex((t) => WRITE_TOOLS.has(t.tool)); + if (firstWrite === -1) return result(false, "never wrote, so the confirmed plan was not applied"); + const confirm = steps.findIndex(isConfirmQuestion); + return result( + confirm !== -1 && confirm < firstWrite, + confirm === -1 ? "wrote without a confirm question" : `confirm at ${confirm}, first write at ${firstWrite}`, + ); +}; + +const repoOverridesUnpicked = (caps) => (output) => { + const maps = traj(output).filter((t) => t.tool === "update-release-autopilot-repo-settings"); + if (!maps.length) return result(false, "never mapped a repo"); + const bad = maps.filter((m) => caps.some((c) => !(m.arguments && m.arguments[c] && m.arguments[c].enabled === false))); + return result(bad.length === 0, bad.length ? `missing ${caps.join("/")} enabled:false on ${bad.length} mapping(s)` : "overrides sent"); +}; + +module.exports.accountWriteLast = (output) => { + const steps = traj(output); + const acct = steps.findIndex((t) => t.tool === "update-release-autopilot-settings"); + if (acct === -1) return result(false, "no account write"); + const laterWrite = steps.slice(acct + 1).some((t) => WRITE_TOOLS.has(t.tool)); + return result(!laterWrite, laterWrite ? "a write came after the account write" : "account write was last"); +}; + +module.exports.neverAutoReleasingOn = (output) => { + const bad = writes(output).filter((t) => t.arguments && t.arguments.autoReleasing && t.arguments.autoReleasing.enabled === true); + return result(bad.length === 0, bad.length ? `turned auto-releasing on via ${bad[0].tool}` : "auto-releasing never turned on"); +}; + +module.exports.noAutoCleanup = (output) => { + const bad = writes(output).filter((t) => t.arguments && Object.prototype.hasOwnProperty.call(t.arguments, "autoCleanup")); + return result(bad.length === 0, bad.length ? "sent autoCleanup" : "no autoCleanup"); +}; + +const noneOf = (tools) => (output) => { + const hit = traj(output).filter((t) => tools.includes(t.tool)); + return result(hit.length === 0, hit.length ? `called ${hit.map((t) => t.tool).join(", ")}` : `never called ${tools.join(", ")}`); +}; + +module.exports.releasePlansCalledAtMostOnce = (output) => { + const n = traj(output).filter((t) => t.tool === "list-release-plans").length; + return result(n <= 1, `list-release-plans called ${n} time(s)`); +}; + +module.exports.repoTurnsAutoReleasingOff = repoOverridesUnpicked(["autoReleasing"]); +module.exports.noReleasePlanCreated = noneOf(["create-release-plan"]); +module.exports.noAccountWrite = noneOf(["update-release-autopilot-settings"]); diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml new file mode 100644 index 00000000..e29f5447 --- /dev/null +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -0,0 +1,238 @@ +# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json +# +# Run with shared defaults: +# promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml +# +# The onboarding wizard for Release Autopilot. These cases lock in the write +# rules found in dry runs against catamorphic: nothing is written before the +# user confirms, Preview and reader runs never write, a repo never inherits a +# capability the user did not pick, the release-plans 403 is treated as +# account-wide policy mode, and account settings stay untouched when they +# already match. Default mocks match a catamorphic-like account: both +# capabilities on, the GitHub App installed, release plans forbidden by the +# account's plan. Tests change state with mock_tool_overrides, and steer +# ask-question by option label. +description: "Evaluation of the Release Autopilot onboarding skill" + +prompts: + - file://../../skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md + +providers: + - id: file://../providers/claude-skill-agent-sdk.js + label: mcp + config: + skill_slug: launchdarkly-release-autopilot-onboarding + allow_builtins: false + expose_ask_question: true + force_skill_invocation: true + +tests: + # ------------------------------------------------------------------ + # Fresh account, auto-flagging only. The repo mapping must turn + # auto-releasing off for the repo, the account write comes after the + # repo write, and no write happens before the confirm question. + # ------------------------------------------------------------------ + - description: "Fresh account, auto-flagging only: confirm first, repo override off, account write last" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 25 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Require approval"] + - ["acme/demo"] + - ["default"] + - ["managed"] + - ["Looks good"] + assert: + - type: javascript + value: file://assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 4 + - type: javascript + value: file://assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 4 + - type: javascript + value: file://assertions.js:accountWriteLast + metric: account_write_last + weight: 2 + - type: javascript + value: file://assertions.js:neverAutoReleasingOn + metric: no_auto_releasing + weight: 3 + - type: javascript + value: file://assertions.js:noAutoCleanup + metric: no_auto_cleanup + weight: 2 + + # ------------------------------------------------------------------ + # Account already on, both capabilities picked, policy mode with no + # release policy. list-release-plans is called once (the 403 is + # account-wide), no plans are created, the repo keeps auto-releasing + # off, and the account settings are not rewritten. + # ------------------------------------------------------------------ + - description: "Policy mode, missing policy: one plans call, no plans, auto-releasing off for the repo, no account write" + vars: + user_request: "Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default." + max_turns: 25 + mock_tool_overrides: + list-release-policies: + policies: [] + count: 0 + warnings: + - code: missing_policy + message: "No release policy matches." + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging", "Auto-releasing"] + - ["acme/demo"] + - ["default"] + - ["managed"] + - ["Test", "Production"] + - ["Looks good"] + assert: + - type: javascript + value: file://assertions.js:releasePlansCalledAtMostOnce + metric: plans_403_account_wide + weight: 3 + - type: javascript + value: file://assertions.js:noReleasePlanCreated + metric: no_plans_created + weight: 3 + - type: javascript + value: file://assertions.js:repoTurnsAutoReleasingOff + metric: auto_releasing_off_without_policy + weight: 4 + - type: javascript + value: file://assertions.js:noAccountWrite + metric: no_redundant_account_write + weight: 3 + - type: javascript + value: file://assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + + # ------------------------------------------------------------------ + # Reader: Preview mode is forced, so nothing is written even after + # every question is answered. + # ------------------------------------------------------------------ + - description: "Reader role: forced Preview, zero writes" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 20 + mock_tool_overrides: + get-member-self: { id: "member-2", email: "reader@acme.example", role: "reader", teams: [] } + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["acme/demo"] + - ["default"] + - ["managed"] + - ["Looks good"] + assert: + - type: javascript + value: file://assertions.js:noWrites + metric: no_writes + weight: 5 + + # ------------------------------------------------------------------ + # Explicit preview request from an admin: zero writes, and the final + # message is a checklist of changes. + # ------------------------------------------------------------------ + - description: "Explicit preview: zero writes, ends with the planned changes" + vars: + user_request: "Do a dry run: preview what you'd change to set up Release Autopilot auto-flagging for acme/demo in project default. Don't change anything." + max_turns: 20 + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["acme/demo"] + - ["default"] + - ["managed"] + assert: + - type: javascript + value: file://assertions.js:noWrites + metric: no_writes + weight: 5 + - type: llm-rubric + value: | + The final message lists the change that would be made: mapping acme/demo to project default, + with auto-releasing turned off for that repository. It says nothing was changed. It does not + show raw field names such as githubRepoId, enabledOverride, or autoReleasing.enabled. + metric: preview_plan + weight: 3 + + # ------------------------------------------------------------------ + # GitHub App not installed, and the user needs an org owner: hand the + # user the link, write nothing, and stop. + # ------------------------------------------------------------------ + - description: "App not installed, org owner needed: link given, zero writes" + vars: + user_request: "Set up Release Autopilot for my repos." + max_turns: 15 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + list-release-autopilot-github-repos: + installed: false + userAuthorized: false + repositories: [] + totalCount: 0 + githubConnectUrl: "https://app.launchdarkly.com/integrations/github/connect?flow=install" + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Require approval"] + - ["Need an org owner"] + assert: + - type: javascript + value: file://assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: | + const text = String(output.response || '') + (output.assistant_turns || []).join('\n'); + const pass = text.includes('flow=install'); + return { pass, score: pass ? 1 : 0, reason: pass ? 'gave the connect link' : 'did not give the githubConnectUrl' }; + metric: gave_connect_link + weight: 3 + + # ------------------------------------------------------------------ + # Writer role on a fresh account: account settings need an admin, so + # the account write is never attempted. The repo write may run. + # ------------------------------------------------------------------ + - description: "Writer role: account write left for an admin" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." + max_turns: 25 + mock_tool_overrides: + get-member-self: { id: "member-3", email: "writer@acme.example", role: "writer", teams: [] } + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Require approval"] + - ["acme/demo"] + - ["default"] + - ["managed"] + - ["Looks good"] + assert: + - type: javascript + value: file://assertions.js:noAccountWrite + metric: no_account_write + weight: 5 + - type: javascript + value: file://assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 2 diff --git a/evals/scripts/_manifest.js b/evals/scripts/_manifest.js index 18022763..34e4a34f 100644 --- a/evals/scripts/_manifest.js +++ b/evals/scripts/_manifest.js @@ -79,6 +79,12 @@ const SUITES = [ skillDir: "skills/onboarding", readme: "skills/onboarding/README.md", }, + { + suite: "release-autopilot-onboarding", + skillKey: "release-autopilot/launchdarkly-release-autopilot-onboarding", + skillDir: "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", + readme: "skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md", + }, { suite: "experiment-guide", skillKey: "experiments/launchdarkly-experiment-guide", diff --git a/evals/tests/provider-match-option.test.js b/evals/tests/provider-match-option.test.js new file mode 100644 index 00000000..f113f5b1 --- /dev/null +++ b/evals/tests/provider-match-option.test.js @@ -0,0 +1,27 @@ +const { test } = require("node:test"); +const assert = require("node:assert/strict"); + +// matchOption resolves a scripted ask-question answer to an option id, so a +// test can steer by label when the agent invents the ids. +const { matchOption } = require("../providers/claude-skill-agent-sdk"); + +const options = [ + { id: "af", label: "Auto-flagging (Recommended)" }, + { id: "ar", label: "Auto-releasing" }, +]; + +test("keeps an exact id", () => { + assert.equal(matchOption("ar", options), "ar"); +}); + +test("matches a label substring, case-insensitively", () => { + assert.equal(matchOption("auto-flagging", options), "af"); +}); + +test("passes an unmatched answer through", () => { + assert.equal(matchOption("nothing", options), "nothing"); +}); + +test("tolerates missing options", () => { + assert.equal(matchOption("x", undefined), "x"); +}); diff --git a/evals/tools/definitions.json b/evals/tools/definitions.json index db990c83..7305f00e 100644 --- a/evals/tools/definitions.json +++ b/evals/tools/definitions.json @@ -685,5 +685,15 @@ }, "required": ["projectKey", "environmentKey", "experimentKey"] } - } + }, + {"name": "get-member-self", "description": "Return the member ID, profile, role, and team memberships for the currently authenticated user.", "input_schema": {"type": "object", "properties": {}}}, + {"name": "get-release-autopilot-settings", "description": "Get account-wide Release Autopilot settings: whether auto-flagging and auto-releasing are on, and whether auto-flagging pull requests need approval. These are the master gate. Do not use autoCleanup.", "input_schema": {"type": "object", "properties": {}}}, + {"name": "update-release-autopilot-settings", "description": "Update account-wide Release Autopilot settings (auto-flagging and auto-releasing). Send only what changed. Requires permission to update account Release Autopilot settings. Do not send autoCleanup.", "input_schema": {"type": "object", "properties": {"autoFlagging": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}, "autoReleasing": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}}}}, + {"name": "list-release-autopilot-github-repos", "description": "List GitHub repositories on this account's LaunchDarkly GitHub App install, with name (owner/name) and numeric githubRepoId. When githubConnectUrl is present, give that link to the user and retry after they finish.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}}}}, + {"name": "list-release-autopilot-repo-settings", "description": "List Release Autopilot settings for GitHub repositories already mapped to a LaunchDarkly project. Each automation is the effective value. githubRepoId is numeric; repo names are not included.", "input_schema": {"type": "object", "properties": {}}}, + {"name": "get-release-autopilot-repo-settings", "description": "Get Release Autopilot settings for one GitHub repository. Pass repo as owner/name, or githubRepoId. 404 means the repo is not mapped yet.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}, "projectKey": {"type": "string"}}}}, + {"name": "update-release-autopilot-repo-settings", "description": "Map a GitHub repository to a LaunchDarkly project and optionally override auto-flagging or auto-releasing. Pass repo as owner/name. projectKey is required on the first map. An omitted automation inherits the account setting. Do not send autoCleanup.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}, "projectKey": {"type": "string"}, "autoFlagging": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}, "autoReleasing": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}}}}, + {"name": "delete-release-autopilot-repo-settings", "description": "Unmap a GitHub repository from Release Autopilot.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}}}}, + {"name": "list-release-plans", "description": "List environment-scoped release plans. A release plan is a reusable progressive-release recipe for one environment. It replaces project-level release policies.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "includeIncomplete": {"type": "boolean"}}, "required": ["projectKey", "environmentKey"]}}, + {"name": "create-release-plan", "description": "Create an environment-scoped release plan. Does not start a release. Every phase needs 1-10 clauses and 1-10 steps.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "key": {"type": "string"}, "name": {"type": "string"}, "progressiveRelease": {"type": "object"}}, "required": ["projectKey", "environmentKey", "key", "name", "progressiveRelease"]}} ] From ee8dbbe2fdce0b15da3ad11ad64a6b5df9320837 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 17:48:26 -0700 Subject: [PATCH 12/30] test(evals): accept CLAUDE_CODE_OAUTH_TOKEN as agent auth Lets engineers without an API key run suites on a Claude subscription via claude setup-token. Co-Authored-By: Claude Opus 5.5 (1M context) --- evals/.env.example | 5 +++++ evals/providers/claude-skill-agent-sdk.js | 13 ++++++++++--- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/evals/.env.example b/evals/.env.example index 95e682f0..4352bdaa 100644 --- a/evals/.env.example +++ b/evals/.env.example @@ -6,6 +6,11 @@ # Request access via #ask-app-eng to get an API key. ANTHROPIC_API_KEY= +# ALTERNATIVE to ANTHROPIC_API_KEY for the agent: a Claude subscription token. +# Run `claude setup-token` in your own terminal and paste the token here. +# llm-rubric assertions still need RUBRIC_MODEL to reach a grader. +# CLAUDE_CODE_OAUTH_TOKEN= + # OPTIONAL: the system-under-test model. Stays on Claude because that's what # real users will run skills against. Defaults to claude-sonnet-4-6 if unset. # AGENT_MODEL=claude-sonnet-4-6 diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 1e76385e..4bf35e77 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -59,6 +59,8 @@ * Environment variables: * AGENT_MODEL - SUT model (default claude-sonnet-4-6). * ANTHROPIC_API_KEY - Auth for the SDK's child Claude Code process. + * CLAUDE_CODE_OAUTH_TOKEN - Alternative auth from `claude setup-token`, for + * running evals on a Claude subscription without an API key. * SKILL_EVAL_DEBUG=1 - Dump every SDK message to a temp file for inspection. */ @@ -226,9 +228,14 @@ class ClaudeSkillAgentSdk { } async callApi(_prompt, context) { - const apiKey = process.env.ANTHROPIC_API_KEY; - if (!apiKey) { - return { error: "ANTHROPIC_API_KEY environment variable is not set" }; + // Either an API key or a Claude subscription token (`claude setup-token`) + // authenticates the SDK's child Claude Code process. The isolated config + // dir hides the interactive /login, so one of these must be in the env. + if (!process.env.ANTHROPIC_API_KEY && !process.env.CLAUDE_CODE_OAUTH_TOKEN) { + return { + error: + "Set ANTHROPIC_API_KEY, or CLAUDE_CODE_OAUTH_TOKEN from `claude setup-token`", + }; } const userRequest = From 0c498310d4b72da52e5cebe417a21636630690c5 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 18:09:27 -0700 Subject: [PATCH 13/30] test(evals): run onboarding suite end to end; ban raw names in admin checklists Fix assertion paths (they resolve from shared/), replace the rubric grader with deterministic checks so the suite runs on a subscription token, and add a no-raw-names check after the first run showed tool and field names in the reader and writer checklists. The skill's voice rule now covers those checklists. All 6 cases pass. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../assertions.js | 20 +++++++ .../promptfooconfig.yaml | 52 +++++++++++-------- .../SKILL.md | 2 +- 3 files changed, 52 insertions(+), 22 deletions(-) diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 8c8960bd..b9ed1ef1 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -75,3 +75,23 @@ module.exports.releasePlansCalledAtMostOnce = (output) => { module.exports.repoTurnsAutoReleasingOff = repoOverridesUnpicked(["autoReleasing"]); module.exports.noReleasePlanCreated = noneOf(["create-release-plan"]); module.exports.noAccountWrite = noneOf(["update-release-autopilot-settings"]); + +const finalText = (output) => String((output && output.response) || ""); + +// Raw tool and field names the skill's voice rules ban from user-facing text. +const RAW_NAMES = [ + "update-release-autopilot", "get-release-autopilot", "list-release-autopilot", + "githubRepoId", "enabledOverride", "autoReleasing", "autoFlagging", "approvalRequired", +]; + +module.exports.noRawNames = (output) => { + const text = finalText(output); + const hits = RAW_NAMES.filter((n) => text.includes(n)); + return result(hits.length === 0, hits.length ? `final message shows: ${hits.join(", ")}` : "no raw names"); +}; + +module.exports.previewNamesTheMapping = (output) => { + const text = finalText(output).toLowerCase(); + const pass = text.includes("acme/demo") && text.includes("default") && /auto-releasing/.test(text); + return result(pass, pass ? "names the repo, project, and auto-releasing" : "final message does not describe the mapping"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index e29f5447..4e165f7d 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -10,7 +10,8 @@ # account-wide policy mode, and account settings stay untouched when they # already match. Default mocks match a catamorphic-like account: both # capabilities on, the GitHub App installed, release plans forbidden by the -# account's plan. Tests change state with mock_tool_overrides, and steer +# account's plan. Mocks are stateless, so read-backs after a write return the +# default state; assertions grade the writes, not the verification reads. Tests change state with mock_tool_overrides, and steer # ask-question by option label. description: "Evaluation of the Release Autopilot onboarding skill" @@ -51,23 +52,23 @@ tests: - ["Looks good"] assert: - type: javascript - value: file://assertions.js:confirmsBeforeWriting + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting metric: confirms_before_writing weight: 4 - type: javascript - value: file://assertions.js:repoTurnsAutoReleasingOff + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff metric: repo_override_unpicked weight: 4 - type: javascript - value: file://assertions.js:accountWriteLast + value: file://../release-autopilot-onboarding/assertions.js:accountWriteLast metric: account_write_last weight: 2 - type: javascript - value: file://assertions.js:neverAutoReleasingOn + value: file://../release-autopilot-onboarding/assertions.js:neverAutoReleasingOn metric: no_auto_releasing weight: 3 - type: javascript - value: file://assertions.js:noAutoCleanup + value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup metric: no_auto_cleanup weight: 2 @@ -98,23 +99,23 @@ tests: - ["Looks good"] assert: - type: javascript - value: file://assertions.js:releasePlansCalledAtMostOnce + value: file://../release-autopilot-onboarding/assertions.js:releasePlansCalledAtMostOnce metric: plans_403_account_wide weight: 3 - type: javascript - value: file://assertions.js:noReleasePlanCreated + value: file://../release-autopilot-onboarding/assertions.js:noReleasePlanCreated metric: no_plans_created weight: 3 - type: javascript - value: file://assertions.js:repoTurnsAutoReleasingOff + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff metric: auto_releasing_off_without_policy weight: 4 - type: javascript - value: file://assertions.js:noAccountWrite + value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite metric: no_redundant_account_write weight: 3 - type: javascript - value: file://assertions.js:confirmsBeforeWriting + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting metric: confirms_before_writing weight: 3 @@ -137,9 +138,13 @@ tests: - ["Looks good"] assert: - type: javascript - value: file://assertions.js:noWrites + value: file://../release-autopilot-onboarding/assertions.js:noWrites metric: no_writes weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 # ------------------------------------------------------------------ # Explicit preview request from an admin: zero writes, and the final @@ -157,16 +162,17 @@ tests: - ["managed"] assert: - type: javascript - value: file://assertions.js:noWrites + value: file://../release-autopilot-onboarding/assertions.js:noWrites metric: no_writes weight: 5 - - type: llm-rubric - value: | - The final message lists the change that would be made: mapping acme/demo to project default, - with auto-releasing turned off for that repository. It says nothing was changed. It does not - show raw field names such as githubRepoId, enabledOverride, or autoReleasing.enabled. + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:previewNamesTheMapping metric: preview_plan weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 # ------------------------------------------------------------------ # GitHub App not installed, and the user needs an org owner: hand the @@ -194,7 +200,7 @@ tests: - ["Need an org owner"] assert: - type: javascript - value: file://assertions.js:noWrites + value: file://../release-autopilot-onboarding/assertions.js:noWrites metric: no_writes weight: 5 - type: javascript @@ -229,10 +235,14 @@ tests: - ["Looks good"] assert: - type: javascript - value: file://assertions.js:noAccountWrite + value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite metric: no_account_write weight: 5 - type: javascript - value: file://assertions.js:confirmsBeforeWriting + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting metric: confirms_before_writing weight: 2 diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 83cf2949..7b05ac08 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -62,7 +62,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Speaking to the user - Use product words: "auto-flagging", "auto-releasing", "repository", "project", "release policy", "release plan", "approval". -- Never show tool names or field names such as `githubRepoId`, `enabledOverride`, or `autoReleasing.enabled`. Say "repository ID", "turned off for this repository", or "auto-releasing is on". +- Never show tool names or field names such as `update-release-autopilot-repo-settings`, `githubRepoId`, `enabledOverride`, or `autoReleasing.enabled`. Say "repository ID", "turned off for this repository", or "auto-releasing is on". This applies to every message, including the admin checklist and the final summary: describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value. - Keep each message to the current step. Do not narrate tool calls. ## Workflow From 680c4941f59833710b5bec15809ad4cf81a314a6 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 18:36:06 -0700 Subject: [PATCH 14/30] fix(release-autopilot-onboarding): only 'Looks good' approves; tighten voice; expand evals to 10 cases Iteration 3 found the agent applying changes after confirm answers that were not 'Looks good', hidden by positional scripted answers drifting onto the wrong questions. The skill now re-asks on any other answer, uses explicit answers from the request, and keeps internal mode and field names out of user-facing text. The harness gains opt-in label-matched answers; assertions check the approved option, the exact repo and project, valid release plan phases, and raw names across all turns. New cases: change at confirm, already-mapped repo override, no remap, and plan mode. Co-Authored-By: Claude Opus 5.5 (1M context) --- evals/providers/claude-skill-agent-sdk.js | 29 +++- .../assertions.js | 64 +++++++-- .../promptfooconfig.yaml | 136 +++++++++++++++++- evals/tools/definitions.json | 2 +- .../SKILL.md | 12 +- 5 files changed, 222 insertions(+), 21 deletions(-) diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 4bf35e77..92025f41 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -49,6 +49,12 @@ * not one of the call's option ids is matched * against option labels (case-insensitive substring), * so tests can steer by label when the agent picks ids + * mock_ask_question_match - when true, each `ask-question` call takes the first + * unused scripted answer whose entries all match its + * options (by id or label), instead of the next one in + * order. Use for flows that may skip questions. A call + * with no match gets its first option and is recorded + * with `scripted: false` * mock_tool_overrides - optional object mapping tool name to a mock template * that replaces mocks/tool-responses.json for this test, * e.g. to return a 403 or a different account state @@ -260,6 +266,8 @@ class ClaudeSkillAgentSdk { : []; const mockFiles = context?.vars?.mock_files || {}; const mockToolOverrides = context?.vars?.mock_tool_overrides || {}; + const matchAnswers = context?.vars?.mock_ask_question_match === true; + const usedAnswers = new Set(); let userMessage = userRequest; if (codebaseContext) { @@ -342,9 +350,24 @@ class ClaudeSkillAgentSdk { }, async (args) => { const idx = askQuestionCallCount++; - const override = askQuestionAnswers[idx]; + let override = askQuestionAnswers[idx]; + let scripted = Array.isArray(override) && override.length > 0; + if (matchAnswers) { + const hit = askQuestionAnswers.findIndex( + (ans, i) => + !usedAnswers.has(i) && + Array.isArray(ans) && + ans.length > 0 && + ans.every((want) => + (args.options || []).some((o) => o.id === matchOption(want, args.options)), + ), + ); + scripted = hit !== -1; + override = scripted ? askQuestionAnswers[hit] : undefined; + if (scripted) usedAnswers.add(hit); + } let selected; - if (Array.isArray(override) && override.length > 0) { + if (scripted) { selected = override.map((want) => matchOption(want, args.options)); } else if (Array.isArray(args.options) && args.options.length > 0) { selected = [args.options[0].id]; @@ -357,6 +380,8 @@ class ClaudeSkillAgentSdk { tool: "ask-question", arguments: args, turn: currentTurn, + scripted, + selected, mock_response_preview: previewMock(response), }); diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index b9ed1ef1..959d9acd 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -30,18 +30,25 @@ module.exports.confirmsBeforeWriting = (output) => { const steps = traj(output); const firstWrite = steps.findIndex((t) => WRITE_TOOLS.has(t.tool)); if (firstWrite === -1) return result(false, "never wrote, so the confirmed plan was not applied"); - const confirm = steps.findIndex(isConfirmQuestion); - return result( - confirm !== -1 && confirm < firstWrite, - confirm === -1 ? "wrote without a confirm question" : `confirm at ${confirm}, first write at ${firstWrite}`, - ); + let confirm = -1; + for (let i = 0; i < firstWrite; i++) if (isConfirmQuestion(steps[i])) confirm = i; + if (confirm === -1) return result(false, "wrote without a confirm question"); + const step = steps[confirm]; + const ok = ((step.arguments && step.arguments.options) || []) + .filter((o) => String(o.label || "").toLowerCase().includes("looks good")) + .map((o) => o.id); + const approved = (step.selected || []).some((id) => ok.includes(id)); + return result(approved, approved ? `approved at ${confirm}, first write at ${firstWrite}` : `wrote after confirm answer ${JSON.stringify(step.selected)}`); }; -const repoOverridesUnpicked = (caps) => (output) => { +// Every repo write targets `repo` in `project` and sends enabled:false for each cap. +const mapsRepoWithOverrides = (repo, project, caps) => (output) => { const maps = traj(output).filter((t) => t.tool === "update-release-autopilot-repo-settings"); - if (!maps.length) return result(false, "never mapped a repo"); - const bad = maps.filter((m) => caps.some((c) => !(m.arguments && m.arguments[c] && m.arguments[c].enabled === false))); - return result(bad.length === 0, bad.length ? `missing ${caps.join("/")} enabled:false on ${bad.length} mapping(s)` : "overrides sent"); + if (!maps.length) return result(false, "never wrote a repo mapping"); + const wrong = maps.filter((m) => (m.arguments || {}).repo !== repo || (project && (m.arguments || {}).projectKey && m.arguments.projectKey !== project)); + if (wrong.length) return result(false, `wrote ${wrong.map((m) => `${m.arguments.repo}@${m.arguments.projectKey}`).join(", ")}; want only ${repo}@${project}`); + const bad = maps.filter((m) => caps.some((c) => !(m.arguments[c] && m.arguments[c].enabled === false))); + return result(bad.length === 0, bad.length ? `missing ${caps.join("/")} enabled:false` : `${repo} written with ${caps.join("/")} off`); }; module.exports.accountWriteLast = (output) => { @@ -72,7 +79,35 @@ module.exports.releasePlansCalledAtMostOnce = (output) => { return result(n <= 1, `list-release-plans called ${n} time(s)`); }; -module.exports.repoTurnsAutoReleasingOff = repoOverridesUnpicked(["autoReleasing"]); +module.exports.repoTurnsAutoReleasingOff = mapsRepoWithOverrides("acme/demo", "default", ["autoReleasing"]); +module.exports.storefrontAutoReleasingOff = mapsRepoWithOverrides("acme/storefront", null, ["autoReleasing"]); +module.exports.noRepoWrite = noneOf(["update-release-autopilot-repo-settings", "delete-release-autopilot-repo-settings"]); + +module.exports.planBeforeRepoWrite = (output) => { + const steps = traj(output); + const plan = steps.findIndex((t) => t.tool === "create-release-plan"); + const repo = steps.findIndex((t) => t.tool === "update-release-autopilot-repo-settings"); + if (plan === -1) return result(false, "never created a release plan"); + return result(repo === -1 || plan < repo, `plan at ${plan}, repo write at ${repo}`); +}; + +module.exports.plansHaveClauses = (output) => { + const plans = traj(output).filter((t) => t.tool === "create-release-plan"); + if (!plans.length) return result(false, "no release plan created"); + const okClause = (c) => c && typeof c.attribute === "string" && typeof c.op === "string" && Array.isArray(c.values); + const okStep = (s) => s && typeof s.allocation === "number" && typeof s.durationMillis === "number"; + for (const p of plans) { + const phases = (((p.arguments || {}).progressiveRelease || {}).phases) || []; + if (!phases.length) return result(false, "plan has no phases"); + for (const ph of phases) { + if (!Array.isArray(ph.clauses) || !ph.clauses.length || !ph.clauses.every(okClause)) + return result(false, `bad clauses: ${JSON.stringify(ph.clauses).slice(0, 120)}`); + if (!Array.isArray(ph.steps) || !ph.steps.length || !ph.steps.every(okStep)) + return result(false, `bad steps: ${JSON.stringify(ph.steps).slice(0, 120)}`); + } + } + return result(true, "every phase has valid clauses and steps"); +}; module.exports.noReleasePlanCreated = noneOf(["create-release-plan"]); module.exports.noAccountWrite = noneOf(["update-release-autopilot-settings"]); @@ -82,12 +117,17 @@ const finalText = (output) => String((output && output.response) || ""); const RAW_NAMES = [ "update-release-autopilot", "get-release-autopilot", "list-release-autopilot", "githubRepoId", "enabledOverride", "autoReleasing", "autoFlagging", "approvalRequired", + "create-release-plan", "list-release-plans", "list-release-policies", ]; module.exports.noRawNames = (output) => { - const text = finalText(output); + const turns = ((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? x : JSON.stringify(x))); + const asks = traj(output) + .filter((t) => t.tool === "ask-question") + .map((t) => [t.arguments.prompt, ...((t.arguments.options || []).map((o) => o.label))].join(" ")); + const text = [finalText(output), ...turns, ...asks].join("\n"); const hits = RAW_NAMES.filter((n) => text.includes(n)); - return result(hits.length === 0, hits.length ? `final message shows: ${hits.join(", ")}` : "no raw names"); + return result(hits.length === 0, hits.length ? `user-facing text shows: ${hits.join(", ")}` : "no raw names"); }; module.exports.previewNamesTheMapping = (output) => { diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 4e165f7d..0d326a0c 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -10,7 +10,9 @@ # account-wide policy mode, and account settings stay untouched when they # already match. Default mocks match a catamorphic-like account: both # capabilities on, the GitHub App installed, release plans forbidden by the -# account's plan. Mocks are stateless, so read-backs after a write return the +# account's plan. Scripted answers are matched to each question by option +# label (mock_ask_question_match), so a skipped question cannot shift them. +# Mocks are stateless, so read-backs after a write return the # default state; assertions grade the writes, not the verification reads. Tests change state with mock_tool_overrides, and steer # ask-question by option label. description: "Evaluation of the Release Autopilot onboarding skill" @@ -43,6 +45,7 @@ tests: autoReleasing: { enabled: false } version: 1 list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - ["Require approval"] @@ -71,6 +74,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup metric: no_auto_cleanup weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 # ------------------------------------------------------------------ # Account already on, both capabilities picked, policy mode with no @@ -89,6 +96,7 @@ tests: warnings: - code: missing_policy message: "No release policy matches." + mock_ask_question_match: true mock_ask_question_answers: - ["Add more"] - ["Auto-flagging", "Auto-releasing"] @@ -102,6 +110,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:releasePlansCalledAtMostOnce metric: plans_403_account_wide weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noReleasePlanCreated metric: no_plans_created @@ -129,6 +141,7 @@ tests: max_turns: 20 mock_tool_overrides: get-member-self: { id: "member-2", email: "reader@acme.example", role: "reader", teams: [] } + mock_ask_question_match: true mock_ask_question_answers: - ["Add more"] - ["Auto-flagging"] @@ -154,6 +167,7 @@ tests: vars: user_request: "Do a dry run: preview what you'd change to set up Release Autopilot auto-flagging for acme/demo in project default. Don't change anything." max_turns: 20 + mock_ask_question_match: true mock_ask_question_answers: - ["Add more"] - ["Auto-flagging"] @@ -194,6 +208,7 @@ tests: repositories: [] totalCount: 0 githubConnectUrl: "https://app.launchdarkly.com/integrations/github/connect?flow=install" + mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - ["Require approval"] @@ -226,6 +241,7 @@ tests: autoReleasing: { enabled: false } version: 1 list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - ["Require approval"] @@ -246,3 +262,121 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting metric: confirms_before_writing weight: 2 + + # ------------------------------------------------------------------ + # "Change something" at the confirm step is not approval: no writes. + # ------------------------------------------------------------------ + - description: "Change something at confirm: zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." + max_turns: 20 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Require approval"] + - ["managed"] + - ["Change something"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + + # ------------------------------------------------------------------ + # acme/storefront is already mapped to default and inherits + # auto-releasing from the account. The user picks auto-flagging only, + # so the only write is an override that turns auto-releasing off. + # ------------------------------------------------------------------ + - description: "Already mapped with an unpicked capability: override write only" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/storefront in project default. I don't want auto-releasing." + max_turns: 25 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["managed"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:storefrontAutoReleasingOff + metric: override_unpicked_on_mapped_repo + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite + metric: no_redundant_account_write + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + + # ------------------------------------------------------------------ + # acme/storefront is mapped to default; the user asks for project web. + # The skill must not remap; it points to the settings skill instead. + # ------------------------------------------------------------------ + - description: "Mapped to another project: no remap" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/storefront in project web." + max_turns: 20 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["managed"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRepoWrite + metric: no_remap + weight: 5 + + # ------------------------------------------------------------------ + # Plan mode: release plans are available and empty. The plan is + # created before the repo write, and every phase has a clause. + # ------------------------------------------------------------------ + - description: "Plan mode: release plan created first, valid phases" + vars: + user_request: > + Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default. + For production only, create a release plan: randomize by user, 1% then 10% then 50% then 100% + with 4h holds, automatic advancement, no metrics, and leave rules in place. + max_turns: 30 + mock_tool_overrides: + list-release-plans: { items: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging", "Auto-releasing"] + - ["managed"] + - ["Production"] + - ["Automatic"] + - ["Leave rules"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:planBeforeRepoWrite + metric: plan_before_repo_write + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:plansHaveClauses + metric: plan_phases_have_clauses + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 diff --git a/evals/tools/definitions.json b/evals/tools/definitions.json index 7305f00e..18c930b1 100644 --- a/evals/tools/definitions.json +++ b/evals/tools/definitions.json @@ -695,5 +695,5 @@ {"name": "update-release-autopilot-repo-settings", "description": "Map a GitHub repository to a LaunchDarkly project and optionally override auto-flagging or auto-releasing. Pass repo as owner/name. projectKey is required on the first map. An omitted automation inherits the account setting. Do not send autoCleanup.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}, "projectKey": {"type": "string"}, "autoFlagging": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}, "autoReleasing": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}}}}, {"name": "delete-release-autopilot-repo-settings", "description": "Unmap a GitHub repository from Release Autopilot.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}}}}, {"name": "list-release-plans", "description": "List environment-scoped release plans. A release plan is a reusable progressive-release recipe for one environment. It replaces project-level release policies.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "includeIncomplete": {"type": "boolean"}}, "required": ["projectKey", "environmentKey"]}}, - {"name": "create-release-plan", "description": "Create an environment-scoped release plan. Does not start a release. Every phase needs 1-10 clauses and 1-10 steps.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "key": {"type": "string"}, "name": {"type": "string"}, "progressiveRelease": {"type": "object"}}, "required": ["projectKey", "environmentKey", "key", "name", "progressiveRelease"]}} + {"name": "create-release-plan", "description": "Create an environment-scoped release plan, a reusable progressive-release recipe. Does not start a release. PHASES: 1-10; every phase needs 1-10 clauses AND 1-10 steps. An \"everyone\" phase still needs a clause, e.g. { attribute: \"kind\", op: \"in\", values: [\"user\"], contextKind: \"user\" }. STEPS: allocation is per-100k (5000 = 5%, 100000 = 100%); durationMillis is the hold before the next step. METRICS: keyed object; {} = progressive. advancementMode defaults to manual.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "key": {"type": "string"}, "name": {"type": "string"}, "description": {"type": "string"}, "progressiveRelease": {"type": "object", "properties": {"randomizationUnit": {"type": "string"}, "advancementMode": {"type": "string", "enum": ["manual", "automatic"]}, "cleanupBehavior": {"type": "string", "enum": ["leave_rules_in_place", "remove_rules_and_update_default_rule"]}, "phases": {"type": "array", "items": {"type": "object", "properties": {"description": {"type": "string"}, "clauses": {"type": "array", "items": {"type": "object", "properties": {"attribute": {"type": "string"}, "op": {"type": "string", "enum": ["in", "endsWith", "startsWith", "matches", "contains", "lessThan", "lessThanOrEqual", "greaterThan", "greaterThanOrEqual", "before", "after", "semVerEqual", "semVerLessThan", "semVerGreaterThan", "segmentMatch"]}, "values": {"type": "array", "items": {}}, "contextKind": {"type": "string"}, "negate": {"type": "boolean"}}, "required": ["attribute", "op", "values"]}}, "steps": {"type": "array", "items": {"type": "object", "properties": {"allocation": {"type": "number"}, "durationMillis": {"type": "number"}}, "required": ["allocation", "durationMillis"]}}}, "required": ["clauses", "steps"]}}, "metrics": {"type": "object"}}, "required": ["randomizationUnit", "phases"]}}, "required": ["projectKey", "environmentKey", "key", "name", "progressiveRelease"]}} ] diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 7b05ac08..a8bd5aeb 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -62,7 +62,9 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Speaking to the user - Use product words: "auto-flagging", "auto-releasing", "repository", "project", "release policy", "release plan", "approval". -- Never show tool names or field names such as `update-release-autopilot-repo-settings`, `githubRepoId`, `enabledOverride`, or `autoReleasing.enabled`. Say "repository ID", "turned off for this repository", or "auto-releasing is on". This applies to every message, including the admin checklist and the final summary: describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value. +- Never show tool names or field names such as `update-release-autopilot-repo-settings`, `githubRepoId`, `enabledOverride`, or `autoReleasing.enabled`. Say "repository ID", "turned off for this repository", or "auto-releasing is on". This applies to every message, including questions, the admin checklist, and the final summary: describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value. +- "Policy mode" and "plan mode" are names for this skill only. Tell the user "your account uses release policies" or "your account uses release plans". +- If a verification read disagrees with a write, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. Do not name the tools involved. - Keep each message to the current step. Do not narrate tool calls. ## Workflow @@ -71,7 +73,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the **Response scoping:** During this flow, treat user messages as answers to the current question. Do not take unrelated actions until the flow is finished. -**Asking questions:** When a question has options, use `AskUserQuestion` with the labels and descriptions given below. Ask free-text questions as a normal message. +**Asking questions:** When a question has options, use `AskUserQuestion` with the labels and descriptions given below. Ask free-text questions as a normal message. If the user's request already answers a question explicitly, such as naming the repo, the project, or the capabilities, use that answer and say so instead of asking. Never infer the answer to the Step 7 confirmation. **State tracking:** Track every decision. Step 7 re-states all of them. @@ -108,7 +110,7 @@ Run these reads. Do not write anything. 3. `list-release-autopilot-repo-settings`: repos that are already mapped. 4. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. -`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. A mapped `githubRepoId` with no match on the install list is a repo the App can no longer see, or one this member cannot see: show it as "repository ID , not on the GitHub install" with its project, and do not change it in this flow. +`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. A mapped `githubRepoId` with no match on the install list is a repo the App can no longer see, or one this member cannot see: tell the user, for example, "One more repository (ID 999) is mapped to project default but is no longer on the GitHub install; I'll leave it alone." Do not write the field name. Do not change that repo in this flow. Release Autopilot's agents need generative AI turned on for the organization. There is no MCP tool to read that setting. If a later step fails with a generative AI or Vega eligibility error, tell the user that an Admin or Owner must select **Enable generative AI use cases** in Organization settings > Security, then stop. @@ -235,7 +237,7 @@ If there are no writes, say so, list the follow-ups, and stop. - label: "Looks good", description: "Make every change in this plan" - label: "Change something", description: "Adjust the plan before you start" -On "Change something", make the change and show the full plan again. Do not go to Step 8 until the user selects "Looks good". +Only "Looks good" approves the plan. On "Change something", ask what to change, make the change, and show the full plan again. Any other answer, including one that does not match an option, is not approval: ask the confirmation again. Do not go to Step 8 until the user selects "Looks good". ### Step 8: Apply @@ -256,7 +258,7 @@ If a write fails with 403, the member lacks that permission. Report which write 2. `get-release-autopilot-repo-settings` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. 3. Plan mode: `list-release-plans` for each environment lists the new plans. Policy mode: report the policy state from Step 6. -Report a short checklist with each item marked verified or reported by the user, then the follow-ups. Then tell the user how to see it work: +Report a short checklist with each item marked verified, reported by the user, or already set up (for a step the flow skipped because it was already done), then the follow-ups. Then tell the user how to see it work: - Open a pull request in a mapped repo. Release Autopilot reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. - If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. From 698feca896d9a60c429b9e4d7652e027d1dbd6cf Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 18:54:13 -0700 Subject: [PATCH 15/30] =?UTF-8?q?feat(release-autopilot-onboarding):=20ite?= =?UTF-8?q?ration=204=20=E2=80=94=20scoped=20eval=20tools,=20stateful=20mo?= =?UTF-8?q?cks,=20resume=20and=20install-path=20cases?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scope the Release Autopilot tool definitions so other suites see exactly the tools they saw before. Record Release Autopilot writes in the mocks so verification reads reflect them. Add cases for resume (triggered from the user's words, with the Skill tool available), an uninstalled GitHub org, and a deprecated MCP server; 13 cases pass. In the skill: resume skips the intro and still stops at the confirmation, restated picks are never approval, list-metrics is plan-mode only, voice rules cover status codes and step numbers, and mapping sub-cases move to references. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 1 + evals/providers/_mock.js | 47 ++++++ evals/providers/claude-skill-agent-sdk.js | 56 ++++--- .../assertions.js | 24 +++ .../promptfooconfig.yaml | 137 ++++++++++++++++-- evals/tests/mock.test.js | 32 +++- evals/tests/provider-match-option.test.js | 18 ++- evals/tools/definitions.json | 20 +-- .../SKILL.md | 24 ++- .../references/edge-cases.md | 8 + 10 files changed, 311 insertions(+), 56 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c81a95b..e14ce58f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ All notable changes to this repository will be documented in this file. - Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. +- evals: add the `release-autopilot-onboarding` suite (13 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. - Rename `factory/launchdarkly-factory-settings` to `release-autopilot/launchdarkly-release-autopilot-settings` and switch it to the `*-release-autopilot-*` MCP tools (Claude Code plugin 1.0.2, Cursor plugin 1.0.3). Requires a hosted MCP server that exposes the new tool names. diff --git a/evals/providers/_mock.js b/evals/providers/_mock.js index 831b70b1..ce5a12a2 100644 --- a/evals/providers/_mock.js +++ b/evals/providers/_mock.js @@ -126,6 +126,9 @@ function createMockState() { return { configs: {}, // configKey -> config object flags: {}, // flagKey -> flag object + raSettings: null, // Release Autopilot account settings written this run + raRepos: {}, // owner/name -> Release Autopilot repo settings written this run + raPlans: {}, // environmentKey -> release plans created this run }; } @@ -134,9 +137,53 @@ function createMockState() { * `state` is the per-invocation map from `createMockState()`. * `templates` is the full parsed mocks/tool-responses.json. */ +function mergeDeep(base, patch) { + const out = { ...(base || {}) }; + for (const [k, v] of Object.entries(patch || {})) { + out[k] = v && typeof v === "object" && !Array.isArray(v) ? mergeDeep(out[k], v) : v; + } + return out; +} + +// Release Autopilot settings and release plans: writes are recorded so that +// later reads (the skill's verify step) reflect them instead of the template. +function renderReleaseAutopilot(template, input, toolName, state, replacements) { + const args = input || {}; + if (toolName === "update-release-autopilot-settings") { + state.raSettings = mergeDeep(state.raSettings, args); + return { ...state.raSettings }; + } + if (toolName === "get-release-autopilot-settings" && state.raSettings) { + return mergeDeep(walk(template, replacements), state.raSettings); + } + if (toolName === "update-release-autopilot-repo-settings" && args.repo) { + const { repo, githubRepoId, ...rest } = args; + state.raRepos[repo] = mergeDeep(state.raRepos[repo] || { repo }, rest); + return { ...state.raRepos[repo] }; + } + if (toolName === "get-release-autopilot-repo-settings" && args.repo && state.raRepos[args.repo]) { + return { ...state.raRepos[args.repo] }; + } + if (toolName === "create-release-plan" && args.environmentKey) { + const plan = { key: args.key, name: args.name, environmentKey: args.environmentKey, progressiveRelease: args.progressiveRelease }; + (state.raPlans[args.environmentKey] = state.raPlans[args.environmentKey] || []).push(plan); + return plan; + } + if (toolName === "list-release-plans" && state.raPlans[args.environmentKey]) { + const items = state.raPlans[args.environmentKey]; + return { items, totalCount: items.length }; + } + return undefined; +} + function renderMockResponse(template, input, toolName, state) { const replacements = buildReplacements(input); + if (state && state.raRepos) { + const ra = renderReleaseAutopilot(template, input, toolName, state, replacements); + if (ra !== undefined) return ra; + } + // ---------- stateful reads ---------- if (toolName === "get-ai-config") { const key = input.configKey || input.key; diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 92025f41..7e320641 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -27,6 +27,9 @@ * `/` to explicitly invoke the skill via * slash command. Use for routing/advisory skills whose * description-based activation is unreliable in eval. + * tool_scopes - Optional list of tool scopes to expose. Definitions in + * tools/definitions.json that carry a `scope` are hidden + * from suites that do not list it. * expose_ask_question - Default false. When true, expose a second in-process * MCP server with an `ask-question` tool that records * calls into the trajectory and returns a canned answer. @@ -102,6 +105,30 @@ function matchOption(want, options) { return hit ? hit.id : want; } +/** + * Choose the scripted answer for the `idx`-th ask-question call. In order + * mode, take answers[idx]. In match mode, take the first unused answer whose + * entries all resolve to the call's options, and mark it used. Returns the + * selected option ids, or undefined when no scripted answer applies. + */ +function pickAnswer(answers, idx, options, matchMode, used) { + const resolves = (ans) => + Array.isArray(ans) && + ans.length > 0 && + ans.every((want) => (options || []).some((o) => o.id === matchOption(want, options))); + let chosen; + if (matchMode) { + const hit = answers.findIndex((ans, i) => !used.has(i) && resolves(ans)); + if (hit === -1) return undefined; + used.add(hit); + chosen = answers[hit]; + } else { + chosen = answers[idx]; + if (!Array.isArray(chosen) || chosen.length === 0) return undefined; + } + return chosen.map((want) => matchOption(want, options)); +} + const mockTemplates = JSON.parse( fs.readFileSync(path.join(__dirname, "../mocks/tool-responses.json"), "utf-8"), ); @@ -219,6 +246,10 @@ class ClaudeSkillAgentSdk { this.exposeMcpTools = config.expose_mcp_tools !== false; this.forceSkillInvocation = Boolean(config.force_skill_invocation); this.exposeAskQuestion = Boolean(config.expose_ask_question); + // Tool definitions with a `scope` are exposed only to suites that list + // that scope, so adding a domain's tools does not change other suites. + const scopes = new Set(Array.isArray(config.tool_scopes) ? config.tool_scopes : []); + this.toolDefs = toolDefs.filter((def) => !def.scope || scopes.has(def.scope)); const source = resolveSkillSource(this.skillSlug); if (!source) { @@ -291,7 +322,7 @@ class ClaudeSkillAgentSdk { const mockState = createMockState(); const mcpTools = exposeMcpTools - ? toolDefs.map((def) => + ? this.toolDefs.map((def) => tool( def.name, def.description, @@ -350,25 +381,11 @@ class ClaudeSkillAgentSdk { }, async (args) => { const idx = askQuestionCallCount++; - let override = askQuestionAnswers[idx]; - let scripted = Array.isArray(override) && override.length > 0; - if (matchAnswers) { - const hit = askQuestionAnswers.findIndex( - (ans, i) => - !usedAnswers.has(i) && - Array.isArray(ans) && - ans.length > 0 && - ans.every((want) => - (args.options || []).some((o) => o.id === matchOption(want, args.options)), - ), - ); - scripted = hit !== -1; - override = scripted ? askQuestionAnswers[hit] : undefined; - if (scripted) usedAnswers.add(hit); - } + const picked = pickAnswer(askQuestionAnswers, idx, args.options, matchAnswers, usedAnswers); + const scripted = picked !== undefined; let selected; if (scripted) { - selected = override.map((want) => matchOption(want, args.options)); + selected = picked; } else if (Array.isArray(args.options) && args.options.length > 0) { selected = [args.options[0].id]; } else { @@ -399,7 +416,7 @@ class ClaudeSkillAgentSdk { const allowedMcpToolNames = []; if (exposeMcpTools) { - for (const def of toolDefs) { + for (const def of this.toolDefs) { allowedMcpToolNames.push(`mcp__launchdarkly-mocks__${def.name}`); } } @@ -645,3 +662,4 @@ module.exports = ClaudeSkillAgentSdk; module.exports.stripRawWrapper = stripRawWrapper; module.exports.resolveExposeMcpTools = resolveExposeMcpTools; module.exports.matchOption = matchOption; +module.exports.pickAnswer = pickAnswer; diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 959d9acd..77e31b48 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -127,6 +127,7 @@ module.exports.noRawNames = (output) => { .map((t) => [t.arguments.prompt, ...((t.arguments.options || []).map((o) => o.label))].join(" ")); const text = [finalText(output), ...turns, ...asks].join("\n"); const hits = RAW_NAMES.filter((n) => text.includes(n)); + if (/\b(401|403|404)\b/.test(text)) hits.push("HTTP status code"); return result(hits.length === 0, hits.length ? `user-facing text shows: ${hits.join(", ")}` : "no raw names"); }; @@ -135,3 +136,26 @@ module.exports.previewNamesTheMapping = (output) => { const pass = text.includes("acme/demo") && text.includes("default") && /auto-releasing/.test(text); return result(pass, pass ? "names the repo, project, and auto-releasing" : "final message does not describe the mapping"); }; + +const allText = (output) => + [finalText(output), ...(((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? x : JSON.stringify(x))))].join("\n"); + +module.exports.saysOrgOwnerInstall = (output) => { + const text = allText(output).toLowerCase(); + const pass = text.includes("other-org") && /owner/.test(text) && /install/.test(text); + return result(pass, pass ? "tells the user an other-org owner must install the App" : "does not explain the org install"); +}; + +module.exports.noIntro = (output) => { + const text = allText(output); + const pass = !text.includes("Release Autopilot Setup") && !/in a few short steps/i.test(text); + return result(pass, pass ? "no intro" : "showed the Setup intro on resume"); +}; + +module.exports.noPickQuestions = (output) => { + const asked = traj(output) + .filter((t) => t.tool === "ask-question") + .map((t) => ((t.arguments && t.arguments.options) || []).map((o) => String(o.label || "").toLowerCase()).join("|")); + const repeats = asked.filter((l) => l.includes("auto-releasing") || l.includes("require approval") || l.includes("bring your own key")); + return result(repeats.length === 0, repeats.length ? `re-asked: ${repeats.join(" ; ")}` : "no picks re-asked"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 0d326a0c..b64f12e1 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -25,9 +25,20 @@ providers: label: mcp config: skill_slug: launchdarkly-release-autopilot-onboarding + tool_scopes: [release-autopilot] allow_builtins: false expose_ask_question: true force_skill_invocation: true + # Same provider without a forced /slash invocation, for cases where the + # skill must trigger from the user's words (a forced bare invocation would + # correctly show the Setup intro before the user's message arrives). + - id: file://../providers/claude-skill-agent-sdk.js + label: mcp-auto + config: + skill_slug: launchdarkly-release-autopilot-onboarding + tool_scopes: [release-autopilot] + allow_builtins: true + expose_ask_question: true tests: # ------------------------------------------------------------------ @@ -35,7 +46,8 @@ tests: # auto-releasing off for the repo, the account write comes after the # repo write, and no write happens before the confirm question. # ------------------------------------------------------------------ - - description: "Fresh account, auto-flagging only: confirm first, repo override off, account write last" + - providers: ["mcp"] + description: "Fresh account, auto-flagging only: confirm first, repo override off, account write last" vars: user_request: "Set up Release Autopilot for acme/demo in project default." max_turns: 25 @@ -85,7 +97,8 @@ tests: # account-wide), no plans are created, the repo keeps auto-releasing # off, and the account settings are not rewritten. # ------------------------------------------------------------------ - - description: "Policy mode, missing policy: one plans call, no plans, auto-releasing off for the repo, no account write" + - providers: ["mcp"] + description: "Policy mode, missing policy: one plans call, no plans, auto-releasing off for the repo, no account write" vars: user_request: "Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default." max_turns: 25 @@ -135,7 +148,8 @@ tests: # Reader: Preview mode is forced, so nothing is written even after # every question is answered. # ------------------------------------------------------------------ - - description: "Reader role: forced Preview, zero writes" + - providers: ["mcp"] + description: "Reader role: forced Preview, zero writes" vars: user_request: "Set up Release Autopilot for acme/demo in project default." max_turns: 20 @@ -163,7 +177,8 @@ tests: # Explicit preview request from an admin: zero writes, and the final # message is a checklist of changes. # ------------------------------------------------------------------ - - description: "Explicit preview: zero writes, ends with the planned changes" + - providers: ["mcp"] + description: "Explicit preview: zero writes, ends with the planned changes" vars: user_request: "Do a dry run: preview what you'd change to set up Release Autopilot auto-flagging for acme/demo in project default. Don't change anything." max_turns: 20 @@ -192,7 +207,8 @@ tests: # GitHub App not installed, and the user needs an org owner: hand the # user the link, write nothing, and stop. # ------------------------------------------------------------------ - - description: "App not installed, org owner needed: link given, zero writes" + - providers: ["mcp"] + description: "App not installed, org owner needed: link given, zero writes" vars: user_request: "Set up Release Autopilot for my repos." max_turns: 15 @@ -230,7 +246,8 @@ tests: # Writer role on a fresh account: account settings need an admin, so # the account write is never attempted. The repo write may run. # ------------------------------------------------------------------ - - description: "Writer role: account write left for an admin" + - providers: ["mcp"] + description: "Writer role: account write left for an admin" vars: user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." max_turns: 25 @@ -266,7 +283,8 @@ tests: # ------------------------------------------------------------------ # "Change something" at the confirm step is not approval: no writes. # ------------------------------------------------------------------ - - description: "Change something at confirm: zero writes" + - providers: ["mcp"] + description: "Change something at confirm: zero writes" vars: user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." max_turns: 20 @@ -293,7 +311,8 @@ tests: # auto-releasing from the account. The user picks auto-flagging only, # so the only write is an override that turns auto-releasing off. # ------------------------------------------------------------------ - - description: "Already mapped with an unpicked capability: override write only" + - providers: ["mcp"] + description: "Already mapped with an unpicked capability: override write only" vars: user_request: "Set up Release Autopilot auto-flagging for acme/storefront in project default. I don't want auto-releasing." max_turns: 25 @@ -325,7 +344,8 @@ tests: # acme/storefront is mapped to default; the user asks for project web. # The skill must not remap; it points to the settings skill instead. # ------------------------------------------------------------------ - - description: "Mapped to another project: no remap" + - providers: ["mcp"] + description: "Mapped to another project: no remap" vars: user_request: "Set up Release Autopilot auto-flagging for acme/storefront in project web." max_turns: 20 @@ -345,7 +365,8 @@ tests: # Plan mode: release plans are available and empty. The plan is # created before the repo write, and every phase has a clause. # ------------------------------------------------------------------ - - description: "Plan mode: release plan created first, valid phases" + - providers: ["mcp"] + description: "Plan mode: release plan created first, valid phases" vars: user_request: > Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default. @@ -380,3 +401,99 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting metric: confirms_before_writing weight: 3 + + # ------------------------------------------------------------------ + # The user wants a repo whose GitHub organization has no App install. + # No githubConnectUrl appears (the account is installed elsewhere), so + # the skill must say an org owner installs the App, and write nothing. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Repo in an uninstalled GitHub org: org owner install, zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for other-org/app in project default." + max_turns: 15 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:saysOrgOwnerInstall + metric: org_owner_install + weight: 3 + + # ------------------------------------------------------------------ + # An older MCP server flags itself as deprecated. The skill gives the + # install link and stops before asking anything or writing. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Deprecated MCP server: install link, zero writes" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 10 + mock_tool_overrides: + get-member-self: + id: "member-1" + email: "admin@acme.example" + role: "admin" + teams: [] + latestMcp: + deprecated: true + installUrl: "https://mcp.launchdarkly.com/mcp/launchdarkly/install" + serverUrl: "https://mcp.launchdarkly.com/mcp/launchdarkly" + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: | + const text = String(output.response || '') + (output.assistant_turns || []).join('\n'); + const pass = text.includes('mcp.launchdarkly.com/mcp/launchdarkly/install'); + return { pass, score: pass ? 1 : 0, reason: pass ? 'gave the install link' : 'did not give latestMcp.installUrl' }; + metric: gave_install_link + weight: 3 + + # ------------------------------------------------------------------ + # Resume after the GitHub pause. The earlier picks are in the request; + # the App is now installed. No intro, no re-asked capability or + # approval questions, and the confirmed plan maps the repo. + # ------------------------------------------------------------------ + - providers: ["mcp-auto"] + description: "Resume after GitHub pause: no intro, no re-asked picks, maps after confirm" + vars: + user_request: > + Continue Release Autopilot setup. Earlier you recorded my picks: auto-flagging only, approval + required, repo acme/demo in project default, LaunchDarkly managed inference. We paused while + our GitHub org owner installed the App, and that is done now. + max_turns: 20 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noIntro + metric: no_intro + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noPickQuestions + metric: no_repeated_picks + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 3 diff --git a/evals/tests/mock.test.js b/evals/tests/mock.test.js index 4a96127c..df6a6d9e 100644 --- a/evals/tests/mock.test.js +++ b/evals/tests/mock.test.js @@ -12,7 +12,7 @@ const TEMPLATES = require("../mocks/tool-responses.json"); test("createMockState returns fresh empty state each call", () => { const a = createMockState(); const b = createMockState(); - assert.deepEqual(a, { configs: {}, flags: {} }); + assert.deepEqual(a, { configs: {}, flags: {}, raSettings: null, raRepos: {}, raPlans: {} }); a.configs["x"] = {}; assert.deepEqual(b.configs, {}); // independent }); @@ -422,3 +422,33 @@ test("update-ai-config uses template substitution", () => { assert.equal(result.key, "bot"); assert.equal(result.name, "Bot"); }); + +// --------------------------------------------------------------------------- +// Release Autopilot stateful reads +// --------------------------------------------------------------------------- + +test("repo settings read reflects an earlier repo write", () => { + const state = createMockState(); + const tpl = TEMPLATES["get-release-autopilot-repo-settings"]; + assert.ok(renderMockResponse(tpl, { repo: "acme/demo" }, "get-release-autopilot-repo-settings", state).error); + renderMockResponse(TEMPLATES["update-release-autopilot-repo-settings"], { repo: "acme/demo", projectKey: "default", autoReleasing: { enabled: false } }, "update-release-autopilot-repo-settings", state); + const got = renderMockResponse(tpl, { repo: "acme/demo" }, "get-release-autopilot-repo-settings", state); + assert.equal(got.projectKey, "default"); + assert.equal(got.autoReleasing.enabled, false); +}); + +test("account settings read merges an earlier account write", () => { + const state = createMockState(); + renderMockResponse(TEMPLATES["update-release-autopilot-settings"], { autoFlagging: { enabled: false } }, "update-release-autopilot-settings", state); + const got = renderMockResponse(TEMPLATES["get-release-autopilot-settings"], {}, "get-release-autopilot-settings", state); + assert.equal(got.autoFlagging.enabled, false); + assert.equal(got.autoFlagging.approvalRequired, true); +}); + +test("release plans list reflects a created plan", () => { + const state = createMockState(); + renderMockResponse(TEMPLATES["create-release-plan"], { projectKey: "default", environmentKey: "production", key: "p", name: "P", progressiveRelease: { phases: [] } }, "create-release-plan", state); + const got = renderMockResponse(TEMPLATES["list-release-plans"], { projectKey: "default", environmentKey: "production" }, "list-release-plans", state); + assert.equal(got.totalCount, 1); + assert.equal(got.items[0].key, "p"); +}); diff --git a/evals/tests/provider-match-option.test.js b/evals/tests/provider-match-option.test.js index f113f5b1..0310964a 100644 --- a/evals/tests/provider-match-option.test.js +++ b/evals/tests/provider-match-option.test.js @@ -3,7 +3,7 @@ const assert = require("node:assert/strict"); // matchOption resolves a scripted ask-question answer to an option id, so a // test can steer by label when the agent invents the ids. -const { matchOption } = require("../providers/claude-skill-agent-sdk"); +const { matchOption, pickAnswer } = require("../providers/claude-skill-agent-sdk"); const options = [ { id: "af", label: "Auto-flagging (Recommended)" }, @@ -25,3 +25,19 @@ test("passes an unmatched answer through", () => { test("tolerates missing options", () => { assert.equal(matchOption("x", undefined), "x"); }); + +test("order mode takes the answer at the call index", () => { + assert.deepEqual(pickAnswer([["x"], ["Auto-releasing"]], 1, options, false, new Set()), ["ar"]); +}); + +test("match mode skips answers that fit no option, and uses each once", () => { + const used = new Set(); + const answers = [["Looks good"], ["Auto-flagging"], ["Auto-flagging"]]; + assert.deepEqual(pickAnswer(answers, 0, options, true, used), ["af"]); + assert.deepEqual(pickAnswer(answers, 1, options, true, used), ["af"]); + assert.equal(pickAnswer(answers, 2, options, true, used), undefined); +}); + +test("match mode returns undefined when nothing fits", () => { + assert.equal(pickAnswer([["Looks good"]], 0, options, true, new Set()), undefined); +}); diff --git a/evals/tools/definitions.json b/evals/tools/definitions.json index 18c930b1..8b540e99 100644 --- a/evals/tools/definitions.json +++ b/evals/tools/definitions.json @@ -686,14 +686,14 @@ "required": ["projectKey", "environmentKey", "experimentKey"] } }, - {"name": "get-member-self", "description": "Return the member ID, profile, role, and team memberships for the currently authenticated user.", "input_schema": {"type": "object", "properties": {}}}, - {"name": "get-release-autopilot-settings", "description": "Get account-wide Release Autopilot settings: whether auto-flagging and auto-releasing are on, and whether auto-flagging pull requests need approval. These are the master gate. Do not use autoCleanup.", "input_schema": {"type": "object", "properties": {}}}, - {"name": "update-release-autopilot-settings", "description": "Update account-wide Release Autopilot settings (auto-flagging and auto-releasing). Send only what changed. Requires permission to update account Release Autopilot settings. Do not send autoCleanup.", "input_schema": {"type": "object", "properties": {"autoFlagging": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}, "autoReleasing": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}}}}, - {"name": "list-release-autopilot-github-repos", "description": "List GitHub repositories on this account's LaunchDarkly GitHub App install, with name (owner/name) and numeric githubRepoId. When githubConnectUrl is present, give that link to the user and retry after they finish.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}}}}, - {"name": "list-release-autopilot-repo-settings", "description": "List Release Autopilot settings for GitHub repositories already mapped to a LaunchDarkly project. Each automation is the effective value. githubRepoId is numeric; repo names are not included.", "input_schema": {"type": "object", "properties": {}}}, - {"name": "get-release-autopilot-repo-settings", "description": "Get Release Autopilot settings for one GitHub repository. Pass repo as owner/name, or githubRepoId. 404 means the repo is not mapped yet.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}, "projectKey": {"type": "string"}}}}, - {"name": "update-release-autopilot-repo-settings", "description": "Map a GitHub repository to a LaunchDarkly project and optionally override auto-flagging or auto-releasing. Pass repo as owner/name. projectKey is required on the first map. An omitted automation inherits the account setting. Do not send autoCleanup.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}, "projectKey": {"type": "string"}, "autoFlagging": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}, "autoReleasing": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}}}}, - {"name": "delete-release-autopilot-repo-settings", "description": "Unmap a GitHub repository from Release Autopilot.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}}}}, - {"name": "list-release-plans", "description": "List environment-scoped release plans. A release plan is a reusable progressive-release recipe for one environment. It replaces project-level release policies.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "includeIncomplete": {"type": "boolean"}}, "required": ["projectKey", "environmentKey"]}}, - {"name": "create-release-plan", "description": "Create an environment-scoped release plan, a reusable progressive-release recipe. Does not start a release. PHASES: 1-10; every phase needs 1-10 clauses AND 1-10 steps. An \"everyone\" phase still needs a clause, e.g. { attribute: \"kind\", op: \"in\", values: [\"user\"], contextKind: \"user\" }. STEPS: allocation is per-100k (5000 = 5%, 100000 = 100%); durationMillis is the hold before the next step. METRICS: keyed object; {} = progressive. advancementMode defaults to manual.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "key": {"type": "string"}, "name": {"type": "string"}, "description": {"type": "string"}, "progressiveRelease": {"type": "object", "properties": {"randomizationUnit": {"type": "string"}, "advancementMode": {"type": "string", "enum": ["manual", "automatic"]}, "cleanupBehavior": {"type": "string", "enum": ["leave_rules_in_place", "remove_rules_and_update_default_rule"]}, "phases": {"type": "array", "items": {"type": "object", "properties": {"description": {"type": "string"}, "clauses": {"type": "array", "items": {"type": "object", "properties": {"attribute": {"type": "string"}, "op": {"type": "string", "enum": ["in", "endsWith", "startsWith", "matches", "contains", "lessThan", "lessThanOrEqual", "greaterThan", "greaterThanOrEqual", "before", "after", "semVerEqual", "semVerLessThan", "semVerGreaterThan", "segmentMatch"]}, "values": {"type": "array", "items": {}}, "contextKind": {"type": "string"}, "negate": {"type": "boolean"}}, "required": ["attribute", "op", "values"]}}, "steps": {"type": "array", "items": {"type": "object", "properties": {"allocation": {"type": "number"}, "durationMillis": {"type": "number"}}, "required": ["allocation", "durationMillis"]}}}, "required": ["clauses", "steps"]}}, "metrics": {"type": "object"}}, "required": ["randomizationUnit", "phases"]}}, "required": ["projectKey", "environmentKey", "key", "name", "progressiveRelease"]}} + {"name": "get-member-self", "description": "Return the member ID, profile, role, and team memberships for the currently authenticated user.", "input_schema": {"type": "object", "properties": {}}, "scope": "release-autopilot"}, + {"name": "get-release-autopilot-settings", "description": "Get account-wide Release Autopilot settings: whether auto-flagging and auto-releasing are on, and whether auto-flagging pull requests need approval. These are the master gate. Do not use autoCleanup.", "input_schema": {"type": "object", "properties": {}}, "scope": "release-autopilot"}, + {"name": "update-release-autopilot-settings", "description": "Update account-wide Release Autopilot settings (auto-flagging and auto-releasing). Send only what changed. Requires permission to update account Release Autopilot settings. Do not send autoCleanup.", "input_schema": {"type": "object", "properties": {"autoFlagging": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}, "autoReleasing": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}}}, "scope": "release-autopilot"}, + {"name": "list-release-autopilot-github-repos", "description": "List GitHub repositories on this account's LaunchDarkly GitHub App install, with name (owner/name) and numeric githubRepoId. When githubConnectUrl is present, give that link to the user and retry after they finish.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}}}, "scope": "release-autopilot"}, + {"name": "list-release-autopilot-repo-settings", "description": "List Release Autopilot settings for GitHub repositories already mapped to a LaunchDarkly project. Each automation is the effective value. githubRepoId is numeric; repo names are not included.", "input_schema": {"type": "object", "properties": {}}, "scope": "release-autopilot"}, + {"name": "get-release-autopilot-repo-settings", "description": "Get Release Autopilot settings for one GitHub repository. Pass repo as owner/name, or githubRepoId. 404 means the repo is not mapped yet.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}, "projectKey": {"type": "string"}}}, "scope": "release-autopilot"}, + {"name": "update-release-autopilot-repo-settings", "description": "Map a GitHub repository to a LaunchDarkly project and optionally override auto-flagging or auto-releasing. Pass repo as owner/name. projectKey is required on the first map. An omitted automation inherits the account setting. Do not send autoCleanup.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}, "projectKey": {"type": "string"}, "autoFlagging": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}, "autoReleasing": {"type": "object", "properties": {"enabled": {"type": "boolean"}, "approvalRequired": {"type": "boolean"}}}}}, "scope": "release-autopilot"}, + {"name": "delete-release-autopilot-repo-settings", "description": "Unmap a GitHub repository from Release Autopilot.", "input_schema": {"type": "object", "properties": {"repo": {"type": "string"}, "githubRepoId": {"type": "number"}}}, "scope": "release-autopilot"}, + {"name": "list-release-plans", "description": "List environment-scoped release plans. A release plan is a reusable progressive-release recipe for one environment. It replaces project-level release policies.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "includeIncomplete": {"type": "boolean"}}, "required": ["projectKey", "environmentKey"]}, "scope": "release-autopilot"}, + {"name": "create-release-plan", "description": "Create an environment-scoped release plan, a reusable progressive-release recipe. Does not start a release. PHASES: 1-10; every phase needs 1-10 clauses AND 1-10 steps. An \"everyone\" phase still needs a clause, e.g. { attribute: \"kind\", op: \"in\", values: [\"user\"], contextKind: \"user\" }. STEPS: allocation is per-100k (5000 = 5%, 100000 = 100%); durationMillis is the hold before the next step. METRICS: keyed object; {} = progressive. advancementMode defaults to manual.", "input_schema": {"type": "object", "properties": {"projectKey": {"type": "string"}, "environmentKey": {"type": "string"}, "key": {"type": "string"}, "name": {"type": "string"}, "description": {"type": "string"}, "progressiveRelease": {"type": "object", "properties": {"randomizationUnit": {"type": "string"}, "advancementMode": {"type": "string", "enum": ["manual", "automatic"]}, "cleanupBehavior": {"type": "string", "enum": ["leave_rules_in_place", "remove_rules_and_update_default_rule"]}, "phases": {"type": "array", "items": {"type": "object", "properties": {"description": {"type": "string"}, "clauses": {"type": "array", "items": {"type": "object", "properties": {"attribute": {"type": "string"}, "op": {"type": "string", "enum": ["in", "endsWith", "startsWith", "matches", "contains", "lessThan", "lessThanOrEqual", "greaterThan", "greaterThanOrEqual", "before", "after", "semVerEqual", "semVerLessThan", "semVerGreaterThan", "segmentMatch"]}, "values": {"type": "array", "items": {}}, "contextKind": {"type": "string"}, "negate": {"type": "boolean"}}, "required": ["attribute", "op", "values"]}}, "steps": {"type": "array", "items": {"type": "object", "properties": {"allocation": {"type": "number"}, "durationMillis": {"type": "number"}}, "required": ["allocation", "durationMillis"]}}}, "required": ["clauses", "steps"]}}, "metrics": {"type": "object"}}, "required": ["randomizationUnit", "phases"]}}, "required": ["projectKey", "environmentKey", "key", "name", "progressiveRelease"]}, "scope": "release-autopilot"} ] diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index a8bd5aeb..46745105 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -28,10 +28,9 @@ This skill requires the remotely hosted LaunchDarkly MCP server. **Required for auto-releasing:** - `list-release-policies` -- `list-metrics` **Used when available:** -- `list-release-plans` / `create-release-plan`: environment release plans (see [Detect the release mode](#detect-the-release-mode)) +- `list-release-plans` / `create-release-plan` / `list-metrics`: environment release plans (see [Detect the release mode](#detect-the-release-mode)) - `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. @@ -46,7 +45,7 @@ Pick the mode from the user's request before Step 0: |------|---------|--------------| | **Setup** | "set up", "onboard", "turn on" | The full flow | | **Preview** | "preview", "what would you change", "dry run", or a member who cannot write (see Step 1) | Run Steps 1–7, show the plan, and stop. Make no writes. End with the plan as a checklist the user can hand to an admin | -| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Keep the earlier mode: a resumed Preview stays Preview. Re-read live state in Step 1 and say in one sentence what is already done. Use the earlier picks if they are still in the conversation; if not, ask them again with live state as the default. Writes that landed before a Step 8 failure show as already set up, so the new plan lists only what is left | +| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Keep the earlier mode: a resumed Preview stays Preview. Re-read live state in Step 1 and say in one sentence what is already done. Use the earlier picks if they are still in the conversation; if not, ask them again with live state as the default. Writes that landed before a Step 8 failure show as already set up, so the new plan lists only what is left. Resume still stops at Step 7 and waits for "Looks good" | Every mode is safe to run again. A re-run reads live state and proposes only the writes that are still needed. @@ -54,18 +53,16 @@ Every mode is safe to run again. A re-run reads live state and proposes only the 1. **Only set up what they picked.** Release setup matters only for auto-releasing. If the user picks auto-flagging only, skip Step 6. 2. **Least privilege by default.** Recommend auto-flagging with approval required. Auto-releasing is opt-in and you must say it can ship code. -3. **Gather, confirm, then write.** No write happens before [Step 7](#step-7-confirm-the-plan). The confirmation lists every write with its current and proposed state. +3. **Gather, confirm, then write.** No write happens before [Step 7](#step-7-confirm-the-plan). The confirmation lists every write with its current and proposed state. This holds in every mode, including Resume and requests that already state every answer: a restated pick is an answer, never approval. Step 1's reads, including the role check, always run first. 4. **Don't ask what the account already answers.** If live state already settles a question, state the answer and move on. 5. **Never act in the user's browser.** GitHub install, GitHub authorization, release policy edits, and bring-your-own-key setup are browser steps. Give the exact link and wait for the user to say they finished. 6. **Only touch what the user named.** Never map every repo on the install. Never create release plans in environments the user did not pick. ## Speaking to the user -- Use product words: "auto-flagging", "auto-releasing", "repository", "project", "release policy", "release plan", "approval". -- Never show tool names or field names such as `update-release-autopilot-repo-settings`, `githubRepoId`, `enabledOverride`, or `autoReleasing.enabled`. Say "repository ID", "turned off for this repository", or "auto-releasing is on". This applies to every message, including questions, the admin checklist, and the final summary: describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value. -- "Policy mode" and "plan mode" are names for this skill only. Tell the user "your account uses release policies" or "your account uses release plans". -- If a verification read disagrees with a write, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. Do not name the tools involved. -- Keep each message to the current step. Do not narrate tool calls. +- Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("policy mode"); say "your account uses release policies" and "Verification", not "Step 9". +- Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. +- Keep each message to the current step, and do not narrate tool calls. ## Workflow @@ -79,7 +76,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ### Step 0: Welcome the user -In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." In Resume mode, skip the intro. +Decide the mode from the user's first message before you write anything. **In Resume mode, skip this step entirely**: do not show the intro or any part of it, and go straight to Step 1. In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." > **Release Autopilot Setup** > @@ -160,10 +157,7 @@ On "Done", call `list-release-autopilot-github-repos` again. If it still returns If a repo the user wants is not on the list, its owner decides the fix: a listed owner means the App needs access to that repo, and a new owner means an org owner must install the App there (no `githubConnectUrl` appears for this case). See [references/edge-cases.md](references/edge-cases.md). Wait for "Done" and list again. 2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. -3. **Check existing mappings** with the join from Step 1. An unmapped repo returns 404 from `get-release-autopilot-repo-settings`; that is the expected answer for a new repo, not an error. - - Mapped to the chosen project, and its effective capabilities match the user's picks: tell the user it is already set up, and write nothing. - - Mapped to the chosen project, but a capability the user did not pick is effectively on: record a repo override that turns it off, as its own line in Step 7. - - Mapped to a different project: do not remap it in this flow. Remapping changes where its flags are created. Tell the user to use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) to unmap it first, then run this step again. +3. **Check existing mappings** with the join from Step 1 (an unmapped repo returns 404, which is expected). Already mapped to the chosen project with matching capabilities: write nothing. Mapped to the chosen project with an unpicked capability on: record an override that turns it off. Mapped to a different project: never remap here; send the user to the settings skill to unmap first. Details: [references/edge-cases.md](references/edge-cases.md#existing-mappings). 4. **Say what starts when.** If the account already has a capability on, mapping a repo starts that automation in the repo as soon as Step 8 maps it. ### Step 5: Choose inference @@ -230,7 +224,7 @@ List every write in the order you will make it. For each write, state the curren If there are no writes, say so, list the follow-ups, and stop. -**Preview mode:** present the list as a checklist under the heading "Changes an admin would make", then stop. Do not ask for confirmation. +**Preview mode:** present the list as a checklist under the heading "Changes an admin would make", one line per change, then stop. Do not ask for confirmation. **Setup mode:** use `AskUserQuestion` with header "Confirm": diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md index ddafc444..d3ed12e7 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -18,6 +18,14 @@ | Bring your own key chosen | Browser step only. Record it as reported, never as verified | | No writes needed | Say so, list follow-ups, and stop | +## Existing mappings + +| Current mapping | Action | +|-----------------|--------| +| Mapped to the chosen project, effective capabilities match the user's picks | Tell the user it is already set up. Write nothing | +| Mapped to the chosen project, a capability the user did not pick is effectively on | Record a repo override that turns it off, as its own line in Step 7 | +| Mapped to a different project | Do not remap. Remapping changes where its flags are created. The user unmaps it with [`launchdarkly-release-autopilot-settings`](../../launchdarkly-release-autopilot-settings/SKILL.md), then maps it in this flow | + ## Example flows **Fresh account: auto-flagging only, managed inference.** From 13aa205a7b4028daa6ec4052530af82ce31e9549 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 19:07:49 -0700 Subject: [PATCH 16/30] refactor(release-autopilot-onboarding): shorten and align with repo conventions Move Preview/Resume details and bring-your-own-key steps to references/, shorten the description, intro, and settings-skill note, merge workflow boilerplate, add the template's Edge Cases table, trim What NOT to Do to rules not stated elsewhere, and drop example flows that the evals cover. Bump the Claude Code plugin to 1.0.3 and the Cursor plugin to 1.0.4 for the new skill. All 13 eval cases still pass. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude-plugin/marketplace.json | 2 +- .claude-plugin/plugin.json | 2 +- .cursor-plugin/plugin.json | 2 +- CHANGELOG.md | 2 +- skills.json | 2 +- .../SKILL.md | 71 ++++++++----------- .../references/byok.md | 11 +++ .../references/edge-cases.md | 19 +---- .../references/modes.md | 18 +++++ 9 files changed, 66 insertions(+), 63 deletions(-) create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index c699096f..18c3ca14 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -11,7 +11,7 @@ "name": "launchdarkly", "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics", "source": ".", - "version": "1.0.2", + "version": "1.0.3", "author": { "name": "LaunchDarkly", "email": "support@launchdarkly.com" diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 92a9ffbd..81d1293a 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "launchdarkly", - "version": "1.0.2", + "version": "1.0.3", "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics", "author": { "name": "LaunchDarkly", diff --git a/.cursor-plugin/plugin.json b/.cursor-plugin/plugin.json index 75a4df18..591d745a 100644 --- a/.cursor-plugin/plugin.json +++ b/.cursor-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "launchdarkly", - "version": "1.0.3", + "version": "1.0.4", "description": "LaunchDarkly agent skills and mcp server for feature flag management, AgentControl, and skill authoring", "author": { "name": "LaunchDarkly", diff --git a/CHANGELOG.md b/CHANGELOG.md index e14ce58f..22046f60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this repository will be documented in this file. ## Unreleased -- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. +- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. - evals: add the `release-autopilot-onboarding` suite (13 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. diff --git a/skills.json b/skills.json index fc5a6980..f68af77e 100644 --- a/skills.json +++ b/skills.json @@ -421,7 +421,7 @@ }, { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review how each environment releases (release policies today, environment release plans by GA), then verify. Also previews the changes without making them, and resumes a setup that paused. Use when the user wants to set up, onboard to, get started with, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (managed by default, or bring your own key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", "version": "0.2.0-experimental", "license": "Apache-2.0", diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 46745105..3423cff4 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -1,6 +1,6 @@ --- name: launchdarkly-release-autopilot-onboarding -description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick the capabilities you want (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (LaunchDarkly managed by default, or bring your own key), review how each environment releases (release policies today, environment release plans by GA), then verify. Also previews the changes without making them, and resumes a setup that paused. Use when the user wants to set up, onboard to, get started with, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (managed by default, or bring your own key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: @@ -14,7 +14,7 @@ You're using a skill that walks a user through setting up Release Autopilot. The **This is a step-by-step wizard.** Ask one question at a time and wait for each answer. Gather every decision first, confirm one plan, and only then write. Release Autopilot opens and changes pull requests, and auto-releasing can ship code, so every write is production automation. -This skill reuses the rules of [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) for every settings read and write: least privilege, `owner/name` only, never `autoCleanup`, and the current → proposed wording with blast radius. One difference is deliberate: the settings skill confirms each write on its own, and this skill confirms every write in one itemized plan at [Step 7](#step-7-confirm-the-plan). That itemized confirmation satisfies the settings skill's confirm-before-write rule. For anything else, the settings skill wins. +Follow the write rules of [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) (least privilege, `owner/name` only, never `autoCleanup`, current → proposed with blast radius). Its one-at-a-time confirmations are replaced by the single itemized confirmation at [Step 7](#step-7-confirm-the-plan). ## Prerequisites @@ -44,8 +44,8 @@ Pick the mode from the user's request before Step 0: | Mode | Trigger | What changes | |------|---------|--------------| | **Setup** | "set up", "onboard", "turn on" | The full flow | -| **Preview** | "preview", "what would you change", "dry run", or a member who cannot write (see Step 1) | Run Steps 1–7, show the plan, and stop. Make no writes. End with the plan as a checklist the user can hand to an admin | -| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Keep the earlier mode: a resumed Preview stays Preview. Re-read live state in Step 1 and say in one sentence what is already done. Use the earlier picks if they are still in the conversation; if not, ask them again with live state as the default. Writes that landed before a Step 8 failure show as already set up, so the new plan lists only what is left. Resume still stops at Step 7 and waits for "Looks good" | +| **Preview** | "preview", "what would you change", "dry run", or a reader (see Step 1) | Run Steps 1–7 and end with the plan as a checklist for an admin. Make no writes | +| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Skip the intro, re-read live state, and continue at the first unfinished step. Still stops at Step 7 and waits for "Looks good". Details: [references/modes.md](references/modes.md) | Every mode is safe to run again. A re-run reads live state and proposes only the writes that are still needed. @@ -66,32 +66,19 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Workflow -**Progression rule:** When a step is complete, go to the next step. Do not stop between steps unless the step tells you to ask or wait. - -**Response scoping:** During this flow, treat user messages as answers to the current question. Do not take unrelated actions until the flow is finished. +When a step is complete, go to the next one unless it says to ask or wait. Treat user messages as answers to the current question, and track every decision for Step 7. **Asking questions:** When a question has options, use `AskUserQuestion` with the labels and descriptions given below. Ask free-text questions as a normal message. If the user's request already answers a question explicitly, such as naming the repo, the project, or the capabilities, use that answer and say so instead of asking. Never infer the answer to the Step 7 confirmation. -**State tracking:** Track every decision. Step 7 re-states all of them. - ### Step 0: Welcome the user Decide the mode from the user's first message before you write anything. **In Resume mode, skip this step entirely**: do not show the intro or any part of it, and go straight to Step 1. In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." > **Release Autopilot Setup** > -> I'll set up Release Autopilot in a few short steps: -> -> 1. **Check your account**: I'll read your current Release Autopilot settings. -> 2. **Pick capabilities**: choose auto-flagging, auto-releasing, or both. -> 3. **Connect GitHub**: install or authorize the LaunchDarkly GitHub App. -> 4. **Map repositories**: choose which repos Release Autopilot watches, and which LaunchDarkly project each one uses. -> 5. **Choose inference**: run the agents on LaunchDarkly managed inference or on your own model provider key. -> 6. **Review release setup** (auto-releasing only): check how each environment rolls out a change. -> 7. **Confirm the plan**: I'll show every change before I make it. -> 8. **Apply and verify**: I'll make the changes and check the result. +> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose inference, and review release setup if you want auto-releasing. > -> Nothing changes until you approve the plan in step 7. +> Nothing changes until you approve the full plan. Then go to Step 1. Skip any step that the account state already completes, and say why in one line. @@ -107,9 +94,7 @@ Run these reads. Do not write anything. 3. `list-release-autopilot-repo-settings`: repos that are already mapped. 4. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. -`list-release-autopilot-repo-settings` returns only `githubRepoId` and `projectKey`, not repo names. Join it to the install list on `githubRepoId` before you show the user which repos are mapped. A mapped `githubRepoId` with no match on the install list is a repo the App can no longer see, or one this member cannot see: tell the user, for example, "One more repository (ID 999) is mapped to project default but is no longer on the GitHub install; I'll leave it alone." Do not write the field name. Do not change that repo in this flow. - -Release Autopilot's agents need generative AI turned on for the organization. There is no MCP tool to read that setting. If a later step fails with a generative AI or Vega eligibility error, tell the user that an Admin or Owner must select **Enable generative AI use cases** in Organization settings > Security, then stop. +Join mapped repos to the install list by repository ID to get their names. For a mapped ID that is not on the install, tell the user, for example, "One more repository (ID 999) is mapped to project default but is no longer on the GitHub install; I'll leave it alone." Do not write the field name or change that repo. If a later step fails with a generative AI eligibility error, see [references/edge-cases.md](references/edge-cases.md). Summarize what is already set up in two or three sentences. If the account already has a capability on or any repo mapped, ask: @@ -131,8 +116,6 @@ Ask about approval only if the user picked auto-flagging **and** the account doe - label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before Release Autopilot changes code. This is an account setting, so it applies to every mapped repo" - label: "No approval", description: "Release Autopilot changes PRs without an approval step, in every mapped repo" -Do not offer auto-cleanup. It is not part of the Release Autopilot settings API yet. - If the user picks neither capability, stop and tell them there is nothing to set up. ### Step 3: Connect GitHub @@ -167,18 +150,12 @@ Release Autopilot runs its agents on Vega. Bring your own key is optional: a pro - label: "LaunchDarkly managed", description: "The default if you choose nothing. The agents run on Vega inference that LaunchDarkly hosts. No setup" - label: "Bring your own key", description: "The agents run on your own Anthropic or OpenAI account. Your provider bills you for usage" -For "Bring your own key", there is no MCP tool for inference settings yet, so this is a browser step. For each project from Step 4, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). On that page they: - -1. Select bring your own key. If the account has no Anthropic or OpenAI key yet, the page links to the integration form to add one. Keys are account-wide, so later projects can reuse them. -2. Keep or change the suggested default and small models. -3. Save. LaunchDarkly checks the key with the provider before it stores the settings, and shows the cause if the check fails. - -Then ask header "Inference": +For "Bring your own key", this is a browser step (no MCP tool reads or writes inference settings). For each project, follow [references/byok.md](references/byok.md), then ask header "Inference": - label: "Done", description: "I saved the inference settings for every project" - label: "Use managed for now", description: "Skip this. I'll switch to my own key later" -You cannot read inference settings, so record the answer as the user's statement, not as verified. +Record the answer as reported by the user, not as verified. ### Step 6: Review release setup (auto-releasing only) @@ -217,7 +194,7 @@ List every write in the order you will make it. For each write, state the curren 1. **Metrics and release plans** (plan mode only): one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." -3. **Account settings**: only fields that change. For example, "Auto-flagging: off → on, approval required. Mapped repos that inherit this setting can start getting auto-flagging PRs." If auto-releasing turns on, say: "Auto-releasing: off → on. This can ship code." +3. **Account settings**: only fields that change. If auto-releasing turns on, say it can ship code. For a member who is not an admin or owner, list account writes under "Needs an admin" instead. 4. **Browser steps the user reported done**: GitHub and inference, marked as reported, not verified. 5. **Follow-ups for the user**: release policies to add, metric events to instrument. @@ -257,20 +234,30 @@ Report a short checklist with each item marked verified, reported by the user, o - Open a pull request in a mapped repo. Release Autopilot reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. - If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. +## Edge Cases + +| Situation | Action | +|-----------|--------| +| Every Release Autopilot tool returns 404 | The account is not in the alpha. Stop and tell the user | +| Member role is `reader` or `no_access` | Preview mode. End with a checklist for an admin | +| Release plans return 403 | Policy mode. Create no plans | +| Repo not on the install list | Explain the GitHub fix by owner, then list again | + +More cases: [references/edge-cases.md](references/edge-cases.md). + ## What NOT to Do -- Don't write before Step 7's "Looks good", and never write in Preview mode. -- Don't turn on auto-releasing unless the user picked it. -- Don't turn off approval for auto-flagging unless the user picked "No approval". -- Don't map repos the user did not pick, iterate over the install list, or remap a repo to another project. -- Don't install or authorize GitHub, change release policies, or set up a provider key on the user's behalf. -- Don't send `autoCleanup`. -- Don't report a browser step as verified. +- Don't write before Step 7's "Looks good", or at all in Preview mode. +- Don't turn on auto-releasing, or turn off approval, unless the user picked it. +- Don't map unpicked repos, iterate over the install list, or remap a repo to another project. +- Don't act in the user's browser, change release policies, or report a browser step as verified. ## References - [references/release-plans.md](references/release-plans.md): creating environment release plans (plan mode) -- [references/edge-cases.md](references/edge-cases.md): edge cases and example flows +- [references/modes.md](references/modes.md): Preview and Resume details +- [references/byok.md](references/byok.md): bring-your-own-key setup +- [references/edge-cases.md](references/edge-cases.md): edge cases and existing mappings - [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md): settings rules, write confirmations, and PR diagnosis - [Vega inference settings](https://launchdarkly.com/docs/home/getting-started/vega#vega-inference-settings) - [Guarded rollouts](https://launchdarkly.com/docs/home/releases/guarded-rollouts) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md new file mode 100644 index 00000000..bb7de7eb --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md @@ -0,0 +1,11 @@ +# Bring your own key + +Bring your own key is optional. A project that does not set it up uses LaunchDarkly managed Vega inference. No MCP tool reads or writes inference settings, so this is a browser step. + +For each project, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). On that page they: + +1. Select bring your own key. If the account has no Anthropic or OpenAI key yet, the page links to the integration form to add one. Keys are account-wide, so later projects can reuse them. +2. Keep or change the suggested default and small models. +3. Save. LaunchDarkly checks the key with the provider before it stores the settings, and shows the cause if the check fails. + +Record the result as reported by the user. It cannot be verified. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md index d3ed12e7..dbf6f563 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -1,4 +1,4 @@ -# Edge cases and example flows +# Edge cases ## Edge cases @@ -10,11 +10,12 @@ | A write returns 403 | Report which write failed, list the remaining writes for an admin, and stop | | `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | | Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | -| Mapped repo ID not on the install list | Show it by ID with its project. Do not change it | +| Mapped repo ID not on the install list | Mapped-repo reads return only repository IDs, so join them to the install list. An ID with no match is a repo the App can no longer see, or one this member cannot see. Show it by ID with its project. Do not change it | | Repo already mapped to a different project | Do not remap here. The user unmaps it with the settings skill, then maps it in this flow | | `list-release-plans` returns 403 `Plan does not allow this operation` | Policy mode. Show release policies and their warnings, create no plans, and keep auto-releasing off for the repo where a policy is missing or incomplete | | Auto-releasing picked but no metrics exist (plan mode) | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | | Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | +| Generative AI or Vega eligibility error | Release Autopilot's agents need generative AI turned on, and no MCP tool reads that setting. An Admin or Owner selects **Enable generative AI use cases** in Organization settings > Security. Stop | | Bring your own key chosen | Browser step only. Record it as reported, never as verified | | No writes needed | Say so, list follow-ups, and stop | @@ -25,17 +26,3 @@ | Mapped to the chosen project, effective capabilities match the user's picks | Tell the user it is already set up. Write nothing | | Mapped to the chosen project, a capability the user did not pick is effectively on | Record a repo override that turns it off, as its own line in Step 7 | | Mapped to a different project | Do not remap. Remapping changes where its flags are created. The user unmaps it with [`launchdarkly-release-autopilot-settings`](../../launchdarkly-release-autopilot-settings/SKILL.md), then maps it in this flow | - -## Example flows - -**Fresh account: auto-flagging only, managed inference.** -Step 1 → admin, nothing set up, install list has repos. Step 2 → auto-flagging, approval required. Step 3 → skipped silently. Step 4 → maps `acme/web` to project `web`. Step 5 → managed. Step 6 → skipped. Step 7 → two writes: map `acme/web` with auto-releasing off for the repo, then account auto-flagging on with approval. Steps 8–9 → applied and verified. - -**Account already on, policy mode, missing policy.** -Step 1 → both capabilities on, approval required, 3 repos mapped; "Add more". Step 2 → both capabilities; approval question skipped. Step 3 → skipped silently. Step 4 → maps `acme/demo` to `demo`. Step 5 → managed. Step 6 → release plans return 403, so policy mode; `missing_policy`. Step 7 → one write: map `acme/demo` with auto-flagging on and auto-releasing off for the repo; follow-up: add a release policy. Steps 8–9 → applied and verified. - -**Reader: preview only.** -Step 1 → role `reader`, so Preview mode. Steps 2–6 as usual. Step 7 → "Changes an admin would make" checklist. No writes. - -**Org owner needed: pause, then resume.** -Step 3 → `flow=install`, user selects "Need an org owner". Summarize and stop. Later, "continue Release Autopilot setup" → Resume mode: Step 1 shows the App installed, and the flow continues at Step 4 with the earlier picks. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md new file mode 100644 index 00000000..e15b50ef --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md @@ -0,0 +1,18 @@ +# Preview and Resume + +## Preview + +- Run Steps 1–7 with live reads, and make no writes. +- In Step 1, leave out the "Change one setting" option, because that hand-off writes. +- In Step 7, show the plan as a checklist under "Changes an admin would make", one line per change, and stop without asking for confirmation. +- A member whose role is `reader` or `no_access` is always in Preview. + +## Resume + +- Keep the earlier mode: a resumed Preview stays Preview. +- Skip Step 0 entirely. Re-read live state in Step 1, and say in one sentence what is already done. +- Use the earlier picks if they are still in the conversation. If not, ask them again with live state as the default. +- Writes that landed before a Step 8 failure show as already set up, so the new plan lists only what is left. +- A restated pick is an answer, never approval. Resume still stops at Step 7 and waits for "Looks good". + +Every mode is safe to run again: a re-run reads live state and proposes only the writes that are still needed. From 1d971053745b66838c85777e2c14193dad864f86 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 19:46:24 -0700 Subject: [PATCH 17/30] =?UTF-8?q?fix(release-autopilot-onboarding):=20iter?= =?UTF-8?q?ation=206=20=E2=80=94=20partial=20setup,=20short=20previews,=20?= =?UTF-8?q?GA-ready=20evals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lead with 'Partly set up: waiting on an admin' when a change was left for an admin or failed, keep previews to the checklist, repeat the GitHub link in the pause summary, never announce mode names or guess at causes, and rename internal mode headings to product words. Evals: default release-plans mock is the GA shape (the policy case sets today's 403), a new case covers a 403 on the account write, failed writes are no longer recorded by the stateful mocks, and the explicit preview runs without a forced slash invocation. 14 cases pass. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- evals/mocks/tool-responses.json | 2 +- evals/providers/_mock.js | 6 +- .../assertions.js | 42 ++++++++++ .../promptfooconfig.yaml | 83 +++++++++++++++---- evals/tests/mock.test.js | 7 ++ .../SKILL.md | 40 +++++---- .../references/edge-cases.md | 4 +- .../references/release-plans.md | 2 +- 9 files changed, 148 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 22046f60..0ce02f13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to this repository will be documented in this file. - Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. -- evals: add the `release-autopilot-onboarding` suite (13 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. +- evals: add the `release-autopilot-onboarding` suite (14 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. - Rename `factory/launchdarkly-factory-settings` to `release-autopilot/launchdarkly-release-autopilot-settings` and switch it to the `*-release-autopilot-*` MCP tools (Claude Code plugin 1.0.2, Cursor plugin 1.0.3). Requires a hosted MCP server that exposes the new tool names. diff --git a/evals/mocks/tool-responses.json b/evals/mocks/tool-responses.json index d616a93c..9bfea992 100644 --- a/evals/mocks/tool-responses.json +++ b/evals/mocks/tool-responses.json @@ -599,6 +599,6 @@ "get-release-autopilot-repo-settings": {"error": "LaunchDarkly API error: 404: Factory settings for this repository were not found"}, "update-release-autopilot-repo-settings": {"projectKey": "{{projectKey}}", "autoFlagging": {"enabled": true, "approvalRequired": true}, "autoReleasing": {"enabled": false, "enabledOverride": true}, "version": 1}, "delete-release-autopilot-repo-settings": {"deleted": true}, - "list-release-plans": {"error": "LaunchDarkly API error on GET /projects/{{projectKey}}/environments/{{environmentKey}}/release-plans: 403: {\"code\":\"forbidden\",\"message\":\"Plan does not allow this operation\"}"}, + "list-release-plans": {"items": [], "totalCount": 0}, "create-release-plan": {"key": "new-plan", "environmentKey": "{{environmentKey}}", "created": true} } diff --git a/evals/providers/_mock.js b/evals/providers/_mock.js index ce5a12a2..92d89c64 100644 --- a/evals/providers/_mock.js +++ b/evals/providers/_mock.js @@ -179,7 +179,11 @@ function renderReleaseAutopilot(template, input, toolName, state, replacements) function renderMockResponse(template, input, toolName, state) { const replacements = buildReplacements(input); - if (state && state.raRepos) { + // An error template on a write (for example a 403 override) wins over + // stateful handling, so the failed write is not recorded. Reads still use + // state: the unmapped-repo read template is itself a 404 error. + const failedWrite = /^(update|create|delete)-/.test(toolName) && template && template.error; + if (state && state.raRepos && !failedWrite) { const ra = renderReleaseAutopilot(template, input, toolName, state, replacements); if (ra !== undefined) return ra; } diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 77e31b48..15ebd84f 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -118,6 +118,7 @@ const RAW_NAMES = [ "update-release-autopilot", "get-release-autopilot", "list-release-autopilot", "githubRepoId", "enabledOverride", "autoReleasing", "autoFlagging", "approvalRequired", "create-release-plan", "list-release-plans", "list-release-policies", + "preview mode", "policy mode", "plan mode", ]; module.exports.noRawNames = (output) => { @@ -159,3 +160,44 @@ module.exports.noPickQuestions = (output) => { const repeats = asked.filter((l) => l.includes("auto-releasing") || l.includes("require approval") || l.includes("bring your own key")); return result(repeats.length === 0, repeats.length ? `re-asked: ${repeats.join(" ; ")}` : "no picks re-asked"); }; + +module.exports.notCalledComplete = (output) => { + const text = finalText(output); + const hit = /setup complete|setup is complete|you're all set|is (now )?live/i.exec(text); + return result(!hit, hit ? `final message says "${hit[0]}"` : "does not claim completion"); +}; + +// The preview section (from the "Changes an admin would make" heading on) is a +// short checklist; any state summary before it belongs to Step 1. +module.exports.previewIsShortChecklist = (output) => { + const text = finalText(output); + const at = text.search(/changes an admin would make/i); + if (at === -1) return result(false, "no 'Changes an admin would make' section"); + const section = text.slice(at); + const words = section.split(/\s+/).filter(Boolean).length; + const items = section.split("\n").filter((l) => /^\s*- \[ \]/.test(l)); + const pass = words <= 150 && items.length > 0; + return result(pass, `preview section: ${words} words, ${items.length} checklist line(s)`); +}; + +module.exports.skippedGithubNotReported = (output) => { + const hit = /github[^\n]{0,100}reported/i.exec(allText(output)); + return result(!hit, hit ? `labels skipped GitHub step: "${hit[0].slice(0, 80)}"` : "skipped GitHub step not labeled reported"); +}; + +module.exports.stopsAfterFailedWrite = (output) => { + const steps = traj(output); + const acct = steps.findIndex((t) => t.tool === "update-release-autopilot-settings"); + const repo = steps.findIndex((t) => t.tool === "update-release-autopilot-repo-settings"); + if (acct === -1) return result(false, "never attempted the account write"); + const after = steps.slice(acct + 1).filter((t) => WRITE_TOOLS.has(t.tool)); + const pass = repo !== -1 && repo < acct && after.length === 0; + return result(pass, `repo write at ${repo}, account write at ${acct}, ${after.length} write(s) after`); +}; + +// The link must be in the final pause summary, which is what the user forwards +// to their GitHub org owner. +module.exports.pauseSummaryHasConnectLink = (output) => { + const pass = finalText(output).includes("flow=install"); + return result(pass, pass ? "pause summary repeats the connect link" : "pause summary lacks the githubConnectUrl"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index b64f12e1..352750bc 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -5,16 +5,18 @@ # # The onboarding wizard for Release Autopilot. These cases lock in the write # rules found in dry runs against catamorphic: nothing is written before the -# user confirms, Preview and reader runs never write, a repo never inherits a +# user approves, Preview and reader runs never write, a repo never inherits a # capability the user did not pick, the release-plans 403 is treated as # account-wide policy mode, and account settings stay untouched when they -# already match. Default mocks match a catamorphic-like account: both -# capabilities on, the GitHub App installed, release plans forbidden by the -# account's plan. Scripted answers are matched to each question by option -# label (mock_ask_question_match), so a skipped question cannot shift them. -# Mocks are stateless, so read-backs after a write return the -# default state; assertions grade the writes, not the verification reads. Tests change state with mock_tool_overrides, and steer -# ask-question by option label. +# already match. +# +# Default mocks match a catamorphic-like account with both capabilities on, +# the GitHub App installed, and release plans available but empty (the GA +# shape). The policy-mode case overrides release plans to the 403 accounts +# return today. Mocks record Release Autopilot writes, so verification reads +# reflect them. Tests change state with mock_tool_overrides, and scripted +# answers are matched to each question by option label +# (mock_ask_question_match), so a skipped question cannot shift them. description: "Evaluation of the Release Autopilot onboarding skill" prompts: @@ -101,8 +103,10 @@ tests: description: "Policy mode, missing policy: one plans call, no plans, auto-releasing off for the repo, no account write" vars: user_request: "Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default." - max_turns: 25 + max_turns: 20 mock_tool_overrides: + list-release-plans: + error: "LaunchDarkly API error: 403: {\"code\":\"forbidden\",\"message\":\"Plan does not allow this operation\"}" list-release-policies: policies: [] count: 0 @@ -123,6 +127,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:releasePlansCalledAtMostOnce metric: plans_403_account_wide weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:skippedGithubNotReported + metric: skipped_step_label + weight: 1 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noRawNames metric: no_raw_names @@ -177,7 +185,7 @@ tests: # Explicit preview request from an admin: zero writes, and the final # message is a checklist of changes. # ------------------------------------------------------------------ - - providers: ["mcp"] + - providers: ["mcp-auto"] description: "Explicit preview: zero writes, ends with the planned changes" vars: user_request: "Do a dry run: preview what you'd change to set up Release Autopilot auto-flagging for acme/demo in project default. Don't change anything." @@ -198,6 +206,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:previewNamesTheMapping metric: preview_plan weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:previewIsShortChecklist + metric: preview_format + weight: 2 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noRawNames metric: no_raw_names @@ -235,10 +247,7 @@ tests: metric: no_writes weight: 5 - type: javascript - value: | - const text = String(output.response || '') + (output.assistant_turns || []).join('\n'); - const pass = text.includes('flow=install'); - return { pass, score: pass ? 1 : 0, reason: pass ? 'gave the connect link' : 'did not give the githubConnectUrl' }; + value: file://../release-autopilot-onboarding/assertions.js:pauseSummaryHasConnectLink metric: gave_connect_link weight: 3 @@ -271,6 +280,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite metric: no_account_write weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:notCalledComplete + metric: partly_set_up + weight: 2 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noRawNames metric: no_raw_names @@ -372,9 +385,7 @@ tests: Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default. For production only, create a release plan: randomize by user, 1% then 10% then 50% then 100% with 4h holds, automatic advancement, no metrics, and leave rules in place. - max_turns: 30 - mock_tool_overrides: - list-release-plans: { items: [], totalCount: 0 } + max_turns: 22 mock_ask_question_match: true mock_ask_question_answers: - ["Add more"] @@ -497,3 +508,41 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff metric: repo_override_unpicked weight: 3 + + # ------------------------------------------------------------------ + # The account write fails with 403 for an admin. The repo write comes + # first, nothing is written after the failure, and the account change + # is handed to an admin instead of being called complete. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Account write fails with 403: stop, hand off to an admin" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." + max_turns: 20 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + update-release-autopilot-settings: + error: "LaunchDarkly API error: 403: {\"code\":\"forbidden\",\"message\":\"Access to the requested resource was denied\"}" + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Require approval"] + - ["managed"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:stopsAfterFailedWrite + metric: stops_after_failure + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:notCalledComplete + metric: partly_set_up + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 diff --git a/evals/tests/mock.test.js b/evals/tests/mock.test.js index df6a6d9e..66b91921 100644 --- a/evals/tests/mock.test.js +++ b/evals/tests/mock.test.js @@ -452,3 +452,10 @@ test("release plans list reflects a created plan", () => { assert.equal(got.totalCount, 1); assert.equal(got.items[0].key, "p"); }); + +test("an error template wins over the stateful write and is not recorded", () => { + const state = createMockState(); + const err = { error: "403: denied" }; + assert.equal(renderMockResponse(err, { autoFlagging: { enabled: true } }, "update-release-autopilot-settings", state).error, "403: denied"); + assert.equal(state.raSettings, null); +}); diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 3423cff4..9853b5c2 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -60,8 +60,9 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Speaking to the user -- Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("policy mode"); say "your account uses release policies" and "Verification", not "Step 9". -- Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. +- Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("preview mode", "policy mode", "plan mode"); say "Here's what would change", "your account uses release policies", and "Verification", not "Step 9". +- Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. +- State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"). - Keep each message to the current step, and do not narrate tool calls. ## Workflow @@ -72,7 +73,7 @@ When a step is complete, go to the next one unless it says to ask or wait. Treat ### Step 0: Welcome the user -Decide the mode from the user's first message before you write anything. **In Resume mode, skip this step entirely**: do not show the intro or any part of it, and go straight to Step 1. In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." +Decide the mode from the user's first message before you write anything. Never announce the mode by name; for a preview request, say "I'll show what would change and make no changes." **In Resume mode, skip this step entirely**: do not show the intro or any part of it, and go straight to Step 1. In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." > **Release Autopilot Setup** > @@ -87,7 +88,7 @@ Then go to Step 1. Skip any step that the account state already completes, and s Run these reads. Do not write anything. 1. `get-member-self`: the member's `role`. - - `reader` or `no_access`: switch to Preview mode and say why. + - `reader` or `no_access`: switch to Preview. Tell the user: "You have read-only access, so I'll show what would change and make no changes." - `admin` or `owner`: can make every write in this flow. - Any other role, including `writer` and custom roles: may map repos in projects they can edit, but account settings need an admin or owner. In Step 7, mark account writes as "needs an admin" and leave them out of Step 8. 2. `get-release-autopilot-settings`: the account gates. @@ -132,7 +133,7 @@ Give the user the exact URL. Tell them to open it in a browser where they are lo - label: "Done", description: "I finished the GitHub step" - label: "Need an org owner", description: "I can't install the App myself. Pause setup here" -On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, say so and repeat this step. On "Need an org owner", summarize the decisions so far, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. +On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, say so and repeat this step. On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. ### Step 4: Map repositories @@ -167,17 +168,17 @@ For each project from Step 4, choose environments: use `AskUserQuestion` with he Call `list-release-plans` for the first chosen environment. -- **Policy mode** if the call returns 403 `Plan does not allow this operation`, or if `list-release-plans` is not available. The 403 is account-wide, so do not call it for other environments. This is the mode for every account today: auto-releasing requires a complete release policy for each environment. -- **Plan mode** if the call succeeds. Environment release plans replace release policies by general availability. +- **Release policies** if the call returns 403 `Plan does not allow this operation`, or if `list-release-plans` is not available. The 403 is account-wide, so do not call it for other environments. This is the mode for every account today: auto-releasing requires a complete release policy for each environment. +- **Release plans** if the call succeeds. Environment release plans replace release policies by general availability. -#### Policy mode +#### Release policies (today) Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. This skill does not change policies. - No warnings for the chosen environments: say that auto-releasing in this project uses those policies. - Any warning, such as `missing_policy` or `incomplete_policy`: auto-releasing cannot work there yet. Tell the user to add or complete a release policy in the project's release settings (`/projects/{projectKey}/settings/releases` on their LaunchDarkly instance). Record a repo override that keeps auto-releasing off for each repo in that project, and say in Step 7 that they can turn it on with the settings skill once the policy is complete. -#### Plan mode +#### Release plans (by GA) For each chosen environment, call `list-release-plans`. If a complete plan exists, show its name, phases, and metrics, and ask header "Plan": @@ -192,7 +193,7 @@ To create a plan, follow [references/release-plans.md](references/release-plans. List every write in the order you will make it. For each write, state the current and the proposed state and the blast radius, using the wording rules from the settings skill's [Confirm before write](../launchdarkly-release-autopilot-settings/SKILL.md#confirm-before-write): -1. **Metrics and release plans** (plan mode only): one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. +1. **Metrics and release plans** (release plans only): one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." 3. **Account settings**: only fields that change. If auto-releasing turns on, say it can ship code. For a member who is not an admin or owner, list account writes under "Needs an admin" instead. @@ -201,7 +202,12 @@ List every write in the order you will make it. For each write, state the curren If there are no writes, say so, list the follow-ups, and stop. -**Preview mode:** present the list as a checklist under the heading "Changes an admin would make", one line per change, then stop. Do not ask for confirmation. +**Preview mode:** the final message contains only the heading "Changes an admin would make", the checklist (one line per change), and any follow-ups. Do not repeat the account state, and do not ask for confirmation. For example: + +``` +- [ ] Map acme/demo to project default, with auto-releasing off for this repository +- [ ] Turn on auto-flagging for the account, with approval required +``` **Setup mode:** use `AskUserQuestion` with header "Confirm": @@ -214,8 +220,8 @@ Only "Looks good" approves the plan. On "Change something", ask what to change, Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. -1. Plan mode only: metrics recorded in Step 6, created in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). Skip its instrumentation steps. -2. Plan mode only: `create-release-plan` for each new plan. +1. Release plans only: metrics recorded in Step 6, created in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). Skip its instrumentation steps. +2. Release plans only: `create-release-plan` for each new plan. 3. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. For each capability the user did not pick, or that cannot work yet, send `{ "enabled": false }` for it, so the repo does not inherit it from the account. 4. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. @@ -227,9 +233,9 @@ If a write fails with 403, the member lacks that permission. Report which write 1. `get-release-autopilot-settings`: the account gates match the plan. 2. `get-release-autopilot-repo-settings` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. -3. Plan mode: `list-release-plans` for each environment lists the new plans. Policy mode: report the policy state from Step 6. +3. With release plans: `list-release-plans` for each environment lists the new plans. With release policies: report the policy state from Step 6. -Report a short checklist with each item marked verified, reported by the user, or already set up (for a step the flow skipped because it was already done), then the follow-ups. Then tell the user how to see it work: +If any change was left for an admin or failed, lead with "Partly set up: waiting on an admin" and the admin checklist, and never call the setup complete. Then report a short checklist with each item marked verified, reported by the user (browser steps the user did in this flow), or already set up (steps the flow skipped because they were already done), then the follow-ups. Then tell the user how to see it work: - Open a pull request in a mapped repo. Release Autopilot reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. - If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. @@ -240,7 +246,7 @@ Report a short checklist with each item marked verified, reported by the user, o |-----------|--------| | Every Release Autopilot tool returns 404 | The account is not in the alpha. Stop and tell the user | | Member role is `reader` or `no_access` | Preview mode. End with a checklist for an admin | -| Release plans return 403 | Policy mode. Create no plans | +| Release plans return 403 | Use release policies. Create no plans | | Repo not on the install list | Explain the GitHub fix by owner, then list again | More cases: [references/edge-cases.md](references/edge-cases.md). @@ -254,7 +260,7 @@ More cases: [references/edge-cases.md](references/edge-cases.md). ## References -- [references/release-plans.md](references/release-plans.md): creating environment release plans (plan mode) +- [references/release-plans.md](references/release-plans.md): creating environment release plans - [references/modes.md](references/modes.md): Preview and Resume details - [references/byok.md](references/byok.md): bring-your-own-key setup - [references/edge-cases.md](references/edge-cases.md): edge cases and existing mappings diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md index dbf6f563..9e0cc763 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -12,8 +12,8 @@ | Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | | Mapped repo ID not on the install list | Mapped-repo reads return only repository IDs, so join them to the install list. An ID with no match is a repo the App can no longer see, or one this member cannot see. Show it by ID with its project. Do not change it | | Repo already mapped to a different project | Do not remap here. The user unmaps it with the settings skill, then maps it in this flow | -| `list-release-plans` returns 403 `Plan does not allow this operation` | Policy mode. Show release policies and their warnings, create no plans, and keep auto-releasing off for the repo where a policy is missing or incomplete | -| Auto-releasing picked but no metrics exist (plan mode) | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | +| `list-release-plans` returns 403 `Plan does not allow this operation` | Use release policies. Show them and their warnings, create no plans, and keep auto-releasing off for the repo where a policy is missing or incomplete | +| Auto-releasing picked but no metrics exist (release plans) | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | | Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | | Generative AI or Vega eligibility error | Release Autopilot's agents need generative AI turned on, and no MCP tool reads that setting. An Admin or Owner selects **Enable generative AI use cases** in Organization settings > Security. Stop | | Bring your own key chosen | Browser step only. Record it as reported, never as verified | diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md index 25c6a2d6..51bee1bc 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md @@ -1,4 +1,4 @@ -# Create an environment release plan (plan mode) +# Create an environment release plan Use this from Step 6 of the onboarding skill when the account has environment release plans. A release plan is a reusable recipe for one environment, made of phases, rollout steps, and metrics. A plan with metrics makes a release guarded, so a regression on a metric can roll it back. A plan without metrics is progressive: say so. From 4a22c9243b2420975f63fd58d08e36b963a486e3 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Mon, 26 Oct 2026 20:00:01 -0700 Subject: [PATCH 18/30] fix(release-autopilot): conditional next steps while an admin change is pending; refresh settings description Co-Authored-By: Claude Opus 5.5 (1M context) --- evals/release-autopilot-onboarding/assertions.js | 7 ++++++- skills.json | 2 +- .../launchdarkly-release-autopilot-onboarding/SKILL.md | 2 +- .../launchdarkly-release-autopilot-settings/SKILL.md | 2 +- 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 15ebd84f..573f8980 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -163,7 +163,12 @@ module.exports.noPickQuestions = (output) => { module.exports.notCalledComplete = (output) => { const text = finalText(output); - const hit = /setup complete|setup is complete|you're all set|is (now )?live/i.exec(text); + let hit = /setup complete|setup is complete|you're all set|is (now )?live/i.exec(text); + // "Open a pull request" is fine only when tied to the pending admin change. + const unconditional = text + .split(/(?<=[.!?\n])/) + .find((s) => /open a pull request/i.test(s) && !/\b(once|after|when)\b/i.test(s)); + if (!hit && unconditional) hit = [unconditional.trim().slice(0, 80)]; return result(!hit, hit ? `final message says "${hit[0]}"` : "does not claim completion"); }; diff --git a/skills.json b/skills.json index f68af77e..6fe1fccd 100644 --- a/skills.json +++ b/skills.json @@ -439,7 +439,7 @@ }, { "name": "launchdarkly-release-autopilot-settings", - "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, set up release plans), use launchdarkly-release-autopilot-onboarding instead.", + "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, review release policies), use launchdarkly-release-autopilot-onboarding instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-settings", "version": "1.4.0-experimental", "license": "Apache-2.0", diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 9853b5c2..653e8737 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -235,7 +235,7 @@ If a write fails with 403, the member lacks that permission. Report which write 2. `get-release-autopilot-repo-settings` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. 3. With release plans: `list-release-plans` for each environment lists the new plans. With release policies: report the policy state from Step 6. -If any change was left for an admin or failed, lead with "Partly set up: waiting on an admin" and the admin checklist, and never call the setup complete. Then report a short checklist with each item marked verified, reported by the user (browser steps the user did in this flow), or already set up (steps the flow skipped because they were already done), then the follow-ups. Then tell the user how to see it work: +If any change was left for an admin or failed, lead with "Partly set up: waiting on an admin" and the admin checklist, and never call the setup complete. Then report a short checklist with each item marked verified, reported by the user (browser steps the user did in this flow), or already set up (steps the flow skipped because they were already done), then the follow-ups. Tell the user how to see it work; if a change is waiting on an admin, make that conditional ("Once an admin turns on auto-flagging, open a pull request…"): - Open a pull request in a mapped repo. Release Autopilot reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. - If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md index 8c6c1662..6b3290f4 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md @@ -1,6 +1,6 @@ --- name: launchdarkly-release-autopilot-settings -description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, set up release plans), use launchdarkly-release-autopilot-onboarding instead." +description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, review release policies), use launchdarkly-release-autopilot-onboarding instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: From 2d4790f8edeacfd2d677fb3968c043b3609d8aa2 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Tue, 27 Oct 2026 11:30:56 -0700 Subject: [PATCH 19/30] fix(release-autopilot-onboarding): explain the inference choice in plain language Rename 'Choose inference' to choosing whose AI model account runs the agents, explain the choice in one sentence before asking, and label the options 'LaunchDarkly's (default)' and 'Bring your own key'. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../promptfooconfig.yaml | 20 +++++++++---------- skills.json | 2 +- .../README.md | 2 +- .../SKILL.md | 20 +++++++++---------- .../references/byok.md | 2 +- 5 files changed, 23 insertions(+), 23 deletions(-) diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 352750bc..18b2d8d8 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -65,7 +65,7 @@ tests: - ["Require approval"] - ["acme/demo"] - ["default"] - - ["managed"] + - ["LaunchDarkly's"] - ["Looks good"] assert: - type: javascript @@ -119,7 +119,7 @@ tests: - ["Auto-flagging", "Auto-releasing"] - ["acme/demo"] - ["default"] - - ["managed"] + - ["LaunchDarkly's"] - ["Test", "Production"] - ["Looks good"] assert: @@ -169,7 +169,7 @@ tests: - ["Auto-flagging"] - ["acme/demo"] - ["default"] - - ["managed"] + - ["LaunchDarkly's"] - ["Looks good"] assert: - type: javascript @@ -196,7 +196,7 @@ tests: - ["Auto-flagging"] - ["acme/demo"] - ["default"] - - ["managed"] + - ["LaunchDarkly's"] assert: - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noWrites @@ -273,7 +273,7 @@ tests: - ["Require approval"] - ["acme/demo"] - ["default"] - - ["managed"] + - ["LaunchDarkly's"] - ["Looks good"] assert: - type: javascript @@ -311,7 +311,7 @@ tests: mock_ask_question_answers: - ["Auto-flagging"] - ["Require approval"] - - ["managed"] + - ["LaunchDarkly's"] - ["Change something"] assert: - type: javascript @@ -333,7 +333,7 @@ tests: mock_ask_question_answers: - ["Add more"] - ["Auto-flagging"] - - ["managed"] + - ["LaunchDarkly's"] - ["Looks good"] assert: - type: javascript @@ -366,7 +366,7 @@ tests: mock_ask_question_answers: - ["Add more"] - ["Auto-flagging"] - - ["managed"] + - ["LaunchDarkly's"] - ["Looks good"] assert: - type: javascript @@ -390,7 +390,7 @@ tests: mock_ask_question_answers: - ["Add more"] - ["Auto-flagging", "Auto-releasing"] - - ["managed"] + - ["LaunchDarkly's"] - ["Production"] - ["Automatic"] - ["Leave rules"] @@ -531,7 +531,7 @@ tests: mock_ask_question_answers: - ["Auto-flagging"] - ["Require approval"] - - ["managed"] + - ["LaunchDarkly's"] - ["Looks good"] assert: - type: javascript diff --git a/skills.json b/skills.json index 824ac14d..bd948fca 100644 --- a/skills.json +++ b/skills.json @@ -421,7 +421,7 @@ }, { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (managed by default, or bring your own key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", "version": "0.2.0-experimental", "license": "Apache-2.0", diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md index 5406bc6e..1d80c589 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -9,7 +9,7 @@ This skill teaches agents how to: - Ask which capabilities the user wants (auto-flagging, auto-releasing) and set up only what those need - Hand the user the GitHub App install or authorize link, and continue after they finish - Map chosen `owner/name` repos to LaunchDarkly projects -- Guide LaunchDarkly managed inference or bring-your-own-key setup +- Explain whose AI model account runs the agents (LaunchDarkly's by default, or the user's own Anthropic or OpenAI key) and guide bring-your-own-key setup - Review or create environment release plans for auto-releasing, or show release policies where plans are not available - Confirm one plan that lists every write before making any change, then verify diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 653e8737..e67d4c5d 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -1,6 +1,6 @@ --- name: launchdarkly-release-autopilot-onboarding -description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose Vega inference (managed by default, or bring your own key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: @@ -77,7 +77,7 @@ Decide the mode from the user's first message before you write anything. Never a > **Release Autopilot Setup** > -> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose inference, and review release setup if you want auto-releasing. +> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose whose AI model account runs the agents, and review release setup if you want auto-releasing. > > Nothing changes until you approve the full plan. @@ -144,17 +144,17 @@ On "Done", call `list-release-autopilot-github-repos` again. If it still returns 3. **Check existing mappings** with the join from Step 1 (an unmapped repo returns 404, which is expected). Already mapped to the chosen project with matching capabilities: write nothing. Mapped to the chosen project with an unpicked capability on: record an override that turns it off. Mapped to a different project: never remap here; send the user to the settings skill to unmap first. Details: [references/edge-cases.md](references/edge-cases.md#existing-mappings). 4. **Say what starts when.** If the account already has a capability on, mapping a repo starts that automation in the repo as soon as Step 8 maps it. -### Step 5: Choose inference +### Step 5: Choose the AI model account -Release Autopilot runs its agents on Vega. Bring your own key is optional: a project that does not set it up uses LaunchDarkly managed Vega inference. Use `AskUserQuestion` with header "Inference": +Release Autopilot's agents (Vega) use an AI model to read pull requests and write flag changes. This step decides whose model account they use; the product calls it "inference". Most users keep the default. Before asking, tell the user in one sentence what the choice means, for example: "Release Autopilot's agents use an AI model to read your pull requests. By default they run on LaunchDarkly's model account; you can use your own Anthropic or OpenAI key instead." Then use `AskUserQuestion` with header "AI model": -- label: "LaunchDarkly managed", description: "The default if you choose nothing. The agents run on Vega inference that LaunchDarkly hosts. No setup" -- label: "Bring your own key", description: "The agents run on your own Anthropic or OpenAI account. Your provider bills you for usage" +- label: "LaunchDarkly's (default)", description: "The agents use an AI model that LaunchDarkly provides and pays for. No setup" +- label: "Bring your own key", description: "The agents use your own Anthropic or OpenAI API key, for example to keep usage on your provider contract. Your provider bills you for usage" -For "Bring your own key", this is a browser step (no MCP tool reads or writes inference settings). For each project, follow [references/byok.md](references/byok.md), then ask header "Inference": +For "Bring your own key", this is a browser step (no MCP tool reads or writes these settings). For each project, follow [references/byok.md](references/byok.md), then ask header "AI model": -- label: "Done", description: "I saved the inference settings for every project" -- label: "Use managed for now", description: "Skip this. I'll switch to my own key later" +- label: "Done", description: "I saved my key in the Vega settings for every project" +- label: "Use LaunchDarkly's for now", description: "Skip this. I'll switch to my own key later" Record the answer as reported by the user, not as verified. @@ -197,7 +197,7 @@ List every write in the order you will make it. For each write, state the curren 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." 3. **Account settings**: only fields that change. If auto-releasing turns on, say it can ship code. For a member who is not an admin or owner, list account writes under "Needs an admin" instead. -4. **Browser steps the user reported done**: GitHub and inference, marked as reported, not verified. +4. **Browser steps the user reported done**: GitHub and the AI model choice, marked as reported, not verified. 5. **Follow-ups for the user**: release policies to add, metric events to instrument. If there are no writes, say so, list the follow-ups, and stop. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md index bb7de7eb..e43b6922 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md @@ -1,6 +1,6 @@ # Bring your own key -Bring your own key is optional. A project that does not set it up uses LaunchDarkly managed Vega inference. No MCP tool reads or writes inference settings, so this is a browser step. +Release Autopilot's agents use an AI model to read pull requests. Bring your own key is optional: a project that does not set it up uses LaunchDarkly's model account (the product calls this "LaunchDarkly managed inference"). No MCP tool reads or writes inference settings, so this is a browser step. For each project, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). On that page they: From fb587411f2fe6f34d65aece992f70d9082781f13 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Tue, 27 Oct 2026 15:03:57 -0700 Subject: [PATCH 20/30] test(evals): keep scripted-answer order mode unchanged; tighten repo naming Label matching for scripted ask-question answers now applies only in the opt-in match mode, so suites that script answers in order (onboarding, experiment-guide) behave exactly as on main. The skill names repos by owner/name and drops the 'release mode' heading the agent echoed. Co-Authored-By: Claude Opus 5.5 (1M context) --- evals/providers/claude-skill-agent-sdk.js | 26 ++++++++++--------- evals/tests/provider-match-option.test.js | 4 +-- .../SKILL.md | 7 ++--- 3 files changed, 20 insertions(+), 17 deletions(-) diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 7e320641..236defc4 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -48,14 +48,14 @@ * (true/false), e.g. to run one test with no * LaunchDarkly tools * mock_ask_question_answers - optional array of `selected` arrays returned by - * successive `ask-question` calls. An entry that is - * not one of the call's option ids is matched - * against option labels (case-insensitive substring), - * so tests can steer by label when the agent picks ids + * successive `ask-question` calls * mock_ask_question_match - when true, each `ask-question` call takes the first * unused scripted answer whose entries all match its * options (by id or label), instead of the next one in - * order. Use for flows that may skip questions. A call + * order. Entries match an option by id, or else by + * label (case-insensitive substring), so tests can + * steer by label when the agent picks the ids. Use + * for flows that may skip questions. A call * with no match gets its first option and is recorded * with `scripted: false` * mock_tool_overrides - optional object mapping tool name to a mock template @@ -107,9 +107,10 @@ function matchOption(want, options) { /** * Choose the scripted answer for the `idx`-th ask-question call. In order - * mode, take answers[idx]. In match mode, take the first unused answer whose - * entries all resolve to the call's options, and mark it used. Returns the - * selected option ids, or undefined when no scripted answer applies. + * mode, take answers[idx] as written. In match mode, take the first unused + * answer whose entries all resolve to the call's options (by id or label), + * mark it used, and return the matched ids. Returns undefined when no + * scripted answer applies. */ function pickAnswer(answers, idx, options, matchMode, used) { const resolves = (ans) => @@ -122,11 +123,12 @@ function pickAnswer(answers, idx, options, matchMode, used) { if (hit === -1) return undefined; used.add(hit); chosen = answers[hit]; - } else { - chosen = answers[idx]; - if (!Array.isArray(chosen) || chosen.length === 0) return undefined; + return chosen.map((want) => matchOption(want, options)); } - return chosen.map((want) => matchOption(want, options)); + // Order mode is unchanged: the scripted answer is returned as written. + chosen = answers[idx]; + if (!Array.isArray(chosen) || chosen.length === 0) return undefined; + return chosen; } const mockTemplates = JSON.parse( diff --git a/evals/tests/provider-match-option.test.js b/evals/tests/provider-match-option.test.js index 0310964a..b15ad9ae 100644 --- a/evals/tests/provider-match-option.test.js +++ b/evals/tests/provider-match-option.test.js @@ -26,8 +26,8 @@ test("tolerates missing options", () => { assert.equal(matchOption("x", undefined), "x"); }); -test("order mode takes the answer at the call index", () => { - assert.deepEqual(pickAnswer([["x"], ["Auto-releasing"]], 1, options, false, new Set()), ["ar"]); +test("order mode returns the answer at the call index as written", () => { + assert.deepEqual(pickAnswer([["x"], ["Auto-releasing"]], 1, options, false, new Set()), ["Auto-releasing"]); }); test("match mode skips answers that fit no option, and uses each once", () => { diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index e67d4c5d..657eb71b 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -30,7 +30,7 @@ This skill requires the remotely hosted LaunchDarkly MCP server. - `list-release-policies` **Used when available:** -- `list-release-plans` / `create-release-plan` / `list-metrics`: environment release plans (see [Detect the release mode](#detect-the-release-mode)) +- `list-release-plans` / `create-release-plan` / `list-metrics`: environment release plans (see [Check whether the account uses release plans](#check-whether-the-account-uses-release-plans)) - `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. @@ -61,6 +61,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Speaking to the user - Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("preview mode", "policy mode", "plan mode"); say "Here's what would change", "your account uses release policies", and "Verification", not "Step 9". +- Name repos as `owner/name`. Show a repository ID only for a mapped repo with no name. - Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. - State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"). - Keep each message to the current step, and do not narrate tool calls. @@ -162,9 +163,9 @@ Record the answer as reported by the user, not as verified. Skip this step if the user did not pick auto-releasing. -For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then detect the release mode once per account and follow that mode. +For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then check once per account whether it uses release plans or release policies. -#### Detect the release mode +#### Check whether the account uses release plans Call `list-release-plans` for the first chosen environment. From 05727d33c12b2d539a8d76d79b35e94e197b0c4c Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Wed, 28 Oct 2026 11:14:42 -0700 Subject: [PATCH 21/30] feat(release-autopilot-onboarding): offer auto-cleanup; keep links visible; use capability names Offer auto-cleanup as an opt-in capability, set up as a browser step on the project's Lifecycle settings page (the MCP settings tools do not manage it yet, so autoCleanup is never sent). Put links in the question text, because the question dialog hides the message above it. After the welcome, name the capabilities instead of 'Release Autopilot', and never invent UI locations. Evals: 15 cases, adding auto-cleanup, a link-in- question check, and a capability-names check. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- .../assertions.js | 43 +++++++++++-- .../promptfooconfig.yaml | 63 ++++++++++++++++++- skills.json | 2 +- .../README.md | 1 + .../SKILL.md | 47 +++++++++----- .../references/auto-cleanup.md | 18 ++++++ 7 files changed, 152 insertions(+), 24 deletions(-) create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f3df99c..1bb42911 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to this repository will be documented in this file. - Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. -- evals: add the `release-autopilot-onboarding` suite (14 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. +- evals: add the `release-autopilot-onboarding` suite (15 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0). - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 573f8980..cd09581e 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -164,10 +164,13 @@ module.exports.noPickQuestions = (output) => { module.exports.notCalledComplete = (output) => { const text = finalText(output); let hit = /setup complete|setup is complete|you're all set|is (now )?live/i.exec(text); - // "Open a pull request" is fine only when tied to the pending admin change. - const unconditional = text - .split(/(?<=[.!?\n])/) - .find((s) => /open a pull request/i.test(s) && !/\b(once|after|when)\b/i.test(s)); + // "Open a pull request" is fine only when tied to the pending admin change, + // e.g. under a "Once an admin turns on auto-flagging:" lead-in. + let unconditional; + for (const m of text.matchAll(/open a pull request/gi)) { + const before = text.slice(Math.max(0, m.index - 250), m.index); + if (!/\b(once|after|when)\b/i.test(before)) { unconditional = text.slice(m.index, m.index + 80); break; } + } if (!hit && unconditional) hit = [unconditional.trim().slice(0, 80)]; return result(!hit, hit ? `final message says "${hit[0]}"` : "does not claim completion"); }; @@ -206,3 +209,35 @@ module.exports.pauseSummaryHasConnectLink = (output) => { const pass = finalText(output).includes("flow=install"); return result(pass, pass ? "pause summary repeats the connect link" : "pause summary lacks the githubConnectUrl"); }; + +const askPrompts = (output) => + traj(output).filter((t) => t.tool === "ask-question").map((t) => String((t.arguments && t.arguments.prompt) || "")); + +// The question dialog hides earlier text, so links belong in the question. +module.exports.githubQuestionHasLink = (output) => { + const pass = askPrompts(output).some((p) => p.includes("flow=install")); + return result(pass, pass ? "GitHub question contains the connect link" : "no question contains the githubConnectUrl"); +}; + +module.exports.cleanupQuestionHasLifecycleLink = (output) => { + const pass = askPrompts(output).some((p) => p.includes("/settings/lifecycle")); + return result(pass, pass ? "auto-cleanup question contains the lifecycle settings link" : "no question contains /settings/lifecycle"); +}; + +// "Release Autopilot" belongs only in the welcome, the resume phrase, and the +// settings skill's name. +module.exports.noProductNameAfterIntro = (output) => { + const turns = ((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? x : (x && x.text) || "")); + const texts = [...turns, finalText(output), ...askPrompts(output)] + .filter((s) => !/Release Autopilot Setup|in a few short steps/i.test(s)) + .map((s) => s.replace(/continue Release Autopilot setup|Release Autopilot settings skill|release-autopilot-settings/gi, "")); + const hits = texts.filter((s) => /Release Autopilot/.test(s)); + return result(hits.length === 0, hits.length ? `said "Release Autopilot" after the welcome: "${hits[0].slice(Math.max(0, hits[0].indexOf("Release Autopilot") - 40), hits[0].indexOf("Release Autopilot") + 40)}"` : "uses capability names"); +}; + +// The remap case must leave acme/storefront alone; other repo writes come +// from unscripted questions the harness answers with their first option. +module.exports.noStorefrontRemap = (output) => { + const hit = traj(output).filter((t) => t.tool.endsWith("-repo-settings") && /^(update|delete)/.test(t.tool) && (t.arguments || {}).repo === "acme/storefront"); + return result(hit.length === 0, hit.length ? `wrote acme/storefront: ${JSON.stringify(hit[0].arguments)}` : "acme/storefront left alone"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 18b2d8d8..7da6fea1 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -88,6 +88,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup metric: no_auto_cleanup weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noRawNames metric: no_raw_names @@ -210,6 +214,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:previewIsShortChecklist metric: preview_format weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noRawNames metric: no_raw_names @@ -250,6 +258,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:pauseSummaryHasConnectLink metric: gave_connect_link weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:githubQuestionHasLink + metric: link_in_question + weight: 3 # ------------------------------------------------------------------ # Writer role on a fresh account: account settings need an admin, so @@ -284,6 +296,10 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:notCalledComplete metric: partly_set_up weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noRawNames metric: no_raw_names @@ -370,7 +386,7 @@ tests: - ["Looks good"] assert: - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:noRepoWrite + value: file://../release-autopilot-onboarding/assertions.js:noStorefrontRemap metric: no_remap weight: 5 @@ -546,3 +562,48 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:noRawNames metric: no_raw_names weight: 2 + + # ------------------------------------------------------------------ + # Auto-cleanup picked with auto-flagging. Auto-cleanup is a browser + # step on the project's Lifecycle settings page: the link is in the + # question, autoCleanup is never sent, and the repo write is confirmed. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Auto-cleanup: lifecycle settings link in the question, never sends autoCleanup" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 22 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging", "Auto-cleanup"] + - ["Require approval"] + - ["LaunchDarkly's"] + - ["Done"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup + metric: no_auto_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:cleanupQuestionHasLifecycleLink + metric: lifecycle_link_in_question + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 diff --git a/skills.json b/skills.json index bd948fca..f6c0641d 100644 --- a/skills.json +++ b/skills.json @@ -421,7 +421,7 @@ }, { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", "version": "0.2.0-experimental", "license": "Apache-2.0", diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md index 1d80c589..8a5d6b8d 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -9,6 +9,7 @@ This skill teaches agents how to: - Ask which capabilities the user wants (auto-flagging, auto-releasing) and set up only what those need - Hand the user the GitHub App install or authorize link, and continue after they finish - Map chosen `owner/name` repos to LaunchDarkly projects +- Offer auto-cleanup and guide the project's automated flag cleanup settings - Explain whose AI model account runs the agents (LaunchDarkly's by default, or the user's own Anthropic or OpenAI key) and guide bring-your-own-key setup - Review or create environment release plans for auto-releasing, or show release policies where plans are not available - Confirm one plan that lists every write before making any change, then verify diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 657eb71b..3b210607 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -1,6 +1,6 @@ --- name: launchdarkly-release-autopilot-onboarding -description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), and review release policies (release plans by GA). Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: @@ -61,9 +61,10 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Speaking to the user - Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("preview mode", "policy mode", "plan mode"); say "Here's what would change", "your account uses release policies", and "Verification", not "Step 9". +- Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup" and the name of the settings skill. - Name repos as `owner/name`. Show a repository ID only for a mapped repo with no name. - Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. -- State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"). +- State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"), and do not invent UI locations: give only the pages this skill names. For an account change left for an admin, tell them to run [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). - Keep each message to the current step, and do not narrate tool calls. ## Workflow @@ -72,13 +73,15 @@ When a step is complete, go to the next one unless it says to ask or wait. Treat **Asking questions:** When a question has options, use `AskUserQuestion` with the labels and descriptions given below. Ask free-text questions as a normal message. If the user's request already answers a question explicitly, such as naming the repo, the project, or the capabilities, use that answer and say so instead of asking. Never infer the answer to the Step 7 confirmation. +**Links:** the question dialog covers the message above it. When a question follows a link the user must open (GitHub, bring your own key, lifecycle or release settings), put the exact URL in the question text itself, not only in the message before it. + ### Step 0: Welcome the user Decide the mode from the user's first message before you write anything. Never announce the mode by name; for a preview request, say "I'll show what would change and make no changes." **In Resume mode, skip this step entirely**: do not show the intro or any part of it, and go straight to Step 1. In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." > **Release Autopilot Setup** > -> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose whose AI model account runs the agents, and review release setup if you want auto-releasing. +> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose whose AI model account runs the agents, then set up releases and flag cleanup for the capabilities you pick. > > Nothing changes until you approve the full plan. @@ -110,15 +113,16 @@ In Preview mode, leave out "Change one setting", because that hand-off writes. F Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: -- label: "Auto-flagging (Recommended)", description: "Release Autopilot reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" -- label: "Auto-releasing", description: "After a flagged change deploys, Release Autopilot releases it with the environment's release policy or release plan. This can ship code to users" +- label: "Auto-flagging (Recommended)", description: "Reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" +- label: "Auto-releasing", description: "After a flagged change deploys, releases it with the environment's release policy or release plan. This can ship code to users" +- label: "Auto-cleanup", description: "On a schedule, opens pull requests that remove the code for flags that are ready for removal. This removes code" Ask about approval only if the user picked auto-flagging **and** the account does not already have auto-flagging on with approval required. If it already does, say "Auto-flagging already requires approval for every mapped repo. Say 'change approval' if you want to turn it off," and move on unless they say so. Otherwise, or if they ask to change it, ask header "Approval": -- label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before Release Autopilot changes code. This is an account setting, so it applies to every mapped repo" -- label: "No approval", description: "Release Autopilot changes PRs without an approval step, in every mapped repo" +- label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before any code changes. This is an account setting, so it applies to every mapped repo" +- label: "No approval", description: "Auto-flagging changes PRs without an approval step, in every mapped repo" -If the user picks neither capability, stop and tell them there is nothing to set up. +Auto-cleanup is opt-in like auto-releasing: never recommend it by default. If the user picks no capability, stop and tell them there is nothing to set up. ### Step 3: Connect GitHub @@ -126,10 +130,10 @@ Skip this step without mentioning GitHub if Step 1 returned `installed: true`, ` Otherwise read the URL's `flow`: -- `flow=install`: the LaunchDarkly GitHub App is not installed on their GitHub organization. Release Autopilot cannot see any pull request until it is. +- `flow=install`: the LaunchDarkly GitHub App is not installed on their GitHub organization. None of these capabilities can see a pull request until it is. - `flow=authorize`: the App is installed, but this member has not authorized GitHub, so repos cannot be listed. -Give the user the exact URL. Tell them to open it in a browser where they are logged in to LaunchDarkly. To install, they must be a GitHub organization owner, or they must request the install from one. Then ask header "GitHub": +Give the user the exact URL. Tell them to open it in a browser where they are logged in to LaunchDarkly. To install, they must be a GitHub organization owner, or they must request the install from one. Then ask header "GitHub", with the URL in the question text (for example, "Open https://… to install the LaunchDarkly GitHub App. Done?"): - label: "Done", description: "I finished the GitHub step" - label: "Need an org owner", description: "I can't install the App myself. Pause setup here" @@ -141,13 +145,13 @@ On "Done", call `list-release-autopilot-github-repos` again. If it still returns 1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. If a repo the user wants is not on the list, its owner decides the fix: a listed owner means the App needs access to that repo, and a new owner means an org owner must install the App there (no `githubConnectUrl` appears for this case). See [references/edge-cases.md](references/edge-cases.md). Wait for "Done" and list again. -2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where Release Autopilot creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. +2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where auto-flagging creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. 3. **Check existing mappings** with the join from Step 1 (an unmapped repo returns 404, which is expected). Already mapped to the chosen project with matching capabilities: write nothing. Mapped to the chosen project with an unpicked capability on: record an override that turns it off. Mapped to a different project: never remap here; send the user to the settings skill to unmap first. Details: [references/edge-cases.md](references/edge-cases.md#existing-mappings). 4. **Say what starts when.** If the account already has a capability on, mapping a repo starts that automation in the repo as soon as Step 8 maps it. ### Step 5: Choose the AI model account -Release Autopilot's agents (Vega) use an AI model to read pull requests and write flag changes. This step decides whose model account they use; the product calls it "inference". Most users keep the default. Before asking, tell the user in one sentence what the choice means, for example: "Release Autopilot's agents use an AI model to read your pull requests. By default they run on LaunchDarkly's model account; you can use your own Anthropic or OpenAI key instead." Then use `AskUserQuestion` with header "AI model": +The agents behind these capabilities (Vega) use an AI model to read pull requests and write code changes. This step decides whose model account they use; the product calls it "inference". Most users keep the default. Before asking, tell the user in one sentence what the choice means, for example: "The agents behind auto-flagging use an AI model to read your pull requests. By default they run on LaunchDarkly's model account; you can use your own Anthropic or OpenAI key instead." Then use `AskUserQuestion` with header "AI model": - label: "LaunchDarkly's (default)", description: "The agents use an AI model that LaunchDarkly provides and pays for. No setup" - label: "Bring your own key", description: "The agents use your own Anthropic or OpenAI API key, for example to keep usage on your provider contract. Your provider bills you for usage" @@ -159,9 +163,9 @@ For "Bring your own key", this is a browser step (no MCP tool reads or writes th Record the answer as reported by the user, not as verified. -### Step 6: Review release setup (auto-releasing only) +### Step 6: Set up releases and cleanup -Skip this step if the user did not pick auto-releasing. +Skip this step if the user picked neither auto-releasing nor auto-cleanup. For auto-cleanup, follow [Auto-cleanup](#auto-cleanup) below. The rest of this step is for auto-releasing. For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then check once per account whether it uses release plans or release policies. @@ -188,6 +192,15 @@ For each chosen environment, call `list-release-plans`. If a complete plan exist To create a plan, follow [references/release-plans.md](references/release-plans.md). Record the plan and any metrics to create. Do not call `create-release-plan` or create metrics yet. +#### Auto-cleanup + +Auto-cleanup is the per-project **Automated flag cleanup** setting on the Lifecycle settings page. The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: + +- label: "Done", description: "I turned on automated flag cleanup for every project" +- label: "Later", description: "Skip this. I'll turn it on from Lifecycle settings later" + +Record the answer as reported by the user, and list "Done" under browser steps in Step 7. + ### Step 7: Confirm the plan **STOP. Do not write anything until the user explicitly confirms.** A detailed request at the start is not approval. @@ -198,7 +211,7 @@ List every write in the order you will make it. For each write, state the curren 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." 3. **Account settings**: only fields that change. If auto-releasing turns on, say it can ship code. For a member who is not an admin or owner, list account writes under "Needs an admin" instead. -4. **Browser steps the user reported done**: GitHub and the AI model choice, marked as reported, not verified. +4. **Browser steps the user reported done**: GitHub, the AI model choice, and auto-cleanup, marked as reported, not verified. 5. **Follow-ups for the user**: release policies to add, metric events to instrument. If there are no writes, say so, list the follow-ups, and stop. @@ -238,14 +251,14 @@ If a write fails with 403, the member lacks that permission. Report which write If any change was left for an admin or failed, lead with "Partly set up: waiting on an admin" and the admin checklist, and never call the setup complete. Then report a short checklist with each item marked verified, reported by the user (browser steps the user did in this flow), or already set up (steps the flow skipped because they were already done), then the follow-ups. Tell the user how to see it work; if a change is waiting on an admin, make that conditional ("Once an admin turns on auto-flagging, open a pull request…"): -- Open a pull request in a mapped repo. Release Autopilot reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. +- Open a pull request in a mapped repo. Auto-flagging reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. - If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. ## Edge Cases | Situation | Action | |-----------|--------| -| Every Release Autopilot tool returns 404 | The account is not in the alpha. Stop and tell the user | +| Every settings tool returns 404 | The account is not in the alpha. Stop and tell the user | | Member role is `reader` or `no_access` | Preview mode. End with a checklist for an admin | | Release plans return 403 | Use release policies. Create no plans | | Repo not on the install list | Explain the GitHub fix by owner, then list again | diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md new file mode 100644 index 00000000..2d32900b --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md @@ -0,0 +1,18 @@ +# Auto-cleanup + +Auto-cleanup runs Vega on a schedule to find flags that are ready for code removal and open pull requests that remove their code. It is set per project, in the **Automated flag cleanup** section of Lifecycle settings. The MCP settings tools do not manage it yet. + +For each project, give the user `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). In the "Automated flag cleanup" section they: + +1. Check **Enable automated flag cleanups**. +2. Choose a **Schedule frequency**: 1 day, 7 days (the default), or 30 days. +3. Set **Max cleanup jobs per run** (1–20). +4. Select the **GitHub repositories** to clean up. Suggest the repos mapped in this flow. At least one is required. +5. Optionally add **Additional context** for the cleanup agent, such as where flags are defined. Vega also follows the repo's `AGENTS.md`. +6. Click **Save**. + +Cleanup jobs run with the permissions of the member who saved the settings. If that member loses access, someone else must save the settings again. + +Record the result as reported by the user. It cannot be verified. + +Docs: [Automated flag cleanup](https://launchdarkly.com/docs/home/flags/manage/flag-cleanup-vega#automated-flag-cleanup) From 67ae7b99e47aaa9fc062c719980b8548c9d7a417 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Wed, 28 Oct 2026 11:34:45 -0700 Subject: [PATCH 22/30] feat(release-autopilot-onboarding): diagnose a browser signed in to a different account The GitHub connect link acts on whatever LaunchDarkly session the browser has. When the App still reads as not installed after the user finished, name the member this connection uses and ask which account the browser is signed in to, instead of repeating the link. Rarer causes (instance, GitHub identity, org) move to references/github-troubleshooting.md. Every browser step now names the member to sign in as. Adds a 16th eval case. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- .../assertions.js | 12 ++++++ .../promptfooconfig.yaml | 43 +++++++++++++++++++ .../SKILL.md | 17 ++++++-- .../references/github-troubleshooting.md | 12 ++++++ 5 files changed, 81 insertions(+), 5 deletions(-) create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/github-troubleshooting.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 1bb42911..a773077e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to this repository will be documented in this file. - Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. -- evals: add the `release-autopilot-onboarding` suite (15 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. +- evals: add the `release-autopilot-onboarding` suite (16 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, a browser signed in to a different account). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0). - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index cd09581e..d9839594 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -241,3 +241,15 @@ module.exports.noStorefrontRemap = (output) => { const hit = traj(output).filter((t) => t.tool.endsWith("-repo-settings") && /^(update|delete)/.test(t.tool) && (t.arguments || {}).repo === "acme/storefront"); return result(hit.length === 0, hit.length ? `wrote acme/storefront: ${JSON.stringify(hit[0].arguments)}` : "acme/storefront left alone"); }; + +module.exports.relistsAfterDone = (output) => { + const n = traj(output).filter((t) => t.tool === "list-release-autopilot-github-repos").length; + return result(n >= 2, `listed GitHub repos ${n} time(s)`); +}; + +// After "Done" with no install, the skill names this connection's member and +// asks which account the browser is signed in to. +module.exports.asksWhichBrowserAccount = (output) => { + const pass = askPrompts(output).some((p) => p.includes("admin@acme.example") && !p.includes("flow=install")); + return result(pass, pass ? "asked which account the browser uses, naming this connection's member" : "did not ask about the browser account with the member's email"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 7da6fea1..739ecb58 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -607,3 +607,46 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro metric: capability_names weight: 1 + + # ------------------------------------------------------------------ + # The user installs the App and says "Done", but this connection still + # sees no install: the browser was signed in to a different account. + # The skill names this connection's member, asks which account the + # browser uses, and does not just repeat the link. No writes. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Installed from a different browser account: names the member, diagnoses, zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for my repos." + max_turns: 18 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + list-release-autopilot-github-repos: + installed: false + userAuthorized: false + repositories: [] + totalCount: 0 + githubConnectUrl: "https://app.launchdarkly.com/github-connect?flow=install" + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Require approval"] + - ["Done"] + - ["A different one"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:relistsAfterDone + metric: relists_after_done + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:asksWhichBrowserAccount + metric: diagnoses_browser_account + weight: 4 diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 3b210607..60f4be72 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -64,6 +64,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the - Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup" and the name of the settings skill. - Name repos as `owner/name`. Show a repository ID only for a mapped repo with no name. - Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. +- Every browser step (GitHub, bring your own key, Lifecycle or release settings) must be done while signed in as the member from Step 1. Say so when you give the link. - State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"), and do not invent UI locations: give only the pages this skill names. For an account change left for an admin, tell them to run [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). - Keep each message to the current step, and do not narrate tool calls. @@ -91,7 +92,7 @@ Then go to Step 1. Skip any step that the account state already completes, and s Run these reads. Do not write anything. -1. `get-member-self`: the member's `role`. +1. `get-member-self`: the member's `role`, and their `email`. Keep the email: every browser step must happen while signed in to LaunchDarkly as that member (see [Step 3](#step-3-connect-github)). - `reader` or `no_access`: switch to Preview. Tell the user: "You have read-only access, so I'll show what would change and make no changes." - `admin` or `owner`: can make every write in this flow. - Any other role, including `writer` and custom roles: may map repos in projects they can edit, but account settings need an admin or owner. In Step 7, mark account writes as "needs an admin" and leave them out of Step 8. @@ -133,12 +134,18 @@ Otherwise read the URL's `flow`: - `flow=install`: the LaunchDarkly GitHub App is not installed on their GitHub organization. None of these capabilities can see a pull request until it is. - `flow=authorize`: the App is installed, but this member has not authorized GitHub, so repos cannot be listed. -Give the user the exact URL. Tell them to open it in a browser where they are logged in to LaunchDarkly. To install, they must be a GitHub organization owner, or they must request the install from one. Then ask header "GitHub", with the URL in the question text (for example, "Open https://… to install the LaunchDarkly GitHub App. Done?"): +Give the user the exact URL. The link acts on whatever LaunchDarkly session the browser has, so tell them to open it while signed in as the member from Step 1 (for example, "Open it while signed in to LaunchDarkly as admin@acme.example"). To install, they must be a GitHub organization owner, or they must request the install from one. Then ask header "GitHub", with the URL in the question text (for example, "Open https://… to install the LaunchDarkly GitHub App. Done?"): - label: "Done", description: "I finished the GitHub step" - label: "Need an org owner", description: "I can't install the App myself. Pause setup here" -On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, say so and repeat this step. On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. +On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, do not repeat the link. The most common cause is a browser signed in to a different LaunchDarkly account or member, so the App was installed there. Ask header "Browser account", with this connection's email in the question text (for example, "This connection is signed in as admin@acme.example. Which account is your browser signed in to?"): + +- label: "The same", description: "My browser is signed in as that member" +- label: "A different one", description: "My browser is signed in to another account or member" +- label: "Not sure", description: "I need to check" + +For "A different one" or "Not sure", tell them to check at `/profile`, sign in as the member from Step 1, open the link again, and finish there; then list again. For "The same", check the other causes in [references/github-troubleshooting.md](references/github-troubleshooting.md). On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. ### Step 4: Map repositories @@ -210,7 +217,7 @@ List every write in the order you will make it. For each write, state the curren 1. **Metrics and release plans** (release plans only): one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." 3. **Account settings**: only fields that change. If auto-releasing turns on, say it can ship code. - For a member who is not an admin or owner, list account writes under "Needs an admin" instead. + For a member who is not an admin or owner, list account writes under "Needs an admin" instead, and say an admin can make them with the settings skill. Do not name a settings page; there is none to point to. 4. **Browser steps the user reported done**: GitHub, the AI model choice, and auto-cleanup, marked as reported, not verified. 5. **Follow-ups for the user**: release policies to add, metric events to instrument. @@ -262,6 +269,7 @@ If any change was left for an admin or failed, lead with "Partly set up: waiting | Member role is `reader` or `no_access` | Preview mode. End with a checklist for an admin | | Release plans return 403 | Use release policies. Create no plans | | Repo not on the install list | Explain the GitHub fix by owner, then list again | +| GitHub still not connected after "Done" | Likely a different browser account. Follow [references/github-troubleshooting.md](references/github-troubleshooting.md) | More cases: [references/edge-cases.md](references/edge-cases.md). @@ -277,6 +285,7 @@ More cases: [references/edge-cases.md](references/edge-cases.md). - [references/release-plans.md](references/release-plans.md): creating environment release plans - [references/modes.md](references/modes.md): Preview and Resume details - [references/byok.md](references/byok.md): bring-your-own-key setup +- [references/github-troubleshooting.md](references/github-troubleshooting.md): GitHub still not connected after the user finished - [references/edge-cases.md](references/edge-cases.md): edge cases and existing mappings - [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md): settings rules, write confirmations, and PR diagnosis - [Vega inference settings](https://launchdarkly.com/docs/home/getting-started/vega#vega-inference-settings) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/github-troubleshooting.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/github-troubleshooting.md new file mode 100644 index 00000000..08a308bd --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/github-troubleshooting.md @@ -0,0 +1,12 @@ +# GitHub still not connected after "Done" + +The connect link acts on whatever LaunchDarkly session the browser has. This connection is signed in as the member from Step 1 (`get-member-self`). If the browser is signed in to a different LaunchDarkly account, or as a different member, the user installs or authorizes the App there, and this connection still sees nothing. + +Check the causes in this order. Stop at the first that applies. + +1. **Different LaunchDarkly account or member in the browser.** SKILL.md Step 3 asks about this first. An install made from the other account stays there; that account's admin can remove it if it was a mistake. +2. **Different LaunchDarkly instance.** The browser is on another instance (for example EU or federal) than this connection. Give the link with the instance this connection uses. +3. **Authorized with the wrong GitHub account** (`flow=authorize` remains). The GitHub account they authorized cannot see the organization's install. They reconnect with a GitHub account that is a member of that organization. +4. **Installed on a different GitHub organization.** The App is now installed, but not on the organization that owns their repo. See the repo-owner check in Step 4. + +If none applies, say what you checked and stop. Do not retry the link more than once without a change. From 7c582f0b8a289dd9ef16d827b1ef841609c0063c Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Wed, 28 Oct 2026 11:43:47 -0700 Subject: [PATCH 23/30] fix(release-autopilot): say the account lacks the Release Autopilot add-on, not the alpha When every settings tool returns 404, both skills now tell the user 'Your account does not have the Release Autopilot add-on enabled.' Drops the remaining alpha wording, including the runbook note. Adds a 17th eval case. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- .../assertions.js | 10 ++++++-- .../promptfooconfig.yaml | 24 +++++++++++++++++++ .../README.md | 2 +- .../SKILL.md | 6 ++--- .../references/edge-cases.md | 2 +- .../README.md | 2 +- .../SKILL.md | 4 ++-- 8 files changed, 41 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a773077e..92be1f8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to this repository will be documented in this file. - Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. -- evals: add the `release-autopilot-onboarding` suite (16 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, a browser signed in to a different account). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. +- evals: add the `release-autopilot-onboarding` suite (17 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, a browser signed in to a different account, an account without the add-on). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0). - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index d9839594..6e46aa96 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -225,12 +225,12 @@ module.exports.cleanupQuestionHasLifecycleLink = (output) => { }; // "Release Autopilot" belongs only in the welcome, the resume phrase, and the -// settings skill's name. +// settings skill's name, and the add-on name. module.exports.noProductNameAfterIntro = (output) => { const turns = ((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? x : (x && x.text) || "")); const texts = [...turns, finalText(output), ...askPrompts(output)] .filter((s) => !/Release Autopilot Setup|in a few short steps/i.test(s)) - .map((s) => s.replace(/continue Release Autopilot setup|Release Autopilot settings skill|release-autopilot-settings/gi, "")); + .map((s) => s.replace(/continue Release Autopilot setup|Release Autopilot settings skill|release-autopilot-settings|Release Autopilot add-on/gi, "")); const hits = texts.filter((s) => /Release Autopilot/.test(s)); return result(hits.length === 0, hits.length ? `said "Release Autopilot" after the welcome: "${hits[0].slice(Math.max(0, hits[0].indexOf("Release Autopilot") - 40), hits[0].indexOf("Release Autopilot") + 40)}"` : "uses capability names"); }; @@ -253,3 +253,9 @@ module.exports.asksWhichBrowserAccount = (output) => { const pass = askPrompts(output).some((p) => p.includes("admin@acme.example") && !p.includes("flow=install")); return result(pass, pass ? "asked which account the browser uses, naming this connection's member" : "did not ask about the browser account with the member's email"); }; + +module.exports.saysAddOnNotEnabled = (output) => { + const text = allText(output); + const pass = /Release Autopilot add-on enabled/i.test(text) && !/\balpha\b/i.test(text); + return result(pass, pass ? "says the add-on is not enabled" : "did not use the add-on wording (or mentioned the alpha)"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 739ecb58..6a2805aa 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -650,3 +650,27 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:asksWhichBrowserAccount metric: diagnoses_browser_account weight: 4 + + # ------------------------------------------------------------------ + # Every Release Autopilot tool returns 404: the account does not have + # the add-on. The skill says so in those words and stops. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Add-on not enabled: every settings tool 404s, says so, zero writes" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 10 + mock_tool_overrides: + get-release-autopilot-settings: { error: "LaunchDarkly API error: 404 Not Found" } + list-release-autopilot-repo-settings: { error: "LaunchDarkly API error: 404 Not Found" } + list-release-autopilot-github-repos: { error: "LaunchDarkly API error: 404 Not Found" } + get-release-autopilot-repo-settings: { error: "LaunchDarkly API error: 404 Not Found" } + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:saysAddOnNotEnabled + metric: add_on_wording + weight: 4 diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md index 8a5d6b8d..242ac3c8 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -20,7 +20,7 @@ This skill teaches agents how to: ## Prerequisites -The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools, and `list-release-policies` and `list-metrics` for auto-releasing (`list-release-plans` and `create-release-plan` when the account has environment release plans). Release Autopilot is in alpha, so accounts outside the alpha get 404s. +The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools, and `list-release-policies` and `list-metrics` for auto-releasing (`list-release-plans` and `create-release-plan` when the account has environment release plans). Accounts without the Release Autopilot add-on enabled get 404s. ## Usage diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 60f4be72..6796e6f1 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -35,7 +35,7 @@ This skill requires the remotely hosted LaunchDarkly MCP server. If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. -Release Autopilot is in alpha. If every Release Autopilot tool returns 404, the account is not in the alpha. Tell the user and stop. +If every Release Autopilot tool returns 404, the account does not have the Release Autopilot add-on enabled. Tell the user "Your account does not have the Release Autopilot add-on enabled," and stop. ## Modes @@ -61,7 +61,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Speaking to the user - Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("preview mode", "policy mode", "plan mode"); say "Here's what would change", "your account uses release policies", and "Verification", not "Step 9". -- Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup" and the name of the settings skill. +- Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup", the name of the settings skill, and "the Release Autopilot add-on". - Name repos as `owner/name`. Show a repository ID only for a mapped repo with no name. - Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. - Every browser step (GitHub, bring your own key, Lifecycle or release settings) must be done while signed in as the member from Step 1. Say so when you give the link. @@ -265,7 +265,7 @@ If any change was left for an admin or failed, lead with "Partly set up: waiting | Situation | Action | |-----------|--------| -| Every settings tool returns 404 | The account is not in the alpha. Stop and tell the user | +| Every settings tool returns 404 | The account does not have the Release Autopilot add-on enabled. Tell the user and stop | | Member role is `reader` or `no_access` | Preview mode. End with a checklist for an admin | | Release plans return 403 | Use release policies. Create no plans | | Repo not on the install list | Explain the GitHub fix by owner, then list again | diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md index 9e0cc763..47cae1d4 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -4,7 +4,7 @@ | Situation | Action | |-----------|--------| -| Every Release Autopilot tool returns 404 | The account is not in the alpha. Stop and tell the user | +| Every Release Autopilot tool returns 404 | The account does not have the Release Autopilot add-on enabled. Tell the user and stop | | Member role is `reader` or `no_access` | Preview mode. End with the plan as a checklist for an admin | | Member role is `writer` or a custom role | Account writes are marked "needs an admin" and left out of Step 8; repo writes still run | | A write returns 403 | Report which write failed, list the remaining writes for an admin, and stop | diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md index 3e4d98e0..97f6b624 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md @@ -19,7 +19,7 @@ This skill teaches agents how to: The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools (`get-release-autopilot-settings`, `list-release-autopilot-github-repos`, `update-release-autopilot-repo-settings`, and siblings). If `list-release-autopilot-github-repos` returns `githubConnectUrl`, give the user that link. This skill does not complete GitHub OAuth itself. -Release Autopilot settings are in alpha. The endpoints behind these tools are gated by an alpha feature flag, so accounts outside the alpha get 404s. +The endpoints behind these tools require the Release Autopilot add-on, so accounts without it enabled get 404s. ## Usage diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md index 6b3290f4..74a71fdb 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md @@ -24,7 +24,7 @@ This skill requires the remotely hosted LaunchDarkly MCP server. If these tools are missing, stop. Do not invent REST calls, numeric GitHub ids, or "fixes" by guessing settings. -Release Autopilot settings are in alpha: the underlying endpoints are gated by an alpha feature flag. If every Release Autopilot tool returns 404, the account is not in the alpha — tell the user rather than retrying or falling back to REST. +The underlying endpoints require the Release Autopilot add-on. If every Release Autopilot tool returns 404, the account does not have the Release Autopilot add-on enabled — tell the user "Your account does not have the Release Autopilot add-on enabled," rather than retrying or falling back to REST. ## Core Principles @@ -94,7 +94,7 @@ Identify the GitHub repo (`owner/name`) from the PR URL, `git remote`, or the na If all three look fine (account on, repo mapped, repo auto-flagging effective on): - Do **not** flip settings to "try something." -- Point them at the Release Autopilot runbooks. These are internal LaunchDarkly Confluence pages (PD space) while Release Autopilot is in alpha; there is no public docs URL yet. Classification can still fail for reasons this MCP surface cannot see (GitHub App install on the wrong org, PR not in an installed repo, workflow/app permissions, classifier skip rules). +- Point them at the Release Autopilot runbooks. These are internal LaunchDarkly Confluence pages (PD space); there is no public docs URL yet. Classification can still fail for reasons this MCP surface cannot see (GitHub App install on the wrong org, PR not in an installed repo, workflow/app permissions, classifier skip rules). - [Flag Classification (ODD) Runbook](https://launchdarkly.atlassian.net/wiki/spaces/PD/pages/5377097848/WIP+Flag+Classification+ODD+Runbook) — the classifier itself: verdicts (`not-suited`, `already-flagged`), escalations, and the per-PR ledger that records why a PR got the verdict it did. - [GitHub App Runbook](https://launchdarkly.atlassian.net/wiki/spaces/PD/pages/5157388373/GitHub+App+Runbook) — the PR event never reaching Release Autopilot: install scope, permissions, webhooks. - Optional reads: `list-release-autopilot-github-repos` to confirm the repo is on the install list; `autoFlagging.approvalRequired` if they expected a PR and one exists but is waiting on approval. From 6e56d4cf5e74f5cd0ed4bd99c07e996ab60f495b Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Wed, 28 Oct 2026 12:30:05 -0700 Subject: [PATCH 24/30] fix(release-autopilot): route PR diagnosis by request, not account state 'Why didn't my PR get classified?' always runs the settings skill's read-only diagnosis, even on an empty account, and offers onboarding afterward. Onboarding hands diagnosis requests to the settings skill instead of starting the wizard. Adds an 18th eval case. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- .../assertions.js | 12 ++++++++ .../promptfooconfig.yaml | 30 +++++++++++++++++++ .../SKILL.md | 2 ++ .../SKILL.md | 8 +++-- 5 files changed, 50 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 92be1f8d..bc1bcac9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to this repository will be documented in this file. - Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. -- evals: add the `release-autopilot-onboarding` suite (17 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, a browser signed in to a different account, an account without the add-on). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. +- evals: add the `release-autopilot-onboarding` suite (18 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0). - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 6e46aa96..f751141b 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -259,3 +259,15 @@ module.exports.saysAddOnNotEnabled = (output) => { const pass = /Release Autopilot add-on enabled/i.test(text) && !/\balpha\b/i.test(text); return result(pass, pass ? "says the add-on is not enabled" : "did not use the add-on wording (or mentioned the alpha)"); }; + +module.exports.noCapabilityQuestion = (output) => { + const asked = traj(output).filter((t) => t.tool === "ask-question" && + ((t.arguments && t.arguments.options) || []).some((o) => /auto-releasing|auto-cleanup/i.test(String(o.label || "")))); + return result(asked.length === 0, asked.length ? "started the setup wizard (asked for capabilities)" : "did not start the wizard"); +}; + +module.exports.saysAutoFlaggingOff = (output) => { + const text = allText(output).toLowerCase(); + const pass = /auto-flagging (is )?(turned )?off|auto-flagging is not (turned )?on|auto-flagging isn't (turned )?on/.test(text); + return result(pass, pass ? "says auto-flagging is off for the account" : "did not say auto-flagging is off"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 6a2805aa..323f2091 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -674,3 +674,33 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:saysAddOnNotEnabled metric: add_on_wording weight: 4 + + # ------------------------------------------------------------------ + # "Why didn't my PR get classified?" on an account with nothing set up. + # The skill runs the read-only diagnosis instead of the wizard: no + # capability question, no writes, and it says auto-flagging is off. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Diagnosis on an empty account: no wizard, says auto-flagging is off, zero writes" + vars: + user_request: "Why didn't Release Autopilot classify my pull request acme/demo#42?" + max_turns: 12 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCapabilityQuestion + metric: no_wizard + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:saysAutoFlaggingOff + metric: diagnosis_answer + weight: 3 diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 6796e6f1..30da324b 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -78,6 +78,8 @@ When a step is complete, go to the next one unless it says to ask or wait. Treat ### Step 0: Welcome the user +If the user's request is a diagnosis ("why didn't my PR get classified?", "why wasn't it auto-flagged?"), do not start this wizard, whatever the account state: follow the diagnosis in [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified). It is read-only; if it finds that nothing is set up, offer this wizard afterward. + Decide the mode from the user's first message before you write anything. Never announce the mode by name; for a preview request, say "I'll show what would change and make no changes." **In Resume mode, skip this step entirely**: do not show the intro or any part of it, and go straight to Step 1. In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." > **Release Autopilot Setup** diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md index 74a71fdb..e9c5c16c 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md @@ -73,13 +73,15 @@ After they confirm, apply **only** the fields in the proposal. Then verify (do n ## Choose a workflow -- **First-time setup** ("set up Release Autopilot", "onboard us", or no capability on and no repo mapped) → [`launchdarkly-release-autopilot-onboarding`](../launchdarkly-release-autopilot-onboarding/SKILL.md). It gathers every decision first and confirms one plan. -- **"Why didn't Release Autopilot classify / auto-flag my PR?"** (or similar) → [Why didn't my PR get classified?](#why-didnt-my-pr-get-classified) first. Do not write settings in that workflow. +Route on what the user asked for, not on account state. Check these in order: + +- **"Why didn't Release Autopilot classify / auto-flag my PR?"** (or similar) → [Why didn't my PR get classified?](#why-didnt-my-pr-get-classified), even on an account with nothing set up. Do not write settings in that workflow. +- **First-time setup** ("set up Release Autopilot", "onboard us", "get started") → [`launchdarkly-release-autopilot-onboarding`](../launchdarkly-release-autopilot-onboarding/SKILL.md). It gathers every decision first and confirms one plan. - Configure / map / turn on / unmap → [Configure settings](#configure-settings). ## Why didn't my PR get classified? -Use this when the user asks why a PR was not classified, not auto-flagged, or Release Autopilot ignored the PR. **Read-only.** Check in this order. Stop at the first failure and tell them how to fix it (then [Confirm before write](#confirm-before-write) if they ask you to apply the fix). +Use this when the user asks why a PR was not classified, not auto-flagged, or Release Autopilot ignored the PR. **Read-only.** Check in this order. Stop at the first failure and tell them how to fix it (then [Confirm before write](#confirm-before-write) if they ask you to apply the fix). If the answer is that auto-flagging is off for the account or the repo is not mapped, and nothing is set up yet, offer [`launchdarkly-release-autopilot-onboarding`](../launchdarkly-release-autopilot-onboarding/SKILL.md) as the way to set it up. Identify the GitHub repo (`owner/name`) from the PR URL, `git remote`, or the name they gave. If you cannot identify one repo, ask. Do not diagnose a different repo. From c737cf2664640efa442c70b30947cd8a05feccec Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Wed, 28 Oct 2026 12:41:17 -0700 Subject: [PATCH 25/30] feat(release-autopilot-onboarding): offer auto-archival; note where auto-cleanup lives Offer auto-archival as an opt-in capability, set up as a browser step in the project's Lifecycle settings ('Automatic archival', from launchdarkly/gonfalon#75501). Auto-cleanup's section moves to the Vega settings page on accounts where Vega has project settings; the reference now says so. Adds a 19th eval case. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 4 +- .../assertions.js | 5 +++ .../promptfooconfig.yaml | 41 +++++++++++++++++++ skills.json | 4 +- .../README.md | 2 +- .../SKILL.md | 25 +++++++---- .../marketplace.json | 2 + .../references/auto-archival.md | 16 ++++++++ .../references/auto-cleanup.md | 4 +- 9 files changed, 90 insertions(+), 13 deletions(-) create mode 100644 skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-archival.md diff --git a/CHANGELOG.md b/CHANGELOG.md index bc1bcac9..7b2ea72a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,9 +4,9 @@ All notable changes to this repository will be documented in this file. ## Unreleased -- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. +- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing, auto-cleanup, auto-archival), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. -- evals: add the `release-autopilot-onboarding` suite (18 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. +- evals: add the `release-autopilot-onboarding` suite (19 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, auto-archival, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The provider gains `tool_scopes` (scoped tool definitions stay hidden from other suites), `mock_tool_overrides`, `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0). - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. - Claude Code marketplace: add `version`, `author`, `homepage`, `repository`, `license`, `category`, and `keywords` to the `launchdarkly` entry in `.claude-plugin/marketplace.json` so directory listings get the same metadata as `.claude-plugin/plugin.json`. A test keeps the two in sync. diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index f751141b..438a7ecf 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -271,3 +271,8 @@ module.exports.saysAutoFlaggingOff = (output) => { const pass = /auto-flagging (is )?(turned )?off|auto-flagging is not (turned )?on|auto-flagging isn't (turned )?on/.test(text); return result(pass, pass ? "says auto-flagging is off for the account" : "did not say auto-flagging is off"); }; + +module.exports.archivalQuestionHasLifecycleLink = (output) => { + const pass = askPrompts(output).some((p) => p.includes("/settings/lifecycle") && /archiv/i.test(p)); + return result(pass, pass ? "auto-archival question contains the lifecycle settings link" : "no archival question contains /settings/lifecycle"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 323f2091..86b6342d 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -704,3 +704,44 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:saysAutoFlaggingOff metric: diagnosis_answer weight: 3 + + # ------------------------------------------------------------------ + # Auto-archival picked with auto-flagging. It is a browser step on the + # project's Lifecycle settings page: the link is in the question, and + # the only write is the confirmed repo mapping. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Auto-archival: lifecycle settings link in the question, confirmed repo write only" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 22 + mock_tool_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging", "Auto-archival"] + - ["Require approval"] + - ["LaunchDarkly's"] + - ["Done"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:archivalQuestionHasLifecycleLink + metric: lifecycle_link_in_question + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup + metric: no_auto_cleanup + weight: 2 diff --git a/skills.json b/skills.json index f6c0641d..efddcefc 100644 --- a/skills.json +++ b/skills.json @@ -421,7 +421,7 @@ }, { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", "version": "0.2.0-experimental", "license": "Apache-2.0", @@ -433,6 +433,8 @@ "github", "auto-flagging", "auto-releasing", + "auto-cleanup", + "auto-archival", "release-plans", "mcp" ] diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md index 242ac3c8..e76c67fe 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -9,7 +9,7 @@ This skill teaches agents how to: - Ask which capabilities the user wants (auto-flagging, auto-releasing) and set up only what those need - Hand the user the GitHub App install or authorize link, and continue after they finish - Map chosen `owner/name` repos to LaunchDarkly projects -- Offer auto-cleanup and guide the project's automated flag cleanup settings +- Offer auto-cleanup and auto-archival, and guide the project's Lifecycle settings for each - Explain whose AI model account runs the agents (LaunchDarkly's by default, or the user's own Anthropic or OpenAI key) and guide bring-your-own-key setup - Review or create environment release plans for auto-releasing, or show release policies where plans are not available - Confirm one plan that lists every write before making any change, then verify diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 30da324b..1177dc66 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -1,6 +1,6 @@ --- name: launchdarkly-release-autopilot-onboarding -description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: @@ -61,7 +61,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Speaking to the user - Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("preview mode", "policy mode", "plan mode"); say "Here's what would change", "your account uses release policies", and "Verification", not "Step 9". -- Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup", the name of the settings skill, and "the Release Autopilot add-on". +- Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup", "auto-archival"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup", the name of the settings skill, and "the Release Autopilot add-on". - Name repos as `owner/name`. Show a repository ID only for a mapped repo with no name. - Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. - Every browser step (GitHub, bring your own key, Lifecycle or release settings) must be done while signed in as the member from Step 1. Say so when you give the link. @@ -84,7 +84,7 @@ Decide the mode from the user's first message before you write anything. Never a > **Release Autopilot Setup** > -> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose whose AI model account runs the agents, then set up releases and flag cleanup for the capabilities you pick. +> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose whose AI model account runs the agents, then set up releases, flag cleanup, and archival for the capabilities you pick. > > Nothing changes until you approve the full plan. @@ -119,13 +119,14 @@ Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: - label: "Auto-flagging (Recommended)", description: "Reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" - label: "Auto-releasing", description: "After a flagged change deploys, releases it with the environment's release policy or release plan. This can ship code to users" - label: "Auto-cleanup", description: "On a schedule, opens pull requests that remove the code for flags that are ready for removal. This removes code" +- label: "Auto-archival", description: "Archives flags that are ready to archive, after safety checks and a grace period" Ask about approval only if the user picked auto-flagging **and** the account does not already have auto-flagging on with approval required. If it already does, say "Auto-flagging already requires approval for every mapped repo. Say 'change approval' if you want to turn it off," and move on unless they say so. Otherwise, or if they ask to change it, ask header "Approval": - label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before any code changes. This is an account setting, so it applies to every mapped repo" - label: "No approval", description: "Auto-flagging changes PRs without an approval step, in every mapped repo" -Auto-cleanup is opt-in like auto-releasing: never recommend it by default. If the user picks no capability, stop and tell them there is nothing to set up. +Auto-cleanup and auto-archival are opt-in like auto-releasing: never recommend them by default. If the user picks no capability, stop and tell them there is nothing to set up. ### Step 3: Connect GitHub @@ -174,7 +175,7 @@ Record the answer as reported by the user, not as verified. ### Step 6: Set up releases and cleanup -Skip this step if the user picked neither auto-releasing nor auto-cleanup. For auto-cleanup, follow [Auto-cleanup](#auto-cleanup) below. The rest of this step is for auto-releasing. +Skip this step if the user picked none of auto-releasing, auto-cleanup, or auto-archival. For those two, follow [Auto-cleanup](#auto-cleanup) and [Auto-archival](#auto-archival) below. The rest of this step is for auto-releasing. For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then check once per account whether it uses release plans or release policies. @@ -203,13 +204,22 @@ To create a plan, follow [references/release-plans.md](references/release-plans. #### Auto-cleanup -Auto-cleanup is the per-project **Automated flag cleanup** setting on the Lifecycle settings page. The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: +Auto-cleanup is the per-project **Automated flag cleanup** setting, on the Lifecycle settings page (or the Vega settings page, for accounts where Vega has its own project settings). The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: - label: "Done", description: "I turned on automated flag cleanup for every project" - label: "Later", description: "Skip this. I'll turn it on from Lifecycle settings later" Record the answer as reported by the user, and list "Done" under browser steps in Step 7. +#### Auto-archival + +Auto-archival is the per-project **Automatic archival** setting on the Lifecycle settings page. No MCP tool manages it, so this is a browser step. For each project from Step 4, follow [references/auto-archival.md](references/auto-archival.md), then ask header "Auto-archival", with the link in the question text: + +- label: "Done", description: "I turned on automatic archival for every project" +- label: "Later", description: "Skip this, or the section isn't there yet. I'll turn it on from Lifecycle settings later" + +Record the answer as reported by the user, and list "Done" under browser steps in Step 7. + ### Step 7: Confirm the plan **STOP. Do not write anything until the user explicitly confirms.** A detailed request at the start is not approval. @@ -220,7 +230,7 @@ List every write in the order you will make it. For each write, state the curren 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." 3. **Account settings**: only fields that change. If auto-releasing turns on, say it can ship code. For a member who is not an admin or owner, list account writes under "Needs an admin" instead, and say an admin can make them with the settings skill. Do not name a settings page; there is none to point to. -4. **Browser steps the user reported done**: GitHub, the AI model choice, and auto-cleanup, marked as reported, not verified. +4. **Browser steps the user reported done**: GitHub, the AI model choice, auto-cleanup, and auto-archival, marked as reported, not verified. 5. **Follow-ups for the user**: release policies to add, metric events to instrument. If there are no writes, say so, list the follow-ups, and stop. @@ -287,6 +297,7 @@ More cases: [references/edge-cases.md](references/edge-cases.md). - [references/release-plans.md](references/release-plans.md): creating environment release plans - [references/modes.md](references/modes.md): Preview and Resume details - [references/byok.md](references/byok.md): bring-your-own-key setup +- [references/auto-cleanup.md](references/auto-cleanup.md) / [references/auto-archival.md](references/auto-archival.md): turning on flag cleanup and archival - [references/github-troubleshooting.md](references/github-troubleshooting.md): GitHub still not connected after the user finished - [references/edge-cases.md](references/edge-cases.md): edge cases and existing mappings - [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md): settings rules, write confirmations, and PR diagnosis diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json index cec8016f..32071da7 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json @@ -12,6 +12,8 @@ "github", "auto-flagging", "auto-releasing", + "auto-cleanup", + "auto-archival", "release-plans", "mcp" ], diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-archival.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-archival.md new file mode 100644 index 00000000..43f8a6b4 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-archival.md @@ -0,0 +1,16 @@ +# Auto-archival + +Auto-archival archives flags that are ready to archive, after archival safety checks and a grace period. It is set per project, in the **Automatic archival** section of Lifecycle settings. No MCP tool manages it. The section is still rolling out, so some accounts do not see it yet. + +For each project, give the user `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). In the "Automatic archival" section they: + +1. Turn on **Enable automatic archival**, and confirm in the "Enable automatic archival?" dialog. +2. Set the grace period before a ready flag is archived. +3. If the section shows **Require approval before archiving**, choose whether archivals need approval. +4. Click **Save**. + +"View ready to archive flags" and "View auto-archived flags" open the flags list filtered to each set, so they can check what it will do and what it did. + +If the page has no "Automatic archival" section, the feature is not available for their account yet: record it as a follow-up, not as done. + +Record the result as reported by the user. It cannot be verified. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md index 2d32900b..1356ddaf 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md @@ -1,8 +1,8 @@ # Auto-cleanup -Auto-cleanup runs Vega on a schedule to find flags that are ready for code removal and open pull requests that remove their code. It is set per project, in the **Automated flag cleanup** section of Lifecycle settings. The MCP settings tools do not manage it yet. +Auto-cleanup runs Vega on a schedule to find flags that are ready for code removal and open pull requests that remove their code. It is set per project, in the **Automated flag cleanup** section of Lifecycle settings. On accounts where Vega has its own project settings, the section is on the Vega settings page (`/projects/{projectKey}/settings/vega`) instead. The MCP settings tools do not manage it yet. -For each project, give the user `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). In the "Automated flag cleanup" section they: +For each project, give the user `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). If that page has no "Automated flag cleanup" section, they use `/projects/{projectKey}/settings/vega`. In the "Automated flag cleanup" section they: 1. Check **Enable automated flag cleanups**. 2. Choose a **Schedule frequency**: 1 day, 7 days (the default), or 30 days. From ebfff38e21ac22246c8f5d583aa4802ec378c20f Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Wed, 28 Oct 2026 12:59:59 -0700 Subject: [PATCH 26/30] fix(release-autopilot-onboarding): run Step 6 parts only for picked capabilities Step 6 now runs release setup, auto-cleanup, and auto-archival each only when the user picked that capability, so nobody is walked through turning on code removal they did not ask for. Evals check that no unpicked cleanup or archival step appears. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../assertions.js | 13 ++++++++ .../promptfooconfig.yaml | 32 +++++++++++++++++++ .../SKILL.md | 12 ++++++- 3 files changed, 56 insertions(+), 1 deletion(-) diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 438a7ecf..eaae63c4 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -276,3 +276,16 @@ module.exports.archivalQuestionHasLifecycleLink = (output) => { const pass = askPrompts(output).some((p) => p.includes("/settings/lifecycle") && /archiv/i.test(p)); return result(pass, pass ? "auto-archival question contains the lifecycle settings link" : "no archival question contains /settings/lifecycle"); }; + +// Step 6 must not walk the user through cleanup or archival they did not pick. +const asksAbout = (output, re) => askPrompts(output).filter((p) => re.test(p)); +const CLEANUP = /auto-cleanup|automated flag cleanup|flag cleanup/i; +const ARCHIVAL = /auto-archival|automatic archival/i; +module.exports.noCleanupQuestion = (output) => { + const hit = asksAbout(output, CLEANUP).filter((p) => !/capabilit/i.test(p)); + return result(hit.length === 0, hit.length ? `asked about unpicked auto-cleanup: "${hit[0].slice(0, 90)}"` : "no auto-cleanup step"); +}; +module.exports.noArchivalQuestion = (output) => { + const hit = asksAbout(output, ARCHIVAL).filter((p) => !/capabilit/i.test(p)); + return result(hit.length === 0, hit.length ? `asked about unpicked auto-archival: "${hit[0].slice(0, 90)}"` : "no auto-archival step"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 86b6342d..e3394893 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -68,6 +68,14 @@ tests: - ["LaunchDarkly's"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting metric: confirms_before_writing @@ -127,6 +135,14 @@ tests: - ["Test", "Production"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:releasePlansCalledAtMostOnce metric: plans_403_account_wide @@ -412,6 +428,14 @@ tests: - ["Leave rules"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:planBeforeRepoWrite metric: plan_before_repo_write @@ -587,6 +611,10 @@ tests: - ["Done"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup metric: no_auto_cleanup @@ -729,6 +757,10 @@ tests: - ["Done"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:archivalQuestionHasLifecycleLink metric: lifecycle_link_in_question diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 1177dc66..7de6cfa5 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -175,7 +175,11 @@ Record the answer as reported by the user, not as verified. ### Step 6: Set up releases and cleanup -Skip this step if the user picked none of auto-releasing, auto-cleanup, or auto-archival. For those two, follow [Auto-cleanup](#auto-cleanup) and [Auto-archival](#auto-archival) below. The rest of this step is for auto-releasing. +Run each part of this step only for a capability the user picked in Step 2, and skip the step if they picked none of them. Never walk the user through a capability they did not pick: + +- Auto-releasing picked: the release setup below. +- Auto-cleanup picked: [Auto-cleanup](#auto-cleanup). +- Auto-archival picked: [Auto-archival](#auto-archival). For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then check once per account whether it uses release plans or release policies. @@ -204,6 +208,9 @@ To create a plan, follow [references/release-plans.md](references/release-plans. #### Auto-cleanup +Only if the user picked auto-cleanup. + + Auto-cleanup is the per-project **Automated flag cleanup** setting, on the Lifecycle settings page (or the Vega settings page, for accounts where Vega has its own project settings). The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: - label: "Done", description: "I turned on automated flag cleanup for every project" @@ -213,6 +220,9 @@ Record the answer as reported by the user, and list "Done" under browser steps i #### Auto-archival +Only if the user picked auto-archival. + + Auto-archival is the per-project **Automatic archival** setting on the Lifecycle settings page. No MCP tool manages it, so this is a browser step. For each project from Step 4, follow [references/auto-archival.md](references/auto-archival.md), then ask header "Auto-archival", with the link in the question text: - label: "Done", description: "I turned on automatic archival for every project" From 9bd5f90ffd9b10a31691350e712ad231130c1850 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Fri, 30 Oct 2026 09:46:52 -0700 Subject: [PATCH 27/30] fix(release-autopilot): stop offering 'No approval' for auto-flagging The agent backend loads only the enabled flags from code automation settings and ignores approvalRequired, so every auto-flagging proposal waits for '@launchdarkly approve' regardless. Onboarding no longer asks about approval or sends approvalRequired; it explains the approve step instead. The settings skill notes that turning approval off has no effect today. Evals check approvalRequired is never sent. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../assertions.js | 10 +++++++ .../promptfooconfig.yaml | 28 +++++++++++++------ .../SKILL.md | 15 ++++------ .../SKILL.md | 2 +- 4 files changed, 37 insertions(+), 18 deletions(-) diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 44d13335..13693af8 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -311,3 +311,13 @@ module.exports.noUnconfirmedWrites = (output) => { if (writes(output).length === 0) return result(true, "no writes"); return module.exports.confirmsBeforeWriting(output); }; + +// The agent backend ignores approvalRequired; auto-flagging always waits for +// "@launchdarkly approve". The skill never sends the field. +module.exports.neverSendsApproval = (output) => { + const bad = writes(output).filter((t) => { + const a = t.arguments || {}; + return (a.autoFlagging && "approvalRequired" in a.autoFlagging) || (a.autoReleasing && "approvalRequired" in a.autoReleasing); + }); + return result(bad.length === 0, bad.length ? `sent approvalRequired via ${bad[0].tool}` : "never sent approvalRequired"); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index ddc71840..947fad3f 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -65,12 +65,15 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - - ["Require approval"] - ["acme/demo"] - ["default"] - ["LaunchDarkly's"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion metric: no_unpicked_cleanup @@ -266,7 +269,6 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - - ["Require approval"] - ["Need an org owner"] assert: - type: javascript @@ -301,12 +303,15 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - - ["Require approval"] - ["acme/demo"] - ["default"] - ["LaunchDarkly's"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite metric: no_account_write @@ -345,7 +350,6 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - - ["Require approval"] - ["LaunchDarkly's"] - ["Change something"] assert: @@ -535,6 +539,10 @@ tests: mock_ask_question_answers: - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noIntro metric: no_intro @@ -573,7 +581,6 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - - ["Require approval"] - ["LaunchDarkly's"] - ["Looks good"] assert: @@ -609,11 +616,14 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging", "Auto-cleanup"] - - ["Require approval"] - ["LaunchDarkly's"] - ["Done"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion metric: no_unpicked_archival @@ -665,7 +675,6 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging"] - - ["Require approval"] - ["Done"] - ["A different one"] assert: @@ -752,11 +761,14 @@ tests: mock_ask_question_match: true mock_ask_question_answers: - ["Auto-flagging", "Auto-archival"] - - ["Require approval"] - ["LaunchDarkly's"] - ["Done"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion metric: no_unpicked_cleanup diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 7de6cfa5..4dc49e8b 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -52,7 +52,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the ## Core Principles 1. **Only set up what they picked.** Release setup matters only for auto-releasing. If the user picks auto-flagging only, skip Step 6. -2. **Least privilege by default.** Recommend auto-flagging with approval required. Auto-releasing is opt-in and you must say it can ship code. +2. **Least privilege by default.** Recommend auto-flagging, which always waits for a person to approve its proposed change. Auto-releasing is opt-in and you must say it can ship code. 3. **Gather, confirm, then write.** No write happens before [Step 7](#step-7-confirm-the-plan). The confirmation lists every write with its current and proposed state. This holds in every mode, including Resume and requests that already state every answer: a restated pick is an answer, never approval. Step 1's reads, including the role check, always run first. 4. **Don't ask what the account already answers.** If live state already settles a question, state the answer and move on. 5. **Never act in the user's browser.** GitHub install, GitHub authorization, release policy edits, and bring-your-own-key setup are browser steps. Give the exact link and wait for the user to say they finished. @@ -63,7 +63,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the - Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("preview mode", "policy mode", "plan mode"); say "Here's what would change", "your account uses release policies", and "Verification", not "Step 9". - Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup", "auto-archival"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup", the name of the settings skill, and "the Release Autopilot add-on". - Name repos as `owner/name`. Show a repository ID only for a mapped repo with no name. -- Describe each change in words ("Turn on auto-flagging, with approval required"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. +- Describe each change in words ("Turn on auto-flagging for the account"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. - Every browser step (GitHub, bring your own key, Lifecycle or release settings) must be done while signed in as the member from Step 1. Say so when you give the link. - State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"), and do not invent UI locations: give only the pages this skill names. For an account change left for an admin, tell them to run [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). - Keep each message to the current step, and do not narrate tool calls. @@ -121,10 +121,7 @@ Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: - label: "Auto-cleanup", description: "On a schedule, opens pull requests that remove the code for flags that are ready for removal. This removes code" - label: "Auto-archival", description: "Archives flags that are ready to archive, after safety checks and a grace period" -Ask about approval only if the user picked auto-flagging **and** the account does not already have auto-flagging on with approval required. If it already does, say "Auto-flagging already requires approval for every mapped repo. Say 'change approval' if you want to turn it off," and move on unless they say so. Otherwise, or if they ask to change it, ask header "Approval": - -- label: "Require approval (Recommended)", description: "A person approves each auto-flagging PR before any code changes. This is an account setting, so it applies to every mapped repo" -- label: "No approval", description: "Auto-flagging changes PRs without an approval step, in every mapped repo" +Do not ask about approval. Auto-flagging always waits for a person: it posts its proposed change on the pull request, and nothing changes until someone replies `@launchdarkly approve` (or `approve flag`) or presses Approve on the check. If the user picked auto-flagging, say that in one sentence. Never send `approvalRequired`; the backend does not read it today, so turning it off would not skip that step. Auto-cleanup and auto-archival are opt-in like auto-releasing: never recommend them by default. If the user picks no capability, stop and tell them there is nothing to set up. @@ -249,7 +246,7 @@ If there are no writes, say so, list the follow-ups, and stop. ``` - [ ] Map acme/demo to project default, with auto-releasing off for this repository -- [ ] Turn on auto-flagging for the account, with approval required +- [ ] Turn on auto-flagging for the account ``` **Setup mode:** use `AskUserQuestion` with header "Confirm": @@ -280,7 +277,7 @@ If a write fails with 403, the member lacks that permission. Report which write If any change was left for an admin or failed, lead with "Partly set up: waiting on an admin" and the admin checklist, and never call the setup complete. Then report a short checklist with each item marked verified, reported by the user (browser steps the user did in this flow), or already set up (steps the flow skipped because they were already done), then the follow-ups. Tell the user how to see it work; if a change is waiting on an admin, make that conditional ("Once an admin turns on auto-flagging, open a pull request…"): -- Open a pull request in a mapped repo. Auto-flagging reads it within a few minutes. With approval required, the auto-flagging PR waits for approval. +- Open a pull request in a mapped repo. Auto-flagging reads it within a few minutes and posts its proposed change on the PR; nothing changes until someone replies `@launchdarkly approve` or presses Approve on the check. - If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. ## Edge Cases @@ -298,7 +295,7 @@ More cases: [references/edge-cases.md](references/edge-cases.md). ## What NOT to Do - Don't write before Step 7's "Looks good", or at all in Preview mode. -- Don't turn on auto-releasing, or turn off approval, unless the user picked it. +- Don't turn on auto-releasing unless the user picked it, and never send `approvalRequired`. - Don't map unpicked repos, iterate over the install list, or remap a repo to another project. - Don't act in the user's browser, change release policies, or report a browser step as verified. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md index e9c5c16c..b7fd8566 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md @@ -115,7 +115,7 @@ If `githubConnectUrl` is present, give the user that link and stop this turn. Do If an account PATCH is needed, follow [Confirm before write](#confirm-before-write), then `update-release-autopilot-settings` with **only** the confirmed fields: -- `autoFlagging.enabled` / `autoFlagging.approvalRequired` (approval is only valid here) +- `autoFlagging.enabled` / `autoFlagging.approvalRequired` (approval is only valid here). The agent backend does not read `approvalRequired` today: every auto-flagging proposal waits for `@launchdarkly approve` either way. If a user asks to turn approval off, tell them it currently has no effect before proposing the change. - `autoReleasing.enabled` Do not send `autoCleanup`. Requires permission to update account Release Autopilot settings. From b53ed31de72e8462fd8df232fd802a88c655ac94 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Fri, 30 Oct 2026 11:46:12 -0700 Subject: [PATCH 28/30] fix(release-autopilot-onboarding): authorize-flow options; gate release setup on auto-releasing GitHub step: flow=authorize no longer offers 'Need an org owner' (the App is installed); it asks the member to authorize with a GitHub account in the installing organization, with an 'I can't authorize' path. Step 6: environments and release plan/policy checks move under a Release setup section that runs only when auto-releasing is picked. Evals add an authorize-flow case and check cleanup- or archival-only setups never touch release setup. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- .../assertions.js | 20 +++++++++ .../promptfooconfig.yaml | 44 +++++++++++++++++++ .../SKILL.md | 26 +++++++---- 4 files changed, 82 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 887fb053..b0d5e39e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to this repository will be documented in this file. - Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing, auto-cleanup, auto-archival), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. - `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. -- evals: add the `release-autopilot-onboarding` suite (19 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install paths, deprecated MCP, a failed write, auto-cleanup, auto-archival, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The suite keeps its tool definitions and mocks in its own folder (`tool_definitions`, `tool_mocks`). The provider gains `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. +- evals: add the `release-autopilot-onboarding` suite (20 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install and authorize paths, deprecated MCP, a failed write, auto-cleanup, auto-archival, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The suite keeps its tool definitions and mocks in its own folder (`tool_definitions`, `tool_mocks`). The provider gains `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `onboarding/mcp-configure` 0.1.1: Claude Code loads MCP servers only at session start, so a server added mid-session needs the user to quit, resume with `claude --continue`, and authenticate from `/mcp`. The skill no longer says Claude Code needs no restart or that its OAuth prompt appears on the first tool call. - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0). - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 13693af8..7f0c5137 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -321,3 +321,23 @@ module.exports.neverSendsApproval = (output) => { }); return result(bad.length === 0, bad.length ? `sent approvalRequired via ${bad[0].tool}` : "never sent approvalRequired"); }; + +// Release setup runs only for auto-releasing: no environment question and no +// release plan or policy lookup otherwise. +module.exports.noReleaseSetup = (output) => { + const steps = traj(output); + const lookups = steps.filter((t) => t.tool === "list-release-plans" || t.tool === "list-release-policies"); + const envQ = steps.filter((t) => t.tool === "ask-question" && /environment/i.test(String((t.arguments && t.arguments.prompt) || ""))); + const pass = lookups.length === 0 && envQ.length === 0; + return result(pass, `release lookups: ${lookups.length}, environment questions: ${envQ.length}`); +}; + +// On flow=authorize the App is installed; the question is about authorizing, +// with no org-owner pause. +module.exports.authorizeQuestionNoOrgOwner = (output) => { + const q = traj(output).filter((t) => t.tool === "ask-question" && String(t.arguments.prompt || "").includes("flow=authorize")); + if (!q.length) return result(false, "no question contains the authorize link"); + const labels = (q[0].arguments.options || []).map((o) => String(o.label || "")); + const pass = /authoriz/i.test(q[0].arguments.prompt) && !labels.some((l) => /org owner/i.test(l)); + return result(pass, `prompt mentions authorize: ${/authoriz/i.test(q[0].arguments.prompt)}; options: ${labels.join(" | ")}`); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 947fad3f..7185f67b 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -620,6 +620,10 @@ tests: - ["Done"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noReleaseSetup + metric: no_release_setup + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval metric: never_sends_approval @@ -765,6 +769,10 @@ tests: - ["Done"] - ["Looks good"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noReleaseSetup + metric: no_release_setup + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval metric: never_sends_approval @@ -789,3 +797,39 @@ tests: value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup metric: no_auto_cleanup weight: 2 + + # ------------------------------------------------------------------ + # The App is installed but this member has not authorized GitHub. The + # question is about authorizing (no org-owner pause), and nothing is + # written while GitHub is not connected. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Authorize flow: asks to authorize, no org-owner option, zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for my repos." + max_turns: 12 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: true } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + list-release-autopilot-github-repos: + installed: true + userAuthorized: false + repositories: [] + totalCount: 0 + githubConnectUrl: "https://app.launchdarkly.com/github-connect?flow=authorize" + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["I can't authorize"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:authorizeQuestionNoOrgOwner + metric: authorize_question + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 3 diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 4dc49e8b..1ab3d8b4 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -30,7 +30,7 @@ This skill requires the remotely hosted LaunchDarkly MCP server. - `list-release-policies` **Used when available:** -- `list-release-plans` / `create-release-plan` / `list-metrics`: environment release plans (see [Check whether the account uses release plans](#check-whether-the-account-uses-release-plans)) +- `list-release-plans` / `create-release-plan` / `list-metrics`: environment release plans (see [Release setup](#release-setup)) - `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. @@ -134,10 +134,14 @@ Otherwise read the URL's `flow`: - `flow=install`: the LaunchDarkly GitHub App is not installed on their GitHub organization. None of these capabilities can see a pull request until it is. - `flow=authorize`: the App is installed, but this member has not authorized GitHub, so repos cannot be listed. -Give the user the exact URL. The link acts on whatever LaunchDarkly session the browser has, so tell them to open it while signed in as the member from Step 1 (for example, "Open it while signed in to LaunchDarkly as admin@acme.example"). To install, they must be a GitHub organization owner, or they must request the install from one. Then ask header "GitHub", with the URL in the question text (for example, "Open https://… to install the LaunchDarkly GitHub App. Done?"): +Give the user the exact URL. The link acts on whatever LaunchDarkly session the browser has, so tell them to open it while signed in as the member from Step 1 (for example, "Open it while signed in to LaunchDarkly as admin@acme.example"). Then ask header "GitHub", with the URL in the question text. The options depend on the flow: -- label: "Done", description: "I finished the GitHub step" -- label: "Need an org owner", description: "I can't install the App myself. Pause setup here" +- `flow=install`: to install, they must be a GitHub organization owner, or request the install from one. Question: "Open https://… to install the LaunchDarkly GitHub App. Done?" + - label: "Done", description: "I installed the App" + - label: "Need an org owner", description: "I can't install the App myself. Pause setup here" +- `flow=authorize`: the App is already installed, so no org owner is needed. They authorize with a GitHub account that is a member of the organization where the App is installed. Question: "Open https://… and authorize with a GitHub account in the organization that installed the App. Done?" + - label: "Done", description: "I authorized GitHub" + - label: "I can't authorize", description: "My GitHub account can't see that organization's install" On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, do not repeat the link. The most common cause is a browser signed in to a different LaunchDarkly account or member, so the App was installed there. Ask header "Browser account", with this connection's email in the question text (for example, "This connection is signed in as admin@acme.example. Which account is your browser signed in to?"): @@ -145,7 +149,7 @@ On "Done", call `list-release-autopilot-github-repos` again. If it still returns - label: "A different one", description: "My browser is signed in to another account or member" - label: "Not sure", description: "I need to check" -For "A different one" or "Not sure", tell them to check at `/profile`, sign in as the member from Step 1, open the link again, and finish there; then list again. For "The same", check the other causes in [references/github-troubleshooting.md](references/github-troubleshooting.md). On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. +For "A different one" or "Not sure", tell them to check at `/profile`, sign in as the member from Step 1, open the link again, and finish there; then list again. For "The same", check the other causes in [references/github-troubleshooting.md](references/github-troubleshooting.md). On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. On "I can't authorize", tell them to sign in to GitHub with an account that is a member of that organization (or ask an org owner to add theirs), open the link again, and continue; see the GitHub-identity cause in [references/github-troubleshooting.md](references/github-troubleshooting.md). ### Step 4: Map repositories @@ -174,27 +178,31 @@ Record the answer as reported by the user, not as verified. Run each part of this step only for a capability the user picked in Step 2, and skip the step if they picked none of them. Never walk the user through a capability they did not pick: -- Auto-releasing picked: the release setup below. +- Auto-releasing picked: [Release setup](#release-setup). - Auto-cleanup picked: [Auto-cleanup](#auto-cleanup). - Auto-archival picked: [Auto-archival](#auto-archival). +#### Release setup + +Only if the user picked auto-releasing. Otherwise skip to [Auto-cleanup](#auto-cleanup) or [Auto-archival](#auto-archival), and do not ask about environments or look up release plans or policies. + For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then check once per account whether it uses release plans or release policies. -#### Check whether the account uses release plans +##### Check whether the account uses release plans Call `list-release-plans` for the first chosen environment. - **Release policies** if the call returns 403 `Plan does not allow this operation`, or if `list-release-plans` is not available. The 403 is account-wide, so do not call it for other environments. This is the mode for every account today: auto-releasing requires a complete release policy for each environment. - **Release plans** if the call succeeds. Environment release plans replace release policies by general availability. -#### Release policies (today) +##### Release policies (today) Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. This skill does not change policies. - No warnings for the chosen environments: say that auto-releasing in this project uses those policies. - Any warning, such as `missing_policy` or `incomplete_policy`: auto-releasing cannot work there yet. Tell the user to add or complete a release policy in the project's release settings (`/projects/{projectKey}/settings/releases` on their LaunchDarkly instance). Record a repo override that keeps auto-releasing off for each repo in that project, and say in Step 7 that they can turn it on with the settings skill once the policy is complete. -#### Release plans (by GA) +##### Release plans (by GA) For each chosen environment, call `list-release-plans`. If a complete plan exists, show its name, phases, and metrics, and ask header "Plan": From b40e45a49c2d4e9c747a34d64e52ea7b1bc7e065 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Fri, 30 Oct 2026 12:06:34 -0700 Subject: [PATCH 29/30] fix(release-autopilot-onboarding): pause when GitHub authorization fails 'I can't authorize' now pauses like 'Need an org owner' (summary, link, resume phrase) instead of continuing, and the wizard never maps repos while GitHub still needs connecting, so an empty repo list is not mistaken for an organization without the App. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) --- evals/release-autopilot-onboarding/assertions.js | 11 +++++++++++ .../release-autopilot-onboarding/promptfooconfig.yaml | 6 +++++- .../SKILL.md | 4 +++- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 7f0c5137..77106c96 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -341,3 +341,14 @@ module.exports.authorizeQuestionNoOrgOwner = (output) => { const pass = /authoriz/i.test(q[0].arguments.prompt) && !labels.some((l) => /org owner/i.test(l)); return result(pass, `prompt mentions authorize: ${/authoriz/i.test(q[0].arguments.prompt)}; options: ${labels.join(" | ")}`); }; + +// While GitHub is disconnected the wizard pauses: no repo question, and the +// final message gives the resume phrase. +module.exports.pausesBeforeMapping = (output) => { + const repoQ = traj(output).filter((t) => t.tool === "ask-question" && + ((t.arguments && t.arguments.options) || []).some((o) => /\w+\/\w+/.test(String(o.label || ""))) && + /repo/i.test(String((t.arguments && t.arguments.prompt) || ""))); + const resume = /continue Release Autopilot setup/i.test(finalText(output)); + const pass = repoQ.length === 0 && resume; + return result(pass, `repo questions: ${repoQ.length}; resume phrase in final message: ${resume}`); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index 7185f67b..e608f72a 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -804,7 +804,7 @@ tests: # written while GitHub is not connected. # ------------------------------------------------------------------ - providers: ["mcp"] - description: "Authorize flow: asks to authorize, no org-owner option, zero writes" + description: "Authorize flow: asks to authorize, no org-owner option, pauses on failure, zero writes" vars: user_request: "Set up Release Autopilot auto-flagging for my repos." max_turns: 12 @@ -825,6 +825,10 @@ tests: - ["Auto-flagging"] - ["I can't authorize"] assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:pausesBeforeMapping + metric: pauses_before_mapping + weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:authorizeQuestionNoOrgOwner metric: authorize_question diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index 1ab3d8b4..a6cdde02 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -149,7 +149,9 @@ On "Done", call `list-release-autopilot-github-repos` again. If it still returns - label: "A different one", description: "My browser is signed in to another account or member" - label: "Not sure", description: "I need to check" -For "A different one" or "Not sure", tell them to check at `/profile`, sign in as the member from Step 1, open the link again, and finish there; then list again. For "The same", check the other causes in [references/github-troubleshooting.md](references/github-troubleshooting.md). On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. On "I can't authorize", tell them to sign in to GitHub with an account that is a member of that organization (or ask an org owner to add theirs), open the link again, and continue; see the GitHub-identity cause in [references/github-troubleshooting.md](references/github-troubleshooting.md). +For "A different one" or "Not sure", tell them to check at `/profile`, sign in as the member from Step 1, open the link again, and finish there; then list again. For "The same", check the other causes in [references/github-troubleshooting.md](references/github-troubleshooting.md). On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. On "I can't authorize", pause the same way: summarize the decisions so far, tell them to sign in to GitHub with an account that is a member of that organization (or ask an org owner to add theirs), repeat the exact URL, tell them to say "continue Release Autopilot setup" once they have authorized, and stop. See the GitHub-identity cause in [references/github-troubleshooting.md](references/github-troubleshooting.md). + +Never go on to Step 4 while `list-release-autopilot-github-repos` still returns `githubConnectUrl`: an empty repo list then means GitHub is not connected, not that the repo's organization lacks the App. ### Step 4: Map repositories From f0a08d6952028eb0f3394d11cf1dfaa72d96cbe4 Mon Sep 17 00:00:00 2001 From: Farhan Sareshwala Date: Fri, 30 Oct 2026 12:10:26 -0700 Subject: [PATCH 30/30] fix(release-autopilot-onboarding): give the exact Lifecycle settings path inline An eval run showed the agent guessing /settings/projects/{key}/lifecycle when the path lived only in references. The auto-cleanup and auto-archival sections now state /projects/{projectKey}/settings/lifecycle, and the link checks require that shape. Co-Authored-By: Claude Opus 5.5 (1M context) --- evals/release-autopilot-onboarding/assertions.js | 4 ++-- .../launchdarkly-release-autopilot-onboarding/SKILL.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index 77106c96..7ffe522d 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -220,7 +220,7 @@ module.exports.githubQuestionHasLink = (output) => { }; module.exports.cleanupQuestionHasLifecycleLink = (output) => { - const pass = askPrompts(output).some((p) => p.includes("/settings/lifecycle")); + const pass = askPrompts(output).some((p) => /\/projects\/[^/\s]+\/settings\/lifecycle/.test(p)); return result(pass, pass ? "auto-cleanup question contains the lifecycle settings link" : "no question contains /settings/lifecycle"); }; @@ -273,7 +273,7 @@ module.exports.saysAutoFlaggingOff = (output) => { }; module.exports.archivalQuestionHasLifecycleLink = (output) => { - const pass = askPrompts(output).some((p) => p.includes("/settings/lifecycle") && /archiv/i.test(p)); + const pass = askPrompts(output).some((p) => /\/projects\/[^/\s]+\/settings\/lifecycle/.test(p) && /archiv/i.test(p)); return result(pass, pass ? "auto-archival question contains the lifecycle settings link" : "no archival question contains /settings/lifecycle"); }; diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index a6cdde02..52f578b4 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -218,7 +218,7 @@ To create a plan, follow [references/release-plans.md](references/release-plans. Only if the user picked auto-cleanup. -Auto-cleanup is the per-project **Automated flag cleanup** setting, on the Lifecycle settings page (or the Vega settings page, for accounts where Vega has its own project settings). The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: +Auto-cleanup is the per-project **Automated flag cleanup** setting, on the Lifecycle settings page (or the Vega settings page, for accounts where Vega has its own project settings). The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, the link is exactly `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (or `/projects/{projectKey}/settings/vega` if that page has no "Automated flag cleanup" section). Follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: - label: "Done", description: "I turned on automated flag cleanup for every project" - label: "Later", description: "Skip this. I'll turn it on from Lifecycle settings later" @@ -230,7 +230,7 @@ Record the answer as reported by the user, and list "Done" under browser steps i Only if the user picked auto-archival. -Auto-archival is the per-project **Automatic archival** setting on the Lifecycle settings page. No MCP tool manages it, so this is a browser step. For each project from Step 4, follow [references/auto-archival.md](references/auto-archival.md), then ask header "Auto-archival", with the link in the question text: +Auto-archival is the per-project **Automatic archival** setting on the Lifecycle settings page. No MCP tool manages it, so this is a browser step. For each project from Step 4, the link is exactly `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance. Follow [references/auto-archival.md](references/auto-archival.md), then ask header "Auto-archival", with the link in the question text: - label: "Done", description: "I turned on automatic archival for every project" - label: "Later", description: "Skip this, or the section isn't there yet. I'll turn it on from Lifecycle settings later"