Repository navigation
feat(release-autopilot): add onboarding wizard skill and eval suite - #229
Merged
Merged
Conversation
Add launchdarkly-release-autopilot-onboarding: a step-by-step setup that asks which capabilities to enable, hands off the GitHub App link, maps chosen repos, guides managed or bring-your-own-key inference, and creates environment release plans for auto-releasing. One confirmed plan lists every write; plans are created before repos are mapped and account gates are flipped last. Route first-time setup to it from the settings skill. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Handle 403 plan gating on release plans, join mapped repos to the install list by githubRepoId, override unpicked capabilities off on mapped repos, defer metric creation to Step 8, clarify the batched confirmation versus the settings skill, and point deprecated MCP servers at the latest one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… accounts without release plans Found in a catamorphic dry run: a repo in an org without the App install gets no githubConnectUrl, and an account can run auto-releasing on release policies while release plans return 403. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…erence is the default Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Treat release plans as optional with a release policy fallback; keep auto-releasing off where a policy is missing unless the user accepts the fallback; hand remaps to the settings skill; override unpicked capabilities on already-mapped repos; limit metric creation to LaunchDarkly; present managed inference and BYOK as equal options. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…andle unmatched mapped repos Release policies gate auto-releasing until release plans replace them by GA, so a missing or incomplete policy keeps auto-releasing off for the repo. Found in a catamorphic dry run: a mapped repo can be missing from the install list, and the release-plans 403 is account-wide. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ns replace them by GA Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a settings page only Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ion check, fewer questions Add Preview mode (no writes; ends with an admin checklist) and Resume mode (re-read live state, continue at the first unfinished step). Check the member role up front and fall back to Preview for readers. Skip questions the account already answers. Detect the release mode once (policy today, plans by GA). Add voice rules, and move plan creation, edge cases, and examples to references/. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Let users change approval even when the account already requires it; leave account writes to admins and owners; keep the mode and rebuild picks on Resume; drop the writing hand-off in Preview; restore the conditional plan-ordering wording; shorten the uninstalled-org guidance. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Six cases lock in the onboarding write rules found in dry runs: confirm before writing, repo overrides for unpicked capabilities, account write last, one release-plans call in policy mode, no writes for readers or previews, and no account write for writers. The provider gains per-test mock_tool_overrides and label matching for scripted ask-question answers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lets engineers without an API key run suites on a Claude subscription via claude setup-token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…checklists Fix assertion paths (they resolve from shared/), replace the rubric grader with deterministic checks so the suite runs on a subscription token, and add a no-raw-names check after the first run showed tool and field names in the reader and writer checklists. The skill's voice rule now covers those checklists. All 6 cases pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…n voice; expand evals to 10 cases Iteration 3 found the agent applying changes after confirm answers that were not 'Looks good', hidden by positional scripted answers drifting onto the wrong questions. The skill now re-asks on any other answer, uses explicit answers from the request, and keeps internal mode and field names out of user-facing text. The harness gains opt-in label-matched answers; assertions check the approved option, the exact repo and project, valid release plan phases, and raw names across all turns. New cases: change at confirm, already-mapped repo override, no remap, and plan mode. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…stateful mocks, resume and install-path cases Scope the Release Autopilot tool definitions so other suites see exactly the tools they saw before. Record Release Autopilot writes in the mocks so verification reads reflect them. Add cases for resume (triggered from the user's words, with the Skill tool available), an uninstalled GitHub org, and a deprecated MCP server; 13 cases pass. In the skill: resume skips the intro and still stops at the confirmation, restated picks are never approval, list-metrics is plan-mode only, voice rules cover status codes and step numbers, and mapping sub-cases move to references. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…onventions Move Preview/Resume details and bring-your-own-key steps to references/, shorten the description, intro, and settings-skill note, merge workflow boilerplate, add the template's Edge Cases table, trim What NOT to Do to rules not stated elsewhere, and drop example flows that the evals cover. Bump the Claude Code plugin to 1.0.3 and the Cursor plugin to 1.0.4 for the new skill. All 13 eval cases still pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… previews, GA-ready evals Lead with 'Partly set up: waiting on an admin' when a change was left for an admin or failed, keep previews to the checklist, repeat the GitHub link in the pause summary, never announce mode names or guess at causes, and rename internal mode headings to product words. Evals: default release-plans mock is the GA shape (the policy case sets today's 403), a new case covers a 403 on the account write, failed writes are no longer recorded by the stateful mocks, and the explicit preview runs without a forced slash invocation. 14 cases pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…is pending; refresh settings description Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t-onboarding # Conflicts: # CHANGELOG.md
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
…ain language Rename 'Choose inference' to choosing whose AI model account runs the agents, explain the choice in one sentence before asking, and label the options 'LaunchDarkly's (default)' and 'Bring your own key'. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…naming Label matching for scripted ask-question answers now applies only in the opt-in match mode, so suites that script answers in order (onboarding, experiment-guide) behave exactly as on main. The skill names repos by owner/name and drops the 'release mode' heading the agent echoed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…sible; use capability names Offer auto-cleanup as an opt-in capability, set up as a browser step on the project's Lifecycle settings page (the MCP settings tools do not manage it yet, so autoCleanup is never sent). Put links in the question text, because the question dialog hides the message above it. After the welcome, name the capabilities instead of 'Release Autopilot', and never invent UI locations. Evals: 15 cases, adding auto-cleanup, a link-in- question check, and a capability-names check. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… different account The GitHub connect link acts on whatever LaunchDarkly session the browser has. When the App still reads as not installed after the user finished, name the member this connection uses and ask which account the browser is signed in to, instead of repeating the link. Rarer causes (instance, GitHub identity, org) move to references/github-troubleshooting.md. Every browser step now names the member to sign in as. Adds a 16th eval case. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…dd-on, not the alpha When every settings tool returns 404, both skills now tell the user 'Your account does not have the Release Autopilot add-on enabled.' Drops the remaining alpha wording, including the runbook note. Adds a 17th eval case. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fsaresh
marked this pull request as ready for review
October 28, 2026 19:10
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
'Why didn't my PR get classified?' always runs the settings skill's read-only diagnosis, even on an empty account, and offers onboarding afterward. Onboarding hands diagnosis requests to the settings skill instead of starting the wizard. Adds an 18th eval case. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…uto-cleanup lives
Offer auto-archival as an opt-in capability, set up as a browser step in
the project's Lifecycle settings ('Automatic archival', from
launchdarkly/gonfalon#75501). Auto-cleanup's section moves to the Vega
settings page on accounts where Vega has project settings; the reference
now says so. Adds a 19th eval case.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…apabilities Step 6 now runs release setup, auto-cleanup, and auto-archival each only when the user picked that capability, so nobody is walked through turning on code removal they did not ask for. Evals check that no unpicked cleanup or archival step appears. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolve conflicts in the eval provider and README by moving the release-autopilot-onboarding suite onto main's new suite-local mechanisms: tool_definitions and tool_mocks replace tool_scopes and the shared definitions.json/tool-responses.json entries (both files now match main), and main's mock_overrides replaces mock_tool_overrides. Keep mock_ask_question_match (order mode follows main, including [] as a dismissed question), stateful Release Autopilot mocks under overrides, and CLAUDE_CODE_OAUTH_TOKEN. The empty-account diagnosis case now grades diagnose-then-offer order instead of zero writes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The agent backend loads only the enabled flags from code automation settings and ignores approvalRequired, so every auto-flagging proposal waits for '@launchdarkly approve' regardless. Onboarding no longer asks about approval or sends approvalRequired; it explains the approve step instead. The settings skill notes that turning approval off has no effect today. Evals check approvalRequired is never sent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t-onboarding # Conflicts: # CHANGELOG.md
…se setup on auto-releasing GitHub step: flow=authorize no longer offers 'Need an org owner' (the App is installed); it asks the member to authorize with a GitHub account in the installing organization, with an 'I can't authorize' path. Step 6: environments and release plan/policy checks move under a Release setup section that runs only when auto-releasing is picked. Evals add an authorize-flow case and check cleanup- or archival-only setups never touch release setup. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b53ed31. Configure here.
'I can't authorize' now pauses like 'Need an org owner' (summary, link, resume phrase) instead of continuing, and the wizard never maps repos while GitHub still needs connecting, so an empty repo list is not mistaken for an organization without the App. Addresses Bugbot review on #229. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…path inline
An eval run showed the agent guessing /settings/projects/{key}/lifecycle
when the path lived only in references. The auto-cleanup and
auto-archival sections now state /projects/{projectKey}/settings/lifecycle,
and the link checks require that shape.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gcomstock
approved these changes
Oct 30, 2026
vroske-ld
reviewed
Oct 30, 2026
vroske-ld
left a comment
vroske-ld
left a comment
There was a problem hiding this comment.
i know this PR has since merged but i found some potential issues we could maybe take care of in a follow up?
| Only if the user picked auto-cleanup. | ||
|
|
||
|
|
||
| Auto-cleanup is the per-project **Automated flag cleanup** setting, on the Lifecycle settings page (or the Vega settings page, for accounts where Vega has its own project settings). The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, the link is exactly `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (or `/projects/{projectKey}/settings/vega` if that page has no "Automated flag cleanup" section). Follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: |
There was a problem hiding this comment.
blocking: this asks the user to turn on automated cleanup and save it before the Step 7 confirmation. BYOK and automatic archival follow the same ordering, and Preview still runs these steps, so a dry run can change production settings despite promising to make no changes. Could we defer these browser actions until after “Looks good,” and have Preview only list them?
|
|
||
| ### Step 3: Connect GitHub | ||
|
|
||
| Skip this step without mentioning GitHub if Step 1 returned `installed: true`, `userAuthorized: true`, and no `githubConnectUrl`. |
There was a problem hiding this comment.
blocking: Step 2 allows an auto-archival-only setup, but Steps 3 and 4 always require GitHub and create a repo mapping. Automatic archival is project-level and does not need GitHub, so this path either blocks the user on an unrelated App install or adds an irrelevant mapping. Could we gate the GitHub/repo steps on repo-based capabilities and select projects directly for archival-only setup?
|
|
||
| If a repo the user wants is not on the list, its owner decides the fix: a listed owner means the App needs access to that repo, and a new owner means an org owner must install the App there (no `githubConnectUrl` appears for this case). See [references/edge-cases.md](references/edge-cases.md). Wait for "Done" and list again. | ||
| 2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where auto-flagging creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. | ||
| 3. **Check existing mappings** with the join from Step 1 (an unmapped repo returns 404, which is expected). Already mapped to the chosen project with matching capabilities: write nothing. Mapped to the chosen project with an unpicked capability on: record an override that turns it off. Mapped to a different project: never remap here; send the user to the settings skill to unmap first. Details: [references/edge-cases.md](references/edge-cases.md#existing-mappings). |
There was a problem hiding this comment.
blocking: the existing-mapping cases do not handle a capability the user picked being effectively off because this repo already has an override. In that state the account gate can be enabled while the repo stays opted out, and verification may present the setup as complete. Could we add the inverse branch that enables or clears the override for selected capabilities, then verify the effective value?
| const mapsRepoWithOverrides = (repo, project, caps) => (output) => { | ||
| const maps = traj(output).filter((t) => t.tool === "update-release-autopilot-repo-settings"); | ||
| if (!maps.length) return result(false, "never wrote a repo mapping"); | ||
| const wrong = maps.filter((m) => (m.arguments || {}).repo !== repo || (project && (m.arguments || {}).projectKey && m.arguments.projectKey !== project)); |
There was a problem hiding this comment.
blocking: this condition lets an initial mapping pass when projectKey is omitted. That field is required on first map, and the mock accepts the incomplete write too, so several “mapped to default” cases can stay green without creating a valid mapping. Could we require exact projectKey equality whenever project is expected?
|
|
||
| module.exports.releasePlansCalledAtMostOnce = (output) => { | ||
| const n = traj(output).filter((t) => t.tool === "list-release-plans").length; | ||
| return result(n <= 1, `list-release-plans called ${n} time(s)`); |
There was a problem hiding this comment.
blocking: the policy-mode case is meant to prove the 403 detection path, but this assertion passes when list-release-plans is never called, and no assertion requires list-release-policies. The entire release-policy check can be skipped while the case stays green. Could we require exactly one plans lookup and the expected policy lookup?
| const plan = steps.findIndex((t) => t.tool === "create-release-plan"); | ||
| const repo = steps.findIndex((t) => t.tool === "update-release-autopilot-repo-settings"); | ||
| if (plan === -1) return result(false, "never created a release plan"); | ||
| return result(repo === -1 || plan < repo, `plan at ${plan}, repo write at ${repo}`); |
There was a problem hiding this comment.
blocking: this passes when there is no repo write at all. Since the plan-mode fixture starts with acme/demo unmapped and no other assertion requires that mapping, onboarding can omit a core requested change and still receive full credit. Could we require the repo write to exist, then check that the plan precedes it and that it targets the expected repo/project?
Contributor
Author
|
Thanks for the review, @vroske-ld! All six follow-ups are addressed in #237: browser changes now wait for "Looks good" (Preview only lists them), GitHub and repo mapping run only for the capabilities that need them, a picked capability kept off by a repo override is turned back on and verified, and the three eval checks now fail on the behaviors you described. Let's continue the convo over in #237? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
release-autopilot/launchdarkly-release-autopilot-onboarding, a step-by-step wizard that takes a customer from nothing to a working Release Autopilot setup. Todaylaunchdarkly-release-autopilot-settingsconfigures and diagnoses, but assumes the user already knows what to turn on.The wizard:
The settings skill (1.4.0) now routes first-time setup to the new skill. Both skills tell users "Your account does not have the Release Autopilot add-on enabled" when the settings endpoints return 404. Claude Code plugin 1.0.3, Cursor plugin 1.0.4.
Evals
New suite
evals/release-autopilot-onboarding(20 cases), all passing locally: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install and authorize paths, deprecated MCP, a failed write, auto-cleanup, a browser signed in to a different account, an account without the add-on, auto-archival, and a diagnosis on an empty account.The suite keeps its tool definitions and mocks in its own folder, loaded with
tool_definitionsandtool_mocks, so the shareddefinitions.jsonandtool-responses.jsonare unchanged. Shared harness changes, backward compatible for other suites:mock_ask_question_matchthat matches scripted answers to each question by option id or label (the default in-order mode is unchanged)CLAUDE_CODE_OAUTH_TOKENas an alternative toANTHROPIC_API_KEYBecause
evals/providerschanged, CI re-runs every suite on this PR.Testing
python3 scripts/validate_skills.pyandtests/test_*.pypasscd evals && npm test: 110 passnpm run eval:release-autopilot-onboarding, 20/20 passNot covered
autoCleanupfield exists behindenable-auto-cleanup-factory-settingsbut is hidden from the MCP tools🤖 Generated with Claude Code
via LD Research 🤖
Note
Overview
Adds
launchdarkly-release-autopilot-onboarding(0.2.0-experimental): a multi-step wizard for setup, preview, and resume that collects capabilities, GitHub App connection, repo→project mapping, inference choice, and release policy/plan setup, with no MCP writes until the user picks "Looks good".launchdarkly-release-autopilot-settings(1.4.0) now sends first-time setup to the onboarding skill and uses add-on enabled wording instead of alpha when settings APIs return 404. Claude/Cursor plugin manifests register the new skill and bump versions (1.0.3 / 1.0.4).Ships a new
evals/release-autopilot-onboardingsuite (20 cases) with suite-local tool defs/mocks and shared harness updates:mock_ask_question_match, stateful Release Autopilot mocks for verify-after-write, and optionalCLAUDE_CODE_OAUTH_TOKENauth.Reviewed by Cursor Bugbot for commit f0a08d6. Bugbot is set up for automated code reviews on this repo. Configure here.