Skip to content

fix(release-autopilot-onboarding): address #229 review follow-ups #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
fsaresh wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

fix(release-autopilot-onboarding): address #229 review follow-ups #237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
10f748d
fix(release-autopilot-onboarding): defer browser changes, gate GitHub…
fsaresh Oct 30, 2026
56b7323
fix(release-autopilot-onboarding): close gaps from a recheck of the #…
fsaresh Oct 30, 2026
c6e67a4
fix(release-autopilot-onboarding): address second review round on #237
fsaresh Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ All notable changes to this repository will be documented in this file.

## Unreleased

- `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.3.0-experimental: browser steps that change settings (bring your own key, auto-cleanup, auto-archival) now happen only after "Looks good", and Preview lists them instead of asking for them; GitHub and repo mapping run only for the capabilities that need them (auto-archival alone needs neither); a picked capability turned off by a repo override is turned back on and verified. Evals require the exact project on mappings, the policy lookup path, and the repo write in plan mode, and add archival-only, preview-with-cleanup, repo-override, bring-your-own-key, and preview-without-GitHub cases (25 total); release setup lookups are checked for the right project and environments in both release policy and release plan modes.
- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing, auto-cleanup, auto-archival), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4.
- `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill.
- evals: add the `release-autopilot-onboarding` suite (20 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install and authorize paths, deprecated MCP, a failed write, auto-cleanup, auto-archival, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The suite keeps its tool definitions and mocks in its own folder (`tool_definitions`, `tool_mocks`). The provider gains `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`.
Expand Down
99 changes: 97 additions & 2 deletions evals/release-autopilot-onboarding/assertions.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,20 @@ function isConfirmQuestion(step) {
return labels.some((l) => l.includes("looks good"));
}

// Index of the last confirm question before `limit` whose answer was "Looks good",
// or -1. Showing the confirm question is not approval; only that answer is.
function approvedConfirmBefore(steps, limit) {

@vroske-ld vroske-ld Oct 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approvedConfirmBefore walks backward until it finds any “Looks good.” It does not stop at the latest confirmation. An older approval followed by “Change something,” then an automatic-archival browser question, still returns pass: true. The newer answer revoked approval, so the eval can stay green while the agent continues a production-setting step.

Inspect the last confirmation before the browser step and require that selected option to be “Looks good.”

for (let i = limit - 1; i >= 0; i--) {
const step = steps[i];
if (!isConfirmQuestion(step)) continue;
const ok = ((step.arguments && step.arguments.options) || [])
.filter((o) => String(o.label || "").toLowerCase().includes("looks good"))
.map((o) => o.id);
if ((step.selected || []).some((id) => ok.includes(id))) return i;
}
return -1;
}

module.exports.noWrites = (output) => {
const w = writes(output);
return result(w.length === 0, w.length ? `wrote: ${w.map((t) => t.tool).join(", ")}` : "no writes");
Expand All @@ -45,7 +59,7 @@ module.exports.confirmsBeforeWriting = (output) => {
const mapsRepoWithOverrides = (repo, project, caps) => (output) => {
const maps = traj(output).filter((t) => t.tool === "update-release-autopilot-repo-settings");
if (!maps.length) return result(false, "never wrote a repo mapping");
const wrong = maps.filter((m) => (m.arguments || {}).repo !== repo || (project && (m.arguments || {}).projectKey && m.arguments.projectKey !== project));
const wrong = maps.filter((m) => (m.arguments || {}).repo !== repo || (project && (m.arguments || {}).projectKey !== project));
if (wrong.length) return result(false, `wrote ${wrong.map((m) => `${m.arguments.repo}@${m.arguments.projectKey}`).join(", ")}; want only ${repo}@${project}`);
const bad = maps.filter((m) => caps.some((c) => !(m.arguments[c] && m.arguments[c].enabled === false)));
return result(bad.length === 0, bad.length ? `missing ${caps.join("/")} enabled:false` : `${repo} written with ${caps.join("/")} off`);
Expand Down Expand Up @@ -88,7 +102,10 @@ module.exports.planBeforeRepoWrite = (output) => {
const plan = steps.findIndex((t) => t.tool === "create-release-plan");
const repo = steps.findIndex((t) => t.tool === "update-release-autopilot-repo-settings");
if (plan === -1) return result(false, "never created a release plan");
return result(repo === -1 || plan < repo, `plan at ${plan}, repo write at ${repo}`);
if (repo === -1) return result(false, "never mapped acme/demo");
const args = steps[repo].arguments || {};
const pass = plan < repo && args.repo === "acme/demo" && args.projectKey === "default";
return result(pass, `plan at ${plan}, repo write at ${repo} (${args.repo}@${args.projectKey})`);
};

module.exports.plansHaveClauses = (output) => {
Expand Down Expand Up @@ -352,3 +369,81 @@ module.exports.pausesBeforeMapping = (output) => {
const pass = repoQ.length === 0 && resume;
return result(pass, `repo questions: ${repoQ.length}; resume phrase in final message: ${resume}`);
};

// Release setup lookups for one project, valid for both modes. Every
// release-plans call names the project and one of its environments. If the
// first call is the account-wide 403, there is exactly one plans call and a
// release-policies lookup for the project (release policies, today). If it
// succeeds, no policy lookup is required (environment release plans, at GA).
const releaseSetupLookups = (projectKey, envs) => (output) => {
const steps = traj(output);
const plans = steps.filter((t) => t.tool === "list-release-plans");
if (!plans.length) return result(false, "never checked release plans");
const badPlan = plans.find((t) => (t.arguments || {}).projectKey !== projectKey || !envs.includes((t.arguments || {}).environmentKey));

@vroske-ld vroske-ld Oct 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The successful release-plan path accepts any lookup whose environment is test or production. It does not require the environment selected in the case. The plan-mode case selects Production, but a lookup for Test returns pass: true. A multi-environment run also passes after checking only one environment.

Give each assertion the selected environment set and require one successful plan lookup for every selected environment. Keep the one-call account-wide behavior only for the 403 policy path.

if (badPlan) return result(false, `release-plans lookup with ${JSON.stringify(badPlan.arguments)}`);
const forbidden = /403|not allow/i.test(String(plans[0].mock_response_preview || ""));
if (!forbidden) return result(true, `release plans available; ${plans.length} lookup(s) for ${projectKey}`);
const policies = steps.filter((t) => t.tool === "list-release-policies");
const okPolicies = policies.filter((t) => (t.arguments || {}).projectKey === projectKey);
const pass = plans.length === 1 && okPolicies.length >= 1 && okPolicies.length === policies.length;
return result(pass, `release-plans lookups: ${plans.length}; release-policies lookups for ${projectKey}: ${okPolicies.length} of ${policies.length}`);
};
module.exports.releaseSetupLookupsDefault = releaseSetupLookups("default", ["test", "production"]);

// A browser step question: a "Done" option and a settings topic (by link or by
// words), so a pre-confirm question without the right link is still caught.
const BROWSER_TOPIC = /settings\/(lifecycle|vega)|lifecycle settings|vega settings|own key|api key|flag cleanup|automatic archival|auto-archival|auto-cleanup/i;
const isBrowserStepQuestion = (t) => t.tool === "ask-question" &&
((t.arguments && t.arguments.options) || []).some((o) => /^done\b/i.test(String(o.label || ""))) &&
BROWSER_TOPIC.test(String((t.arguments && t.arguments.prompt) || "")) &&
!/github/i.test(String((t.arguments && t.arguments.prompt) || ""));

// Browser steps that change settings come only after "Looks good".
module.exports.browserStepsAfterConfirm = (output) => {

@vroske-ld vroske-ld Oct 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: a rejected confirmation still counts as approval.

this treats showing the confirmation question as approval, regardless of which option the user selected. In the archival-only case there is no MCP write, so no other assertion checks the answer; a trajectory that selects “Change something” and then asks the user to change Lifecycle settings still passes. Could we require the selected option to be “Looks good” before the first browser step?

const steps = traj(output);
const browser = steps.findIndex(isBrowserStepQuestion);
if (browser === -1) return result(false, "no browser step question");
const approved = approvedConfirmBefore(steps, browser);
return result(approved !== -1, approved !== -1 ? `approved at ${approved}, first browser step at ${browser}` : `browser step at ${browser} without a "Looks good" answer before it`);
};

// Preview never asks the user to change something in the browser, and lists it.
module.exports.previewListsBrowserSteps = (output) => {
const asked = traj(output).filter(isBrowserStepQuestion).length;
const listed = /settings\/lifecycle/.test(finalText(output));
return result(asked === 0 && listed, `browser step questions: ${asked}; lifecycle link listed: ${listed}`);
};

// Archival-only setup needs no GitHub step and no repo mapping.
module.exports.noGithubOrMapping = (output) => {
const steps = traj(output);
const repoWrites = steps.filter((t) => /^(update|delete)-release-autopilot-repo-settings$/.test(t.tool)).length;
const ghQ = steps.filter((t) => t.tool === "ask-question" && (
/github/i.test(String((t.arguments && t.arguments.prompt) || "")) ||
((t.arguments && t.arguments.options) || []).some((o) => /^[\w.-]+\/[\w.-]+$/.test(String(o.label || "")))
)).length;
return result(repoWrites === 0 && ghQ === 0, `repo writes: ${repoWrites}; GitHub or repo-picker questions: ${ghQ}`);
};

module.exports.storefrontAutoFlaggingOn = (output) => {
const steps = traj(output);
const w = steps.findIndex((t) => t.tool === "update-release-autopilot-repo-settings" && (t.arguments || {}).repo === "acme/storefront" &&
t.arguments.autoFlagging && t.arguments.autoFlagging.enabled === true);
if (w === -1) return result(false, "did not clear the repo's auto-flagging override");
const verified = steps.slice(w + 1).some((t) => t.tool === "get-release-autopilot-repo-settings" && (t.arguments || {}).repo === "acme/storefront");
return result(verified, verified ? "turned auto-flagging on for acme/storefront and read it back" : "turned it on but never verified the repo's effective value");
};

module.exports.byokQuestionHasVegaLink = (output) => {
const pass = askPrompts(output).some((p) => /\/projects\/[^/\s]+\/settings\/vega/.test(p));
return result(pass, pass ? "bring-your-own-key question contains the Vega settings link" : "no question contains /projects/{key}/settings/vega");
};

// Preview with GitHub disconnected and auto-releasing picked lists the
// deferred release setup, not only repo mapping.
module.exports.previewListsDeferredReleaseSetup = (output) => {
const text = finalText(output).toLowerCase();
const mapping = /map/.test(text) && /github/.test(text);
const release = /environment/.test(text) && /release (polic|plan)/.test(text);
return result(mapping && release, `lists mapping after GitHub: ${mapping}; lists environments and release policies/plans: ${release}`);
};
186 changes: 184 additions & 2 deletions evals/release-autopilot-onboarding/promptfooconfig.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,8 @@ tests:
metric: no_unpicked_archival
weight: 3
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:releasePlansCalledAtMostOnce
metric: plans_403_account_wide
value: file://../release-autopilot-onboarding/assertions.js:releaseSetupLookupsDefault
metric: release_setup_lookups
weight: 3
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:skippedGithubNotReported
Expand Down Expand Up @@ -435,6 +435,10 @@ tests:
- ["Leave rules"]
- ["Looks good"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:releaseSetupLookupsDefault
metric: release_setup_lookups
weight: 3
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion
metric: no_unpicked_cleanup
Expand Down Expand Up @@ -620,6 +624,10 @@ tests:
- ["Done"]
- ["Looks good"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:browserStepsAfterConfirm
metric: browser_after_confirm
weight: 3
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:noReleaseSetup
metric: no_release_setup
Expand Down Expand Up @@ -769,6 +777,10 @@ tests:
- ["Done"]
- ["Looks good"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:browserStepsAfterConfirm
metric: browser_after_confirm
weight: 3
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:noReleaseSetup
metric: no_release_setup
Expand Down Expand Up @@ -837,3 +849,173 @@ tests:
value: file://../release-autopilot-onboarding/assertions.js:noWrites
metric: no_writes
weight: 3

# ------------------------------------------------------------------
# Auto-archival only: project-level, no GitHub, no repo mapping. The
# Lifecycle settings step comes after "Looks good".
# ------------------------------------------------------------------
- providers: ["mcp"]
description: "Archival only: no GitHub or repo mapping, browser step after confirm"
vars:
user_request: "Set up Release Autopilot auto-archival for project default."
max_turns: 16
mock_overrides:
get-release-autopilot-settings:
autoFlagging: { enabled: false, approvalRequired: true }
autoReleasing: { enabled: false }
version: 1
list-release-autopilot-repo-settings: { repos: [], totalCount: 0 }
list-release-autopilot-github-repos:
installed: false
userAuthorized: false
repositories: []
totalCount: 0
githubConnectUrl: "https://app.launchdarkly.com/github-connect?flow=install"
mock_ask_question_match: true
mock_ask_question_answers:
- ["Auto-archival"]
- ["default"]
- ["Looks good"]
- ["Done"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:noGithubOrMapping
metric: no_github_or_mapping
weight: 4
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:browserStepsAfterConfirm
metric: browser_after_confirm
weight: 3
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:archivalQuestionHasLifecycleLink
metric: lifecycle_link_in_question
weight: 2

# ------------------------------------------------------------------
# Preview with auto-cleanup picked: no writes, no browser changes asked
# for, and the cleanup step is listed with its link.
# ------------------------------------------------------------------
- providers: ["mcp-auto"]
description: "Preview with auto-cleanup: lists the browser step, asks for no browser change"
vars:
user_request: "Do a dry run: preview what you'd change to set up Release Autopilot auto-flagging and auto-cleanup for acme/demo in project default. Don't change anything."
max_turns: 20
mock_ask_question_match: true
mock_ask_question_answers:
- ["Add more"]
- ["Auto-flagging", "Auto-cleanup"]
- ["LaunchDarkly's"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:noWrites
metric: no_writes
weight: 4
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:previewListsBrowserSteps
metric: preview_lists_browser_steps
weight: 4

# ------------------------------------------------------------------
# acme/storefront is mapped with auto-flagging turned off by a repo
# override, and the account has it on. Picking auto-flagging must clear
# that override, not leave the repo opted out.
# ------------------------------------------------------------------
- providers: ["mcp"]
description: "Picked capability off by a repo override: turns it on for the repo"
vars:
user_request: "Set up Release Autopilot auto-flagging for acme/storefront in project default. I don't want auto-releasing."
max_turns: 22
mock_overrides:
list-release-autopilot-repo-settings:
repos:
- githubRepoId: 101
projectKey: default
autoFlagging: { enabled: false, approvalRequired: true, enabledOverride: true }
autoReleasing: { enabled: false, enabledOverride: true }
totalCount: 1
mock_ask_question_match: true
mock_ask_question_answers:
- ["Add more"]
- ["Auto-flagging"]
- ["LaunchDarkly's"]
- ["Looks good"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:storefrontAutoFlaggingOn
metric: clears_repo_override
weight: 4
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting
metric: confirms_before_writing
weight: 3

# ------------------------------------------------------------------
# Bring your own key is a browser change: it comes after "Looks good",
# with the project's Vega settings link in the question.
# ------------------------------------------------------------------
- providers: ["mcp"]
description: "Bring your own key: Vega settings step comes after confirm"
vars:
user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default, using our own Anthropic key."
max_turns: 22
mock_overrides:
get-release-autopilot-settings:
autoFlagging: { enabled: false, approvalRequired: true }
autoReleasing: { enabled: false }
version: 1
list-release-autopilot-repo-settings: { repos: [], totalCount: 0 }
mock_ask_question_match: true
mock_ask_question_answers:
- ["Auto-flagging"]
- ["Bring your own key"]
- ["Looks good"]
- ["Done"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:browserStepsAfterConfirm
metric: browser_after_confirm
weight: 4
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:byokQuestionHasVegaLink
metric: vega_link_in_question
weight: 3
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff
metric: repo_override_unpicked
weight: 2

# ------------------------------------------------------------------
# Preview, GitHub disconnected, auto-releasing picked: the checklist
# lists GitHub, the later repo mapping, and the later release setup
# (environments and release policies or plans). No writes.
# ------------------------------------------------------------------
- providers: ["mcp-auto"]
description: "Preview, GitHub disconnected, auto-releasing: lists deferred mapping and release setup"
vars:
user_request: "Do a dry run: preview what you'd change to set up Release Autopilot auto-flagging and auto-releasing for project default. Don't change anything."
max_turns: 18
mock_overrides:
get-release-autopilot-settings:
autoFlagging: { enabled: false, approvalRequired: true }
autoReleasing: { enabled: false }
version: 1
list-release-autopilot-repo-settings: { repos: [], totalCount: 0 }
list-release-autopilot-github-repos:
installed: false
userAuthorized: false
repositories: []
totalCount: 0
githubConnectUrl: "https://app.launchdarkly.com/github-connect?flow=install"
mock_ask_question_match: true
mock_ask_question_answers:
- ["Auto-flagging", "Auto-releasing"]
- ["LaunchDarkly's"]
assert:
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:noWrites
metric: no_writes
weight: 4
- type: javascript
value: file://../release-autopilot-onboarding/assertions.js:previewListsDeferredReleaseSetup
metric: deferred_release_setup
weight: 4
Loading
Loading