diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index c699096f..18c3ca14 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -11,7 +11,7 @@ "name": "launchdarkly", "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics", "source": ".", - "version": "1.0.2", + "version": "1.0.3", "author": { "name": "LaunchDarkly", "email": "support@launchdarkly.com" diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index cf847c65..81d1293a 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "launchdarkly", - "version": "1.0.2", + "version": "1.0.3", "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics", "author": { "name": "LaunchDarkly", @@ -36,6 +36,7 @@ "./skills/agentcontrol/online-evals", "./skills/agentcontrol/projects", "./skills/agentcontrol/tools", - "./skills/release-autopilot/launchdarkly-release-autopilot-settings" + "./skills/release-autopilot/launchdarkly-release-autopilot-settings", + "./skills/release-autopilot/launchdarkly-release-autopilot-onboarding" ] } diff --git a/.cursor-plugin/plugin.json b/.cursor-plugin/plugin.json index 75a4df18..591d745a 100644 --- a/.cursor-plugin/plugin.json +++ b/.cursor-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "launchdarkly", - "version": "1.0.3", + "version": "1.0.4", "description": "LaunchDarkly agent skills and mcp server for feature flag management, AgentControl, and skill authoring", "author": { "name": "LaunchDarkly", diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f69d169..b0d5e39e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,9 @@ All notable changes to this repository will be documented in this file. ## Unreleased +- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing, auto-cleanup, auto-archival), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4. +- `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill. +- evals: add the `release-autopilot-onboarding` suite (20 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install and authorize paths, deprecated MCP, a failed write, auto-cleanup, auto-archival, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The suite keeps its tool definitions and mocks in its own folder (`tool_definitions`, `tool_mocks`). The provider gains `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`. - `onboarding/mcp-configure` 0.1.1: Claude Code loads MCP servers only at session start, so a server added mid-session needs the user to quit, resume with `claude --continue`, and authenticate from `/mcp`. The skill no longer says Claude Code needs no restart or that its OAuth prompt appears on the first tool call. - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0). - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths. diff --git a/README.md b/README.md index f8302c81..d38137c0 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor | Skill | Description | |-------|-------------| +| `release-autopilot/launchdarkly-release-autopilot-onboarding` | Step-by-step setup: pick capabilities, connect the GitHub App, map repos, choose inference, review release setup | | `release-autopilot/launchdarkly-release-autopilot-settings` | Turn on GitHub App auto-flagging / auto-releasing, map `owner/name` repos, or diagnose why a PR was not classified (confirm before every write) | ### AgentControl diff --git a/evals/.env.example b/evals/.env.example index 95e682f0..4352bdaa 100644 --- a/evals/.env.example +++ b/evals/.env.example @@ -6,6 +6,11 @@ # Request access via #ask-app-eng to get an API key. ANTHROPIC_API_KEY= +# ALTERNATIVE to ANTHROPIC_API_KEY for the agent: a Claude subscription token. +# Run `claude setup-token` in your own terminal and paste the token here. +# llm-rubric assertions still need RUBRIC_MODEL to reach a grader. +# CLAUDE_CODE_OAUTH_TOKEN= + # OPTIONAL: the system-under-test model. Stays on Claude because that's what # real users will run skills against. Defaults to claude-sonnet-4-6 if unset. # AGENT_MODEL=claude-sonnet-4-6 diff --git a/evals/README.md b/evals/README.md index a9a3fb2d..b0ac7b3e 100644 --- a/evals/README.md +++ b/evals/README.md @@ -35,6 +35,7 @@ npm run eval:configs-variations # agentcontrol/configs-variations npm run eval:flag-create # feature-flags/launchdarkly-flag-create npm run eval:onboarding # onboarding npm run eval:experiment-orchestration # aws-devops-agent/experiment-orchestration +npm run eval:release-autopilot-onboarding # release-autopilot/launchdarkly-release-autopilot-onboarding # Quick smoke check — first test case only (~15-20s, ~$0.05) npm run eval:configs-create:single @@ -44,6 +45,7 @@ npm run eval:configs-variations:single npm run eval:flag-create:single npm run eval:onboarding:single npm run eval:experiment-orchestration:single +npm run eval:release-autopilot-onboarding:single # Aggregate and CI operations npm run eval:all # Run every suite and rebuild ../eval-scores.json diff --git a/evals/package.json b/evals/package.json index 51f90ca4..9c1e1d26 100644 --- a/evals/package.json +++ b/evals/package.json @@ -25,6 +25,8 @@ "eval:flag-and-release-change:single": "promptfoo eval -c shared/defaults.yaml -c flag-and-release-change/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:onboarding": "promptfoo eval -c shared/defaults.yaml -c onboarding/promptfooconfig.yaml --env-file .env --no-cache -o onboarding/results.json", "eval:onboarding:single": "promptfoo eval -c shared/defaults.yaml -c onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", + "eval:release-autopilot-onboarding": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache -o release-autopilot-onboarding/results.json", + "eval:release-autopilot-onboarding:single": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:experiment-guide": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache -o experiment-guide/results.json", "eval:experiment-guide:single": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:experiment-orchestration": "promptfoo eval -c shared/defaults.yaml -c experiment-orchestration/promptfooconfig.yaml --env-file .env --no-cache -o experiment-orchestration/results.json", diff --git a/evals/providers/_mock.js b/evals/providers/_mock.js index f5ab11ee..136c6836 100644 --- a/evals/providers/_mock.js +++ b/evals/providers/_mock.js @@ -126,6 +126,9 @@ function createMockState() { return { configs: {}, // configKey -> config object flags: {}, // flagKey -> flag object + raSettings: null, // Release Autopilot account settings written this run + raRepos: {}, // owner/name -> Release Autopilot repo settings written this run + raPlans: {}, // environmentKey -> release plans created this run }; } @@ -134,9 +137,57 @@ function createMockState() { * `state` is the per-invocation map from `createMockState()`. * `templates` is the full parsed mocks/tool-responses.json. */ +function mergeDeep(base, patch) { + const out = { ...(base || {}) }; + for (const [k, v] of Object.entries(patch || {})) { + out[k] = v && typeof v === "object" && !Array.isArray(v) ? mergeDeep(out[k], v) : v; + } + return out; +} + +// Release Autopilot settings and release plans: writes are recorded so that +// later reads (the skill's verify step) reflect them instead of the template. +function renderReleaseAutopilot(template, input, toolName, state, replacements) { + const args = input || {}; + if (toolName === "update-release-autopilot-settings") { + state.raSettings = mergeDeep(state.raSettings, args); + return { ...state.raSettings }; + } + if (toolName === "get-release-autopilot-settings" && state.raSettings) { + return mergeDeep(walk(template, replacements), state.raSettings); + } + if (toolName === "update-release-autopilot-repo-settings" && args.repo) { + const { repo, githubRepoId, ...rest } = args; + state.raRepos[repo] = mergeDeep(state.raRepos[repo] || { repo }, rest); + return { ...state.raRepos[repo] }; + } + if (toolName === "get-release-autopilot-repo-settings" && args.repo && state.raRepos[args.repo]) { + return { ...state.raRepos[args.repo] }; + } + if (toolName === "create-release-plan" && args.environmentKey) { + const plan = { key: args.key, name: args.name, environmentKey: args.environmentKey, progressiveRelease: args.progressiveRelease }; + (state.raPlans[args.environmentKey] = state.raPlans[args.environmentKey] || []).push(plan); + return plan; + } + if (toolName === "list-release-plans" && state.raPlans[args.environmentKey]) { + const items = state.raPlans[args.environmentKey]; + return { items, totalCount: items.length }; + } + return undefined; +} + function renderMockResponse(template, input, toolName, state) { const replacements = buildReplacements(input); + // An error template on a write (for example a 403 override) wins over + // stateful handling, so the failed write is not recorded. Reads still use + // state: the unmapped-repo read template is itself a 404 error. + const failedWrite = /^(update|create|delete)-/.test(toolName) && template && template.error; + if (state && state.raRepos && !failedWrite) { + const ra = renderReleaseAutopilot(template, input, toolName, state, replacements); + if (ra !== undefined) return ra; + } + // ---------- stateful reads ---------- if (toolName === "get-ai-config") { const key = input.configKey || input.key; diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index d787f28f..2c8a1b25 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -67,6 +67,15 @@ * successive `ask-question` calls; an empty * array is a dismissed question (no selection). * Calls past the end get the first option. + * mock_ask_question_match - when true, each `ask-question` call takes the first + * unused scripted answer whose entries all match its + * options (by id or label), instead of the next one in + * order. Entries match an option by id, or else by + * label (case-insensitive substring), so tests can + * steer by label when the agent picks the ids. Use + * for flows that may skip questions. A call with no match gets + * its first option and is recorded with + * `scripted: false` * mock_files - optional object mapping relative file paths to * string content, written into the isolated cwd * before the agent runs @@ -80,6 +89,8 @@ * Environment variables: * AGENT_MODEL - SUT model (default claude-sonnet-4-6). * ANTHROPIC_API_KEY - Auth for the SDK's child Claude Code process. + * CLAUDE_CODE_OAUTH_TOKEN - Alternative auth from `claude setup-token`, for + * running evals on a Claude subscription without an API key. * SKILL_EVAL_DEBUG=1 - Dump every SDK message to a temp file for inspection. */ @@ -107,6 +118,45 @@ const toolDefs = JSON.parse( fs.readFileSync(path.join(__dirname, "../tools/definitions.json"), "utf-8"), ); +/** + * Resolve a scripted answer to an option id. Exact id wins; otherwise the + * first option whose label contains the answer (case-insensitive). Unmatched + * answers pass through unchanged. + */ +function matchOption(want, options) { + const list = Array.isArray(options) ? options : []; + if (list.some((o) => o.id === want)) return want; + const needle = String(want).toLowerCase(); + const hit = list.find((o) => String(o.label || "").toLowerCase().includes(needle)); + return hit ? hit.id : want; +} + +/** + * Choose the scripted answer for the `idx`-th ask-question call. In order + * mode, take answers[idx] as written. In match mode, take the first unused + * answer whose entries all resolve to the call's options (by id or label), + * mark it used, and return the matched ids. Returns undefined when no + * scripted answer applies. + */ +function pickAnswer(answers, idx, options, matchMode, used) { + const resolves = (ans) => + Array.isArray(ans) && + ans.length > 0 && + ans.every((want) => (options || []).some((o) => o.id === matchOption(want, options))); + let chosen; + if (matchMode) { + const hit = answers.findIndex((ans, i) => !used.has(i) && resolves(ans)); + if (hit === -1) return undefined; + used.add(hit); + chosen = answers[hit]; + return chosen.map((want) => matchOption(want, options)); + } + // Order mode is unchanged: the scripted answer is returned as written, + // and an explicit [] is a dismissed question (no selection). + chosen = answers[idx]; + return Array.isArray(chosen) ? chosen : undefined; +} + const mockTemplates = JSON.parse( fs.readFileSync(path.join(__dirname, "../mocks/tool-responses.json"), "utf-8"), ); @@ -286,9 +336,14 @@ class ClaudeSkillAgentSdk { } async callApi(_prompt, context) { - const apiKey = process.env.ANTHROPIC_API_KEY; - if (!apiKey) { - return { error: "ANTHROPIC_API_KEY environment variable is not set" }; + // Either an API key or a Claude subscription token (`claude setup-token`) + // authenticates the SDK's child Claude Code process. The isolated config + // dir hides the interactive /login, so one of these must be in the env. + if (!process.env.ANTHROPIC_API_KEY && !process.env.CLAUDE_CODE_OAUTH_TOKEN) { + return { + error: + "Set ANTHROPIC_API_KEY, or CLAUDE_CODE_OAUTH_TOKEN from `claude setup-token`", + }; } const userRequest = @@ -313,6 +368,8 @@ class ClaudeSkillAgentSdk { : []; const mockFiles = context?.vars?.mock_files || {}; const mockOverrides = context?.vars?.mock_overrides || {}; + const matchAnswers = context?.vars?.mock_ask_question_match === true; + const usedAnswers = new Set(); let userMessage = userRequest; if (codebaseContext) { @@ -349,7 +406,11 @@ class ClaudeSkillAgentSdk { const template = this.mockTemplates[def.name]; let mock; if (override !== undefined) { - mock = renderOverride(override, args); + // Release Autopilot tools keep their recorded state under an + // override (an error override on a write is not recorded). + mock = /release-autopilot|release-plan/.test(def.name) + ? renderMockResponse(override, args, def.name, mockState) + : renderOverride(override, args); } else if (template) { mock = renderMockResponse(template, args, def.name, mockState); } else { @@ -399,11 +460,11 @@ class ClaudeSkillAgentSdk { }, async (args) => { const idx = askQuestionCallCount++; - const override = askQuestionAnswers[idx]; + const picked = pickAnswer(askQuestionAnswers, idx, args.options, matchAnswers, usedAnswers); + const scripted = picked !== undefined; let selected; - // An explicit [] is a dismissed question: no selection. - if (Array.isArray(override)) { - selected = override; + if (scripted) { + selected = picked; } else if (Array.isArray(args.options) && args.options.length > 0) { selected = [args.options[0].id]; } else { @@ -415,6 +476,8 @@ class ClaudeSkillAgentSdk { tool: "ask-question", arguments: args, turn: currentTurn, + scripted, + selected, mock_response_preview: previewMock(response), }); @@ -684,3 +747,5 @@ module.exports.filterToolDefs = filterToolDefs; module.exports.mergeToolDefs = mergeToolDefs; module.exports.skillInstructions = skillInstructions; module.exports.overrideFor = overrideFor; +module.exports.matchOption = matchOption; +module.exports.pickAnswer = pickAnswer; diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js new file mode 100644 index 00000000..7ffe522d --- /dev/null +++ b/evals/release-autopilot-onboarding/assertions.js @@ -0,0 +1,354 @@ +// Assertions for the Release Autopilot onboarding suite. Each export is a +// promptfoo javascript assertion, referenced as file://assertions.js:. + +const WRITE_TOOLS = new Set([ + "update-release-autopilot-settings", + "update-release-autopilot-repo-settings", + "delete-release-autopilot-repo-settings", + "create-release-plan", + "create-metric", +]); + +const traj = (output) => (output && output.trajectory) || []; +const writes = (output) => traj(output).filter((t) => WRITE_TOOLS.has(t.tool)); +const result = (pass, reason) => ({ pass, score: pass ? 1 : 0, reason }); + +function isConfirmQuestion(step) { + if (step.tool !== "ask-question") return false; + const labels = ((step.arguments && step.arguments.options) || []).map((o) => + String(o.label || "").toLowerCase(), + ); + return labels.some((l) => l.includes("looks good")); +} + +module.exports.noWrites = (output) => { + const w = writes(output); + return result(w.length === 0, w.length ? `wrote: ${w.map((t) => t.tool).join(", ")}` : "no writes"); +}; + +module.exports.confirmsBeforeWriting = (output) => { + const steps = traj(output); + const firstWrite = steps.findIndex((t) => WRITE_TOOLS.has(t.tool)); + if (firstWrite === -1) return result(false, "never wrote, so the confirmed plan was not applied"); + let confirm = -1; + for (let i = 0; i < firstWrite; i++) if (isConfirmQuestion(steps[i])) confirm = i; + if (confirm === -1) return result(false, "wrote without a confirm question"); + const step = steps[confirm]; + const ok = ((step.arguments && step.arguments.options) || []) + .filter((o) => String(o.label || "").toLowerCase().includes("looks good")) + .map((o) => o.id); + const approved = (step.selected || []).some((id) => ok.includes(id)); + return result(approved, approved ? `approved at ${confirm}, first write at ${firstWrite}` : `wrote after confirm answer ${JSON.stringify(step.selected)}`); +}; + +// Every repo write targets `repo` in `project` and sends enabled:false for each cap. +const mapsRepoWithOverrides = (repo, project, caps) => (output) => { + const maps = traj(output).filter((t) => t.tool === "update-release-autopilot-repo-settings"); + if (!maps.length) return result(false, "never wrote a repo mapping"); + const wrong = maps.filter((m) => (m.arguments || {}).repo !== repo || (project && (m.arguments || {}).projectKey && m.arguments.projectKey !== project)); + if (wrong.length) return result(false, `wrote ${wrong.map((m) => `${m.arguments.repo}@${m.arguments.projectKey}`).join(", ")}; want only ${repo}@${project}`); + const bad = maps.filter((m) => caps.some((c) => !(m.arguments[c] && m.arguments[c].enabled === false))); + return result(bad.length === 0, bad.length ? `missing ${caps.join("/")} enabled:false` : `${repo} written with ${caps.join("/")} off`); +}; + +module.exports.accountWriteLast = (output) => { + const steps = traj(output); + const acct = steps.findIndex((t) => t.tool === "update-release-autopilot-settings"); + if (acct === -1) return result(false, "no account write"); + const laterWrite = steps.slice(acct + 1).some((t) => WRITE_TOOLS.has(t.tool)); + return result(!laterWrite, laterWrite ? "a write came after the account write" : "account write was last"); +}; + +module.exports.neverAutoReleasingOn = (output) => { + const bad = writes(output).filter((t) => t.arguments && t.arguments.autoReleasing && t.arguments.autoReleasing.enabled === true); + return result(bad.length === 0, bad.length ? `turned auto-releasing on via ${bad[0].tool}` : "auto-releasing never turned on"); +}; + +module.exports.noAutoCleanup = (output) => { + const bad = writes(output).filter((t) => t.arguments && Object.prototype.hasOwnProperty.call(t.arguments, "autoCleanup")); + return result(bad.length === 0, bad.length ? "sent autoCleanup" : "no autoCleanup"); +}; + +const noneOf = (tools) => (output) => { + const hit = traj(output).filter((t) => tools.includes(t.tool)); + return result(hit.length === 0, hit.length ? `called ${hit.map((t) => t.tool).join(", ")}` : `never called ${tools.join(", ")}`); +}; + +module.exports.releasePlansCalledAtMostOnce = (output) => { + const n = traj(output).filter((t) => t.tool === "list-release-plans").length; + return result(n <= 1, `list-release-plans called ${n} time(s)`); +}; + +module.exports.repoTurnsAutoReleasingOff = mapsRepoWithOverrides("acme/demo", "default", ["autoReleasing"]); +module.exports.storefrontAutoReleasingOff = mapsRepoWithOverrides("acme/storefront", null, ["autoReleasing"]); +module.exports.noRepoWrite = noneOf(["update-release-autopilot-repo-settings", "delete-release-autopilot-repo-settings"]); + +module.exports.planBeforeRepoWrite = (output) => { + const steps = traj(output); + const plan = steps.findIndex((t) => t.tool === "create-release-plan"); + const repo = steps.findIndex((t) => t.tool === "update-release-autopilot-repo-settings"); + if (plan === -1) return result(false, "never created a release plan"); + return result(repo === -1 || plan < repo, `plan at ${plan}, repo write at ${repo}`); +}; + +module.exports.plansHaveClauses = (output) => { + const plans = traj(output).filter((t) => t.tool === "create-release-plan"); + if (!plans.length) return result(false, "no release plan created"); + const okClause = (c) => c && typeof c.attribute === "string" && typeof c.op === "string" && Array.isArray(c.values); + const okStep = (s) => s && typeof s.allocation === "number" && typeof s.durationMillis === "number"; + for (const p of plans) { + const phases = (((p.arguments || {}).progressiveRelease || {}).phases) || []; + if (!phases.length) return result(false, "plan has no phases"); + for (const ph of phases) { + if (!Array.isArray(ph.clauses) || !ph.clauses.length || !ph.clauses.every(okClause)) + return result(false, `bad clauses: ${JSON.stringify(ph.clauses).slice(0, 120)}`); + if (!Array.isArray(ph.steps) || !ph.steps.length || !ph.steps.every(okStep)) + return result(false, `bad steps: ${JSON.stringify(ph.steps).slice(0, 120)}`); + } + } + return result(true, "every phase has valid clauses and steps"); +}; +module.exports.noReleasePlanCreated = noneOf(["create-release-plan"]); +module.exports.noAccountWrite = noneOf(["update-release-autopilot-settings"]); + +const finalText = (output) => String((output && output.response) || ""); + +// Raw tool and field names the skill's voice rules ban from user-facing text. +const RAW_NAMES = [ + "update-release-autopilot", "get-release-autopilot", "list-release-autopilot", + "githubRepoId", "enabledOverride", "autoReleasing", "autoFlagging", "approvalRequired", + "create-release-plan", "list-release-plans", "list-release-policies", + "preview mode", "policy mode", "plan mode", +]; + +module.exports.noRawNames = (output) => { + const turns = ((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? x : JSON.stringify(x))); + const asks = traj(output) + .filter((t) => t.tool === "ask-question") + .map((t) => [t.arguments.prompt, ...((t.arguments.options || []).map((o) => o.label))].join(" ")); + const text = [finalText(output), ...turns, ...asks].join("\n"); + const hits = RAW_NAMES.filter((n) => text.includes(n)); + if (/\b(401|403|404)\b/.test(text)) hits.push("HTTP status code"); + return result(hits.length === 0, hits.length ? `user-facing text shows: ${hits.join(", ")}` : "no raw names"); +}; + +module.exports.previewNamesTheMapping = (output) => { + const text = finalText(output).toLowerCase(); + const pass = text.includes("acme/demo") && text.includes("default") && /auto-releasing/.test(text); + return result(pass, pass ? "names the repo, project, and auto-releasing" : "final message does not describe the mapping"); +}; + +const allText = (output) => + [finalText(output), ...(((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? x : JSON.stringify(x))))].join("\n"); + +module.exports.saysOrgOwnerInstall = (output) => { + const text = allText(output).toLowerCase(); + const pass = text.includes("other-org") && /owner/.test(text) && /install/.test(text); + return result(pass, pass ? "tells the user an other-org owner must install the App" : "does not explain the org install"); +}; + +module.exports.noIntro = (output) => { + const text = allText(output); + const pass = !text.includes("Release Autopilot Setup") && !/in a few short steps/i.test(text); + return result(pass, pass ? "no intro" : "showed the Setup intro on resume"); +}; + +module.exports.noPickQuestions = (output) => { + const asked = traj(output) + .filter((t) => t.tool === "ask-question") + .map((t) => ((t.arguments && t.arguments.options) || []).map((o) => String(o.label || "").toLowerCase()).join("|")); + const repeats = asked.filter((l) => l.includes("auto-releasing") || l.includes("require approval") || l.includes("bring your own key")); + return result(repeats.length === 0, repeats.length ? `re-asked: ${repeats.join(" ; ")}` : "no picks re-asked"); +}; + +module.exports.notCalledComplete = (output) => { + const text = finalText(output); + let hit = /setup complete|setup is complete|you're all set|is (now )?live/i.exec(text); + // "Open a pull request" is fine only when tied to the pending admin change, + // e.g. under a "Once an admin turns on auto-flagging:" lead-in. + let unconditional; + for (const m of text.matchAll(/open a pull request/gi)) { + const before = text.slice(Math.max(0, m.index - 250), m.index); + if (!/\b(once|after|when)\b/i.test(before)) { unconditional = text.slice(m.index, m.index + 80); break; } + } + if (!hit && unconditional) hit = [unconditional.trim().slice(0, 80)]; + return result(!hit, hit ? `final message says "${hit[0]}"` : "does not claim completion"); +}; + +// The preview section (from the "Changes an admin would make" heading on) is a +// short checklist; any state summary before it belongs to Step 1. +module.exports.previewIsShortChecklist = (output) => { + const text = finalText(output); + const at = text.search(/changes an admin would make/i); + if (at === -1) return result(false, "no 'Changes an admin would make' section"); + const section = text.slice(at); + const words = section.split(/\s+/).filter(Boolean).length; + const items = section.split("\n").filter((l) => /^\s*- \[ \]/.test(l)); + const pass = words <= 150 && items.length > 0; + return result(pass, `preview section: ${words} words, ${items.length} checklist line(s)`); +}; + +module.exports.skippedGithubNotReported = (output) => { + const hit = /github[^\n]{0,100}reported/i.exec(allText(output)); + return result(!hit, hit ? `labels skipped GitHub step: "${hit[0].slice(0, 80)}"` : "skipped GitHub step not labeled reported"); +}; + +module.exports.stopsAfterFailedWrite = (output) => { + const steps = traj(output); + const acct = steps.findIndex((t) => t.tool === "update-release-autopilot-settings"); + const repo = steps.findIndex((t) => t.tool === "update-release-autopilot-repo-settings"); + if (acct === -1) return result(false, "never attempted the account write"); + const after = steps.slice(acct + 1).filter((t) => WRITE_TOOLS.has(t.tool)); + const pass = repo !== -1 && repo < acct && after.length === 0; + return result(pass, `repo write at ${repo}, account write at ${acct}, ${after.length} write(s) after`); +}; + +// The link must be in the final pause summary, which is what the user forwards +// to their GitHub org owner. +module.exports.pauseSummaryHasConnectLink = (output) => { + const pass = finalText(output).includes("flow=install"); + return result(pass, pass ? "pause summary repeats the connect link" : "pause summary lacks the githubConnectUrl"); +}; + +const askPrompts = (output) => + traj(output).filter((t) => t.tool === "ask-question").map((t) => String((t.arguments && t.arguments.prompt) || "")); + +// The question dialog hides earlier text, so links belong in the question. +module.exports.githubQuestionHasLink = (output) => { + const pass = askPrompts(output).some((p) => p.includes("flow=install")); + return result(pass, pass ? "GitHub question contains the connect link" : "no question contains the githubConnectUrl"); +}; + +module.exports.cleanupQuestionHasLifecycleLink = (output) => { + const pass = askPrompts(output).some((p) => /\/projects\/[^/\s]+\/settings\/lifecycle/.test(p)); + return result(pass, pass ? "auto-cleanup question contains the lifecycle settings link" : "no question contains /settings/lifecycle"); +}; + +// "Release Autopilot" belongs only in the welcome, the resume phrase, and the +// settings skill's name, and the add-on name. +module.exports.noProductNameAfterIntro = (output) => { + const turns = ((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? x : (x && x.text) || "")); + const texts = [...turns, finalText(output), ...askPrompts(output)] + .filter((s) => !/Release Autopilot Setup|in a few short steps/i.test(s)) + .map((s) => s.replace(/continue Release Autopilot setup|Release Autopilot settings skill|release-autopilot-settings|Release Autopilot add-on/gi, "")); + const hits = texts.filter((s) => /Release Autopilot/.test(s)); + return result(hits.length === 0, hits.length ? `said "Release Autopilot" after the welcome: "${hits[0].slice(Math.max(0, hits[0].indexOf("Release Autopilot") - 40), hits[0].indexOf("Release Autopilot") + 40)}"` : "uses capability names"); +}; + +// The remap case must leave acme/storefront alone; other repo writes come +// from unscripted questions the harness answers with their first option. +module.exports.noStorefrontRemap = (output) => { + const hit = traj(output).filter((t) => t.tool.endsWith("-repo-settings") && /^(update|delete)/.test(t.tool) && (t.arguments || {}).repo === "acme/storefront"); + return result(hit.length === 0, hit.length ? `wrote acme/storefront: ${JSON.stringify(hit[0].arguments)}` : "acme/storefront left alone"); +}; + +module.exports.relistsAfterDone = (output) => { + const n = traj(output).filter((t) => t.tool === "list-release-autopilot-github-repos").length; + return result(n >= 2, `listed GitHub repos ${n} time(s)`); +}; + +// After "Done" with no install, the skill names this connection's member and +// asks which account the browser is signed in to. +module.exports.asksWhichBrowserAccount = (output) => { + const pass = askPrompts(output).some((p) => p.includes("admin@acme.example") && !p.includes("flow=install")); + return result(pass, pass ? "asked which account the browser uses, naming this connection's member" : "did not ask about the browser account with the member's email"); +}; + +module.exports.saysAddOnNotEnabled = (output) => { + const text = allText(output); + const pass = /Release Autopilot add-on enabled/i.test(text) && !/\balpha\b/i.test(text); + return result(pass, pass ? "says the add-on is not enabled" : "did not use the add-on wording (or mentioned the alpha)"); +}; + +module.exports.noCapabilityQuestion = (output) => { + const asked = traj(output).filter((t) => t.tool === "ask-question" && + ((t.arguments && t.arguments.options) || []).some((o) => /auto-releasing|auto-cleanup/i.test(String(o.label || "")))); + return result(asked.length === 0, asked.length ? "started the setup wizard (asked for capabilities)" : "did not start the wizard"); +}; + +module.exports.saysAutoFlaggingOff = (output) => { + const text = allText(output).toLowerCase(); + const pass = /auto-flagging (is )?(turned )?off|auto-flagging is not (turned )?on|auto-flagging isn't (turned )?on/.test(text); + return result(pass, pass ? "says auto-flagging is off for the account" : "did not say auto-flagging is off"); +}; + +module.exports.archivalQuestionHasLifecycleLink = (output) => { + const pass = askPrompts(output).some((p) => /\/projects\/[^/\s]+\/settings\/lifecycle/.test(p) && /archiv/i.test(p)); + return result(pass, pass ? "auto-archival question contains the lifecycle settings link" : "no archival question contains /settings/lifecycle"); +}; + +// Step 6 must not walk the user through cleanup or archival they did not pick. +const asksAbout = (output, re) => askPrompts(output).filter((p) => re.test(p)); +const CLEANUP = /auto-cleanup|automated flag cleanup|flag cleanup/i; +const ARCHIVAL = /auto-archival|automatic archival/i; +module.exports.noCleanupQuestion = (output) => { + const hit = asksAbout(output, CLEANUP).filter((p) => !/capabilit/i.test(p)); + return result(hit.length === 0, hit.length ? `asked about unpicked auto-cleanup: "${hit[0].slice(0, 90)}"` : "no auto-cleanup step"); +}; +module.exports.noArchivalQuestion = (output) => { + const hit = asksAbout(output, ARCHIVAL).filter((p) => !/capabilit/i.test(p)); + return result(hit.length === 0, hit.length ? `asked about unpicked auto-archival: "${hit[0].slice(0, 90)}"` : "no auto-archival step"); +}; + +// A diagnosis request is answered before any setup: the agent says auto-flagging +// is off before its first question, and that question is an offer, not the +// capability picker. (An unscripted offer gets the harness's first option, so +// the flow may continue into setup; writes are then graded by confirmsBeforeWriting.) +module.exports.diagnosesBeforeOffering = (output) => { + const steps = traj(output); + const firstAsk = steps.findIndex((t) => t.tool === "ask-question"); + const turns = ((output && output.assistant_turns) || []).map((x) => (typeof x === "string" ? { text: x } : x || {})); + const firstAskTurn = firstAsk === -1 ? Infinity : steps[firstAsk].turn; + const before = turns.filter((x) => (x.turn === undefined ? true : x.turn <= firstAskTurn)).map((x) => x.text || "").join("\n").toLowerCase() + + (firstAsk === -1 ? finalText(output).toLowerCase() : String(steps[firstAsk].arguments.prompt || "").toLowerCase()); + const saidOff = /auto-flagging (is )?(turned )?off|auto-flagging is not (turned )?on|auto-flagging isn't (turned )?on|nothing is set up|isn't set up|not set up/.test(before); + const firstIsPicker = firstAsk !== -1 && ((steps[firstAsk].arguments.options || []).some((o) => /auto-releasing|auto-cleanup|auto-archival/i.test(String(o.label || "")))); + const pass = saidOff && !firstIsPicker; + return result(pass, `said why before asking: ${saidOff}; first question is the capability picker: ${firstIsPicker}`); +}; + +module.exports.noUnconfirmedWrites = (output) => { + if (writes(output).length === 0) return result(true, "no writes"); + return module.exports.confirmsBeforeWriting(output); +}; + +// The agent backend ignores approvalRequired; auto-flagging always waits for +// "@launchdarkly approve". The skill never sends the field. +module.exports.neverSendsApproval = (output) => { + const bad = writes(output).filter((t) => { + const a = t.arguments || {}; + return (a.autoFlagging && "approvalRequired" in a.autoFlagging) || (a.autoReleasing && "approvalRequired" in a.autoReleasing); + }); + return result(bad.length === 0, bad.length ? `sent approvalRequired via ${bad[0].tool}` : "never sent approvalRequired"); +}; + +// Release setup runs only for auto-releasing: no environment question and no +// release plan or policy lookup otherwise. +module.exports.noReleaseSetup = (output) => { + const steps = traj(output); + const lookups = steps.filter((t) => t.tool === "list-release-plans" || t.tool === "list-release-policies"); + const envQ = steps.filter((t) => t.tool === "ask-question" && /environment/i.test(String((t.arguments && t.arguments.prompt) || ""))); + const pass = lookups.length === 0 && envQ.length === 0; + return result(pass, `release lookups: ${lookups.length}, environment questions: ${envQ.length}`); +}; + +// On flow=authorize the App is installed; the question is about authorizing, +// with no org-owner pause. +module.exports.authorizeQuestionNoOrgOwner = (output) => { + const q = traj(output).filter((t) => t.tool === "ask-question" && String(t.arguments.prompt || "").includes("flow=authorize")); + if (!q.length) return result(false, "no question contains the authorize link"); + const labels = (q[0].arguments.options || []).map((o) => String(o.label || "")); + const pass = /authoriz/i.test(q[0].arguments.prompt) && !labels.some((l) => /org owner/i.test(l)); + return result(pass, `prompt mentions authorize: ${/authoriz/i.test(q[0].arguments.prompt)}; options: ${labels.join(" | ")}`); +}; + +// While GitHub is disconnected the wizard pauses: no repo question, and the +// final message gives the resume phrase. +module.exports.pausesBeforeMapping = (output) => { + const repoQ = traj(output).filter((t) => t.tool === "ask-question" && + ((t.arguments && t.arguments.options) || []).some((o) => /\w+\/\w+/.test(String(o.label || ""))) && + /repo/i.test(String((t.arguments && t.arguments.prompt) || ""))); + const resume = /continue Release Autopilot setup/i.test(finalText(output)); + const pass = repoQ.length === 0 && resume; + return result(pass, `repo questions: ${repoQ.length}; resume phrase in final message: ${resume}`); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml new file mode 100644 index 00000000..e608f72a --- /dev/null +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -0,0 +1,839 @@ +# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json +# +# Run with shared defaults: +# promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml +# +# The onboarding wizard for Release Autopilot. These cases lock in the write +# rules found in dry runs against catamorphic: nothing is written before the +# user approves, Preview and reader runs never write, a repo never inherits a +# capability the user did not pick, the release-plans 403 is treated as +# account-wide policy mode, and account settings stay untouched when they +# already match. +# +# Tool definitions and default mocks live in this folder (tool_definitions, +# tool_mocks). Default mocks match a catamorphic-like account with both capabilities on, +# the GitHub App installed, and release plans available but empty (the GA +# shape). The policy-mode case overrides release plans to the 403 accounts +# return today. Mocks record Release Autopilot writes, so verification reads +# reflect them. Tests change state with mock_overrides, and scripted +# answers are matched to each question by option label +# (mock_ask_question_match), so a skipped question cannot shift them. +description: "Evaluation of the Release Autopilot onboarding skill" + +prompts: + - file://../../skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md + +providers: + - id: file://../providers/claude-skill-agent-sdk.js + label: mcp + config: + skill_slug: launchdarkly-release-autopilot-onboarding + tool_definitions: release-autopilot-onboarding/tool-definitions.json + tool_mocks: release-autopilot-onboarding/tool-mocks.json + allow_builtins: false + expose_ask_question: true + force_skill_invocation: true + # Same provider without a forced /slash invocation, for cases where the + # skill must trigger from the user's words (a forced bare invocation would + # correctly show the Setup intro before the user's message arrives). + - id: file://../providers/claude-skill-agent-sdk.js + label: mcp-auto + config: + skill_slug: launchdarkly-release-autopilot-onboarding + tool_definitions: release-autopilot-onboarding/tool-definitions.json + tool_mocks: release-autopilot-onboarding/tool-mocks.json + allow_builtins: true + expose_ask_question: true + +tests: + # ------------------------------------------------------------------ + # Fresh account, auto-flagging only. The repo mapping must turn + # auto-releasing off for the repo, the account write comes after the + # repo write, and no write happens before the confirm question. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Fresh account, auto-flagging only: confirm first, repo override off, account write last" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 25 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["acme/demo"] + - ["default"] + - ["LaunchDarkly's"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:accountWriteLast + metric: account_write_last + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverAutoReleasingOn + metric: no_auto_releasing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup + metric: no_auto_cleanup + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + + # ------------------------------------------------------------------ + # Account already on, both capabilities picked, policy mode with no + # release policy. list-release-plans is called once (the 403 is + # account-wide), no plans are created, the repo keeps auto-releasing + # off, and the account settings are not rewritten. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Policy mode, missing policy: one plans call, no plans, auto-releasing off for the repo, no account write" + vars: + user_request: "Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default." + max_turns: 20 + mock_overrides: + list-release-plans: + error: "LaunchDarkly API error: 403: {\"code\":\"forbidden\",\"message\":\"Plan does not allow this operation\"}" + list-release-policies: + policies: [] + count: 0 + warnings: + - code: missing_policy + message: "No release policy matches." + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging", "Auto-releasing"] + - ["acme/demo"] + - ["default"] + - ["LaunchDarkly's"] + - ["Test", "Production"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:releasePlansCalledAtMostOnce + metric: plans_403_account_wide + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:skippedGithubNotReported + metric: skipped_step_label + weight: 1 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noReleasePlanCreated + metric: no_plans_created + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: auto_releasing_off_without_policy + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite + metric: no_redundant_account_write + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + + # ------------------------------------------------------------------ + # Reader: Preview mode is forced, so nothing is written even after + # every question is answered. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Reader role: forced Preview, zero writes" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 20 + mock_overrides: + get-member-self: { id: "member-2", email: "reader@acme.example", role: "reader", teams: [] } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["acme/demo"] + - ["default"] + - ["LaunchDarkly's"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + + # ------------------------------------------------------------------ + # Explicit preview request from an admin: zero writes, and the final + # message is a checklist of changes. + # ------------------------------------------------------------------ + - providers: ["mcp-auto"] + description: "Explicit preview: zero writes, ends with the planned changes" + vars: + user_request: "Do a dry run: preview what you'd change to set up Release Autopilot auto-flagging for acme/demo in project default. Don't change anything." + max_turns: 20 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["acme/demo"] + - ["default"] + - ["LaunchDarkly's"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:previewNamesTheMapping + metric: preview_plan + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:previewIsShortChecklist + metric: preview_format + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + + # ------------------------------------------------------------------ + # GitHub App not installed, and the user needs an org owner: hand the + # user the link, write nothing, and stop. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "App not installed, org owner needed: link given, zero writes" + vars: + user_request: "Set up Release Autopilot for my repos." + max_turns: 15 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + list-release-autopilot-github-repos: + installed: false + userAuthorized: false + repositories: [] + totalCount: 0 + githubConnectUrl: "https://app.launchdarkly.com/integrations/github/connect?flow=install" + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Need an org owner"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:pauseSummaryHasConnectLink + metric: gave_connect_link + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:githubQuestionHasLink + metric: link_in_question + weight: 3 + + # ------------------------------------------------------------------ + # Writer role on a fresh account: account settings need an admin, so + # the account write is never attempted. The repo write may run. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Writer role: account write left for an admin" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." + max_turns: 25 + mock_overrides: + get-member-self: { id: "member-3", email: "writer@acme.example", role: "writer", teams: [] } + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["acme/demo"] + - ["default"] + - ["LaunchDarkly's"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite + metric: no_account_write + weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:notCalledComplete + metric: partly_set_up + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 2 + + # ------------------------------------------------------------------ + # "Change something" at the confirm step is not approval: no writes. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Change something at confirm: zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." + max_turns: 20 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["LaunchDarkly's"] + - ["Change something"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + + # ------------------------------------------------------------------ + # acme/storefront is already mapped to default and inherits + # auto-releasing from the account. The user picks auto-flagging only, + # so the only write is an override that turns auto-releasing off. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Already mapped with an unpicked capability: override write only" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/storefront in project default. I don't want auto-releasing." + max_turns: 25 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["LaunchDarkly's"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:storefrontAutoReleasingOff + metric: override_unpicked_on_mapped_repo + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAccountWrite + metric: no_redundant_account_write + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + + # ------------------------------------------------------------------ + # acme/storefront is mapped to default; the user asks for project web. + # The skill must not remap; it points to the settings skill instead. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Mapped to another project: no remap" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/storefront in project web." + max_turns: 20 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + - ["LaunchDarkly's"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noStorefrontRemap + metric: no_remap + weight: 5 + + # ------------------------------------------------------------------ + # Plan mode: release plans are available and empty. The plan is + # created before the repo write, and every phase has a clause. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Plan mode: release plan created first, valid phases" + vars: + user_request: > + Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default. + For production only, create a release plan: randomize by user, 1% then 10% then 50% then 100% + with 4h holds, automatic advancement, no metrics, and leave rules in place. + max_turns: 22 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging", "Auto-releasing"] + - ["LaunchDarkly's"] + - ["Production"] + - ["Automatic"] + - ["Leave rules"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:planBeforeRepoWrite + metric: plan_before_repo_write + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:plansHaveClauses + metric: plan_phases_have_clauses + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + + # ------------------------------------------------------------------ + # The user wants a repo whose GitHub organization has no App install. + # No githubConnectUrl appears (the account is installed elsewhere), so + # the skill must say an org owner installs the App, and write nothing. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Repo in an uninstalled GitHub org: org owner install, zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for other-org/app in project default." + max_turns: 15 + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:saysOrgOwnerInstall + metric: org_owner_install + weight: 3 + + # ------------------------------------------------------------------ + # An older MCP server flags itself as deprecated. The skill gives the + # install link and stops before asking anything or writing. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Deprecated MCP server: install link, zero writes" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 10 + mock_overrides: + get-member-self: + id: "member-1" + email: "admin@acme.example" + role: "admin" + teams: [] + latestMcp: + deprecated: true + installUrl: "https://mcp.launchdarkly.com/mcp/launchdarkly/install" + serverUrl: "https://mcp.launchdarkly.com/mcp/launchdarkly" + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 5 + - type: javascript + value: | + const text = String(output.response || '') + (output.assistant_turns || []).join('\n'); + const pass = text.includes('mcp.launchdarkly.com/mcp/launchdarkly/install'); + return { pass, score: pass ? 1 : 0, reason: pass ? 'gave the install link' : 'did not give latestMcp.installUrl' }; + metric: gave_install_link + weight: 3 + + # ------------------------------------------------------------------ + # Resume after the GitHub pause. The earlier picks are in the request; + # the App is now installed. No intro, no re-asked capability or + # approval questions, and the confirmed plan maps the repo. + # ------------------------------------------------------------------ + - providers: ["mcp-auto"] + description: "Resume after GitHub pause: no intro, no re-asked picks, maps after confirm" + vars: + user_request: > + Continue Release Autopilot setup. Earlier you recorded my picks: auto-flagging only, approval + required, repo acme/demo in project default, LaunchDarkly managed inference. We paused while + our GitHub org owner installed the App, and that is done now. + max_turns: 20 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noIntro + metric: no_intro + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noPickQuestions + metric: no_repeated_picks + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 3 + + # ------------------------------------------------------------------ + # The account write fails with 403 for an admin. The repo write comes + # first, nothing is written after the failure, and the account change + # is handed to an admin instead of being called complete. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Account write fails with 403: stop, hand off to an admin" + vars: + user_request: "Set up Release Autopilot auto-flagging for acme/demo in project default." + max_turns: 20 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + update-release-autopilot-settings: + error: "LaunchDarkly API error: 403: {\"code\":\"forbidden\",\"message\":\"Access to the requested resource was denied\"}" + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["LaunchDarkly's"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:stopsAfterFailedWrite + metric: stops_after_failure + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:notCalledComplete + metric: partly_set_up + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 + + # ------------------------------------------------------------------ + # Auto-cleanup picked with auto-flagging. Auto-cleanup is a browser + # step on the project's Lifecycle settings page: the link is in the + # question, autoCleanup is never sent, and the repo write is confirmed. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Auto-cleanup: lifecycle settings link in the question, never sends autoCleanup" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 22 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging", "Auto-cleanup"] + - ["LaunchDarkly's"] + - ["Done"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noReleaseSetup + metric: no_release_setup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion + metric: no_unpicked_archival + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup + metric: no_auto_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:cleanupQuestionHasLifecycleLink + metric: lifecycle_link_in_question + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noProductNameAfterIntro + metric: capability_names + weight: 1 + + # ------------------------------------------------------------------ + # The user installs the App and says "Done", but this connection still + # sees no install: the browser was signed in to a different account. + # The skill names this connection's member, asks which account the + # browser uses, and does not just repeat the link. No writes. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Installed from a different browser account: names the member, diagnoses, zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for my repos." + max_turns: 18 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + list-release-autopilot-github-repos: + installed: false + userAuthorized: false + repositories: [] + totalCount: 0 + githubConnectUrl: "https://app.launchdarkly.com/github-connect?flow=install" + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["Done"] + - ["A different one"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:relistsAfterDone + metric: relists_after_done + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:asksWhichBrowserAccount + metric: diagnoses_browser_account + weight: 4 + + # ------------------------------------------------------------------ + # Every Release Autopilot tool returns 404: the account does not have + # the add-on. The skill says so in those words and stops. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Add-on not enabled: every settings tool 404s, says so, zero writes" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 10 + mock_overrides: + get-release-autopilot-settings: { error: "LaunchDarkly API error: 404 Not Found" } + list-release-autopilot-repo-settings: { error: "LaunchDarkly API error: 404 Not Found" } + list-release-autopilot-github-repos: { error: "LaunchDarkly API error: 404 Not Found" } + get-release-autopilot-repo-settings: { error: "LaunchDarkly API error: 404 Not Found" } + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:saysAddOnNotEnabled + metric: add_on_wording + weight: 4 + + # ------------------------------------------------------------------ + # "Why didn't my PR get classified?" on an account with nothing set up. + # The skill answers first (auto-flagging is off), then offers setup + # instead of starting the wizard. The harness answers the unscripted offer + # with its first option, so any later writes must follow "Looks good". + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Diagnosis on an empty account: explains first, offers setup, no unconfirmed writes" + vars: + user_request: "Why didn't Release Autopilot classify my pull request acme/demo#42?" + max_turns: 12 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:diagnosesBeforeOffering + metric: diagnosis_first + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noUnconfirmedWrites + metric: no_unconfirmed_writes + weight: 4 + + # ------------------------------------------------------------------ + # Auto-archival picked with auto-flagging. It is a browser step on the + # project's Lifecycle settings page: the link is in the question, and + # the only write is the confirmed repo mapping. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Auto-archival: lifecycle settings link in the question, confirmed repo write only" + vars: + user_request: "Set up Release Autopilot for acme/demo in project default." + max_turns: 22 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: false } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging", "Auto-archival"] + - ["LaunchDarkly's"] + - ["Done"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noReleaseSetup + metric: no_release_setup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:neverSendsApproval + metric: never_sends_approval + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion + metric: no_unpicked_cleanup + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:archivalQuestionHasLifecycleLink + metric: lifecycle_link_in_question + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: repo_override_unpicked + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noAutoCleanup + metric: no_auto_cleanup + weight: 2 + + # ------------------------------------------------------------------ + # The App is installed but this member has not authorized GitHub. The + # question is about authorizing (no org-owner pause), and nothing is + # written while GitHub is not connected. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Authorize flow: asks to authorize, no org-owner option, pauses on failure, zero writes" + vars: + user_request: "Set up Release Autopilot auto-flagging for my repos." + max_turns: 12 + mock_overrides: + get-release-autopilot-settings: + autoFlagging: { enabled: false, approvalRequired: true } + autoReleasing: { enabled: false } + version: 1 + list-release-autopilot-repo-settings: { repos: [], totalCount: 0 } + list-release-autopilot-github-repos: + installed: true + userAuthorized: false + repositories: [] + totalCount: 0 + githubConnectUrl: "https://app.launchdarkly.com/github-connect?flow=authorize" + mock_ask_question_match: true + mock_ask_question_answers: + - ["Auto-flagging"] + - ["I can't authorize"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:pausesBeforeMapping + metric: pauses_before_mapping + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:authorizeQuestionNoOrgOwner + metric: authorize_question + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 3 diff --git a/evals/release-autopilot-onboarding/tool-definitions.json b/evals/release-autopilot-onboarding/tool-definitions.json new file mode 100644 index 00000000..5a29e75e --- /dev/null +++ b/evals/release-autopilot-onboarding/tool-definitions.json @@ -0,0 +1,303 @@ +[ + { + "name": "get-member-self", + "description": "Return the member ID, profile, role, and team memberships for the currently authenticated user.", + "input_schema": { + "type": "object", + "properties": {} + } + }, + { + "name": "get-release-autopilot-settings", + "description": "Get account-wide Release Autopilot settings: whether auto-flagging and auto-releasing are on, and whether auto-flagging pull requests need approval. These are the master gate. Do not use autoCleanup.", + "input_schema": { + "type": "object", + "properties": {} + } + }, + { + "name": "update-release-autopilot-settings", + "description": "Update account-wide Release Autopilot settings (auto-flagging and auto-releasing). Send only what changed. Requires permission to update account Release Autopilot settings. Do not send autoCleanup.", + "input_schema": { + "type": "object", + "properties": { + "autoFlagging": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "approvalRequired": { + "type": "boolean" + } + } + }, + "autoReleasing": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "approvalRequired": { + "type": "boolean" + } + } + } + } + } + }, + { + "name": "list-release-autopilot-github-repos", + "description": "List GitHub repositories on this account's LaunchDarkly GitHub App install, with name (owner/name) and numeric githubRepoId. When githubConnectUrl is present, give that link to the user and retry after they finish.", + "input_schema": { + "type": "object", + "properties": { + "projectKey": { + "type": "string" + } + } + } + }, + { + "name": "list-release-autopilot-repo-settings", + "description": "List Release Autopilot settings for GitHub repositories already mapped to a LaunchDarkly project. Each automation is the effective value. githubRepoId is numeric; repo names are not included.", + "input_schema": { + "type": "object", + "properties": {} + } + }, + { + "name": "get-release-autopilot-repo-settings", + "description": "Get Release Autopilot settings for one GitHub repository. Pass repo as owner/name, or githubRepoId. 404 means the repo is not mapped yet.", + "input_schema": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "githubRepoId": { + "type": "number" + }, + "projectKey": { + "type": "string" + } + } + } + }, + { + "name": "update-release-autopilot-repo-settings", + "description": "Map a GitHub repository to a LaunchDarkly project and optionally override auto-flagging or auto-releasing. Pass repo as owner/name. projectKey is required on the first map. An omitted automation inherits the account setting. Do not send autoCleanup.", + "input_schema": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "githubRepoId": { + "type": "number" + }, + "projectKey": { + "type": "string" + }, + "autoFlagging": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "approvalRequired": { + "type": "boolean" + } + } + }, + "autoReleasing": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "approvalRequired": { + "type": "boolean" + } + } + } + } + } + }, + { + "name": "delete-release-autopilot-repo-settings", + "description": "Unmap a GitHub repository from Release Autopilot.", + "input_schema": { + "type": "object", + "properties": { + "repo": { + "type": "string" + }, + "githubRepoId": { + "type": "number" + } + } + } + }, + { + "name": "list-release-plans", + "description": "List environment-scoped release plans. A release plan is a reusable progressive-release recipe for one environment. It replaces project-level release policies.", + "input_schema": { + "type": "object", + "properties": { + "projectKey": { + "type": "string" + }, + "environmentKey": { + "type": "string" + }, + "includeIncomplete": { + "type": "boolean" + } + }, + "required": [ + "projectKey", + "environmentKey" + ] + } + }, + { + "name": "create-release-plan", + "description": "Create an environment-scoped release plan, a reusable progressive-release recipe. Does not start a release. PHASES: 1-10; every phase needs 1-10 clauses AND 1-10 steps. An \"everyone\" phase still needs a clause, e.g. { attribute: \"kind\", op: \"in\", values: [\"user\"], contextKind: \"user\" }. STEPS: allocation is per-100k (5000 = 5%, 100000 = 100%); durationMillis is the hold before the next step. METRICS: keyed object; {} = progressive. advancementMode defaults to manual.", + "input_schema": { + "type": "object", + "properties": { + "projectKey": { + "type": "string" + }, + "environmentKey": { + "type": "string" + }, + "key": { + "type": "string" + }, + "name": { + "type": "string" + }, + "description": { + "type": "string" + }, + "progressiveRelease": { + "type": "object", + "properties": { + "randomizationUnit": { + "type": "string" + }, + "advancementMode": { + "type": "string", + "enum": [ + "manual", + "automatic" + ] + }, + "cleanupBehavior": { + "type": "string", + "enum": [ + "leave_rules_in_place", + "remove_rules_and_update_default_rule" + ] + }, + "phases": { + "type": "array", + "items": { + "type": "object", + "properties": { + "description": { + "type": "string" + }, + "clauses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "attribute": { + "type": "string" + }, + "op": { + "type": "string", + "enum": [ + "in", + "endsWith", + "startsWith", + "matches", + "contains", + "lessThan", + "lessThanOrEqual", + "greaterThan", + "greaterThanOrEqual", + "before", + "after", + "semVerEqual", + "semVerLessThan", + "semVerGreaterThan", + "segmentMatch" + ] + }, + "values": { + "type": "array", + "items": {} + }, + "contextKind": { + "type": "string" + }, + "negate": { + "type": "boolean" + } + }, + "required": [ + "attribute", + "op", + "values" + ] + } + }, + "steps": { + "type": "array", + "items": { + "type": "object", + "properties": { + "allocation": { + "type": "number" + }, + "durationMillis": { + "type": "number" + } + }, + "required": [ + "allocation", + "durationMillis" + ] + } + } + }, + "required": [ + "clauses", + "steps" + ] + } + }, + "metrics": { + "type": "object" + } + }, + "required": [ + "randomizationUnit", + "phases" + ] + } + }, + "required": [ + "projectKey", + "environmentKey", + "key", + "name", + "progressiveRelease" + ] + } + } +] diff --git a/evals/release-autopilot-onboarding/tool-mocks.json b/evals/release-autopilot-onboarding/tool-mocks.json new file mode 100644 index 00000000..b8bd7c35 --- /dev/null +++ b/evals/release-autopilot-onboarding/tool-mocks.json @@ -0,0 +1,108 @@ +{ + "get-member-self": { + "id": "member-1", + "email": "admin@acme.example", + "role": "admin", + "teams": [] + }, + "get-release-autopilot-settings": { + "autoFlagging": { + "enabled": true, + "approvalRequired": true + }, + "autoReleasing": { + "enabled": true + }, + "version": 9 + }, + "update-release-autopilot-settings": { + "autoFlagging": { + "enabled": true, + "approvalRequired": true + }, + "autoReleasing": { + "enabled": false + }, + "version": 10 + }, + "list-release-autopilot-github-repos": { + "installed": true, + "userAuthorized": true, + "repositories": [ + { + "githubRepoId": 101, + "name": "acme/storefront", + "isPublic": false + }, + { + "githubRepoId": 102, + "name": "acme/api", + "isPublic": false + }, + { + "githubRepoId": 103, + "name": "acme/demo", + "isPublic": true + } + ], + "totalCount": 3 + }, + "list-release-autopilot-repo-settings": { + "repos": [ + { + "githubRepoId": 101, + "projectKey": "default", + "autoFlagging": { + "enabled": true, + "approvalRequired": true, + "enabledOverride": false + }, + "autoReleasing": { + "enabled": true, + "enabledOverride": false + } + }, + { + "githubRepoId": 999, + "projectKey": "default", + "autoFlagging": { + "enabled": true, + "approvalRequired": true, + "enabledOverride": false + }, + "autoReleasing": { + "enabled": true, + "enabledOverride": false + } + } + ], + "totalCount": 2 + }, + "get-release-autopilot-repo-settings": { + "error": "LaunchDarkly API error: 404: Factory settings for this repository were not found" + }, + "update-release-autopilot-repo-settings": { + "projectKey": "{{projectKey}}", + "autoFlagging": { + "enabled": true, + "approvalRequired": true + }, + "autoReleasing": { + "enabled": false, + "enabledOverride": true + }, + "version": 1 + }, + "delete-release-autopilot-repo-settings": { + "deleted": true + }, + "list-release-plans": { + "items": [], + "totalCount": 0 + }, + "create-release-plan": { + "key": "new-plan", + "environmentKey": "{{environmentKey}}", + "created": true + } +} diff --git a/evals/scripts/_manifest.js b/evals/scripts/_manifest.js index 6882661f..a4eb0707 100644 --- a/evals/scripts/_manifest.js +++ b/evals/scripts/_manifest.js @@ -79,6 +79,12 @@ const SUITES = [ skillDir: "skills/onboarding", readme: "skills/onboarding/README.md", }, + { + suite: "release-autopilot-onboarding", + skillKey: "release-autopilot/launchdarkly-release-autopilot-onboarding", + skillDir: "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", + readme: "skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md", + }, { suite: "experiment-guide", skillKey: "experiments/launchdarkly-experiment-guide", diff --git a/evals/tests/mock.test.js b/evals/tests/mock.test.js index 4a96127c..852dae5a 100644 --- a/evals/tests/mock.test.js +++ b/evals/tests/mock.test.js @@ -12,7 +12,7 @@ const TEMPLATES = require("../mocks/tool-responses.json"); test("createMockState returns fresh empty state each call", () => { const a = createMockState(); const b = createMockState(); - assert.deepEqual(a, { configs: {}, flags: {} }); + assert.deepEqual(a, { configs: {}, flags: {}, raSettings: null, raRepos: {}, raPlans: {} }); a.configs["x"] = {}; assert.deepEqual(b.configs, {}); // independent }); @@ -422,3 +422,42 @@ test("update-ai-config uses template substitution", () => { assert.equal(result.key, "bot"); assert.equal(result.name, "Bot"); }); + +// --------------------------------------------------------------------------- +// Release Autopilot stateful reads (templates from the suite's own mocks) + +const RA_TEMPLATES = require("../release-autopilot-onboarding/tool-mocks.json"); +// --------------------------------------------------------------------------- + +test("repo settings read reflects an earlier repo write", () => { + const state = createMockState(); + const tpl = RA_TEMPLATES["get-release-autopilot-repo-settings"]; + assert.ok(renderMockResponse(tpl, { repo: "acme/demo" }, "get-release-autopilot-repo-settings", state).error); + renderMockResponse(RA_TEMPLATES["update-release-autopilot-repo-settings"], { repo: "acme/demo", projectKey: "default", autoReleasing: { enabled: false } }, "update-release-autopilot-repo-settings", state); + const got = renderMockResponse(tpl, { repo: "acme/demo" }, "get-release-autopilot-repo-settings", state); + assert.equal(got.projectKey, "default"); + assert.equal(got.autoReleasing.enabled, false); +}); + +test("account settings read merges an earlier account write", () => { + const state = createMockState(); + renderMockResponse(RA_TEMPLATES["update-release-autopilot-settings"], { autoFlagging: { enabled: false } }, "update-release-autopilot-settings", state); + const got = renderMockResponse(RA_TEMPLATES["get-release-autopilot-settings"], {}, "get-release-autopilot-settings", state); + assert.equal(got.autoFlagging.enabled, false); + assert.equal(got.autoFlagging.approvalRequired, true); +}); + +test("release plans list reflects a created plan", () => { + const state = createMockState(); + renderMockResponse(RA_TEMPLATES["create-release-plan"], { projectKey: "default", environmentKey: "production", key: "p", name: "P", progressiveRelease: { phases: [] } }, "create-release-plan", state); + const got = renderMockResponse(RA_TEMPLATES["list-release-plans"], { projectKey: "default", environmentKey: "production" }, "list-release-plans", state); + assert.equal(got.totalCount, 1); + assert.equal(got.items[0].key, "p"); +}); + +test("an error template wins over the stateful write and is not recorded", () => { + const state = createMockState(); + const err = { error: "403: denied" }; + assert.equal(renderMockResponse(err, { autoFlagging: { enabled: true } }, "update-release-autopilot-settings", state).error, "403: denied"); + assert.equal(state.raSettings, null); +}); diff --git a/evals/tests/provider-match-option.test.js b/evals/tests/provider-match-option.test.js new file mode 100644 index 00000000..0e8c90e8 --- /dev/null +++ b/evals/tests/provider-match-option.test.js @@ -0,0 +1,47 @@ +const { test } = require("node:test"); +const assert = require("node:assert/strict"); + +// matchOption resolves a scripted ask-question answer to an option id, so a +// test can steer by label when the agent invents the ids. +const { matchOption, pickAnswer } = require("../providers/claude-skill-agent-sdk"); + +const options = [ + { id: "af", label: "Auto-flagging (Recommended)" }, + { id: "ar", label: "Auto-releasing" }, +]; + +test("keeps an exact id", () => { + assert.equal(matchOption("ar", options), "ar"); +}); + +test("matches a label substring, case-insensitively", () => { + assert.equal(matchOption("auto-flagging", options), "af"); +}); + +test("passes an unmatched answer through", () => { + assert.equal(matchOption("nothing", options), "nothing"); +}); + +test("tolerates missing options", () => { + assert.equal(matchOption("x", undefined), "x"); +}); + +test("order mode returns the answer at the call index as written", () => { + assert.deepEqual(pickAnswer([["x"], ["Auto-releasing"]], 1, options, false, new Set()), ["Auto-releasing"]); +}); + +test("match mode skips answers that fit no option, and uses each once", () => { + const used = new Set(); + const answers = [["Looks good"], ["Auto-flagging"], ["Auto-flagging"]]; + assert.deepEqual(pickAnswer(answers, 0, options, true, used), ["af"]); + assert.deepEqual(pickAnswer(answers, 1, options, true, used), ["af"]); + assert.equal(pickAnswer(answers, 2, options, true, used), undefined); +}); + +test("match mode returns undefined when nothing fits", () => { + assert.equal(pickAnswer([["Looks good"]], 0, options, true, new Set()), undefined); +}); + +test("order mode keeps an explicit [] as a dismissed question", () => { + assert.deepEqual(pickAnswer([[]], 0, options, false, new Set()), []); +}); diff --git a/skills.json b/skills.json index acfae194..742d1f88 100644 --- a/skills.json +++ b/skills.json @@ -427,11 +427,31 @@ "license": "Apache-2.0", "compatibility": "Requires the remotely hosted LaunchDarkly MCP server" }, + { + "name": "launchdarkly-release-autopilot-onboarding", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", + "version": "0.2.0-experimental", + "license": "Apache-2.0", + "compatibility": "Requires the remotely hosted LaunchDarkly MCP server", + "tags": [ + "launchdarkly", + "release-autopilot", + "onboarding", + "github", + "auto-flagging", + "auto-releasing", + "auto-cleanup", + "auto-archival", + "release-plans", + "mcp" + ] + }, { "name": "launchdarkly-release-autopilot-settings", - "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR.", + "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, review release policies), use launchdarkly-release-autopilot-onboarding instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-settings", - "version": "1.3.0-experimental", + "version": "1.4.0-experimental", "license": "Apache-2.0", "compatibility": "Requires the remotely hosted LaunchDarkly MCP server", "tags": [ diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md new file mode 100644 index 00000000..e76c67fe --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -0,0 +1,43 @@ +# LaunchDarkly Release Autopilot Onboarding Skill + +An Agent Skill that walks a user through setting up Release Autopilot: pick capabilities, connect the GitHub App, map repositories, choose Vega inference, and set up environment release plans. + +## Overview + +This skill teaches agents how to: + +- Ask which capabilities the user wants (auto-flagging, auto-releasing) and set up only what those need +- Hand the user the GitHub App install or authorize link, and continue after they finish +- Map chosen `owner/name` repos to LaunchDarkly projects +- Offer auto-cleanup and auto-archival, and guide the project's Lifecycle settings for each +- Explain whose AI model account runs the agents (LaunchDarkly's by default, or the user's own Anthropic or OpenAI key) and guide bring-your-own-key setup +- Review or create environment release plans for auto-releasing, or show release policies where plans are not available +- Confirm one plan that lists every write before making any change, then verify + +## Installation (Local) + +- **Generic**: copy `skills/release-autopilot/launchdarkly-release-autopilot-onboarding/` into your client's skills path + +## Prerequisites + +The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools, and `list-release-policies` and `list-metrics` for auto-releasing (`list-release-plans` and `create-release-plan` when the account has environment release plans). Accounts without the Release Autopilot add-on enabled get 404s. + +## Usage + +``` +Set up Release Autopilot for this repo +``` + +``` +Onboard us to Release Autopilot with auto-flagging and auto-releasing +``` + +## Related + +- [LaunchDarkly Release Autopilot Settings](../launchdarkly-release-autopilot-settings/): single settings changes and "why didn't my PR get classified?" +- [LaunchDarkly Metric Choose](../../metrics/launchdarkly-metric-choose/): pick metrics for guarded release plans +- [LaunchDarkly MCP Server](https://github.com/launchdarkly/mcp-server) + +## License + +Apache-2.0 diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md new file mode 100644 index 00000000..52f578b4 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -0,0 +1,322 @@ +--- +name: launchdarkly-release-autopilot-onboarding +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), review release policies (release plans by GA), and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +license: Apache-2.0 +compatibility: Requires the remotely hosted LaunchDarkly MCP server +metadata: + author: launchdarkly + version: "0.2.0-experimental" +--- + +# LaunchDarkly Release Autopilot Onboarding + +You're using a skill that walks a user through setting up Release Autopilot. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and the environment release setup for auto-releasing. + +**This is a step-by-step wizard.** Ask one question at a time and wait for each answer. Gather every decision first, confirm one plan, and only then write. Release Autopilot opens and changes pull requests, and auto-releasing can ship code, so every write is production automation. + +Follow the write rules of [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md) (least privilege, `owner/name` only, never `autoCleanup`, current → proposed with blast radius). Its one-at-a-time confirmations are replaced by the single itemized confirmation at [Step 7](#step-7-confirm-the-plan). + +## Prerequisites + +This skill requires the remotely hosted LaunchDarkly MCP server. + +**Required MCP tools:** +- `get-release-autopilot-settings` / `update-release-autopilot-settings` +- `list-release-autopilot-github-repos` +- `list-release-autopilot-repo-settings` / `get-release-autopilot-repo-settings` / `update-release-autopilot-repo-settings` +- `get-member-self` + +**Required for auto-releasing:** +- `list-release-policies` + +**Used when available:** +- `list-release-plans` / `create-release-plan` / `list-metrics`: environment release plans (see [Release setup](#release-setup)) +- `list-projects` / `get-project`: offer project and environment keys as options instead of asking for free text + +If a required tool is missing, stop and tell the user which one. Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. + +If every Release Autopilot tool returns 404, the account does not have the Release Autopilot add-on enabled. Tell the user "Your account does not have the Release Autopilot add-on enabled," and stop. + +## Modes + +Pick the mode from the user's request before Step 0: + +| Mode | Trigger | What changes | +|------|---------|--------------| +| **Setup** | "set up", "onboard", "turn on" | The full flow | +| **Preview** | "preview", "what would you change", "dry run", or a reader (see Step 1) | Run Steps 1–7 and end with the plan as a checklist for an admin. Make no writes | +| **Resume** | "continue", "pick up where we left off", or a return after a browser step | Skip the intro, re-read live state, and continue at the first unfinished step. Still stops at Step 7 and waits for "Looks good". Details: [references/modes.md](references/modes.md) | + +Every mode is safe to run again. A re-run reads live state and proposes only the writes that are still needed. + +## Core Principles + +1. **Only set up what they picked.** Release setup matters only for auto-releasing. If the user picks auto-flagging only, skip Step 6. +2. **Least privilege by default.** Recommend auto-flagging, which always waits for a person to approve its proposed change. Auto-releasing is opt-in and you must say it can ship code. +3. **Gather, confirm, then write.** No write happens before [Step 7](#step-7-confirm-the-plan). The confirmation lists every write with its current and proposed state. This holds in every mode, including Resume and requests that already state every answer: a restated pick is an answer, never approval. Step 1's reads, including the role check, always run first. +4. **Don't ask what the account already answers.** If live state already settles a question, state the answer and move on. +5. **Never act in the user's browser.** GitHub install, GitHub authorization, release policy edits, and bring-your-own-key setup are browser steps. Give the exact link and wait for the user to say they finished. +6. **Only touch what the user named.** Never map every repo on the install. Never create release plans in environments the user did not pick. + +## Speaking to the user + +- Use product words ("auto-flagging", "release policy", "approval") in every message, including questions, checklists, and the final summary. Never show tool names, field names (`githubRepoId`, `autoReleasing.enabled`), HTTP status codes, step numbers, or this skill's mode names ("preview mode", "policy mode", "plan mode"); say "Here's what would change", "your account uses release policies", and "Verification", not "Step 9". +- Say "Release Autopilot" only in the welcome. After that, name the capability ("auto-flagging", "auto-releasing", "auto-cleanup", "auto-archival"), or say "these capabilities". The exceptions are the resume phrase "continue Release Autopilot setup", the name of the settings skill, and "the Release Autopilot add-on". +- Name repos as `owner/name`. Show a repository ID only for a mapped repo with no name. +- Describe each change in words ("Turn on auto-flagging for the account"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. +- Every browser step (GitHub, bring your own key, Lifecycle or release settings) must be done while signed in as the member from Step 1. Say so when you give the link. +- State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"), and do not invent UI locations: give only the pages this skill names. For an account change left for an admin, tell them to run [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). +- Keep each message to the current step, and do not narrate tool calls. + +## Workflow + +When a step is complete, go to the next one unless it says to ask or wait. Treat user messages as answers to the current question, and track every decision for Step 7. + +**Asking questions:** When a question has options, use `AskUserQuestion` with the labels and descriptions given below. Ask free-text questions as a normal message. If the user's request already answers a question explicitly, such as naming the repo, the project, or the capabilities, use that answer and say so instead of asking. Never infer the answer to the Step 7 confirmation. + +**Links:** the question dialog covers the message above it. When a question follows a link the user must open (GitHub, bring your own key, lifecycle or release settings), put the exact URL in the question text itself, not only in the message before it. + +### Step 0: Welcome the user + +If the user's request is a diagnosis ("why didn't my PR get classified?", "why wasn't it auto-flagged?"), do not start this wizard, whatever the account state: follow the diagnosis in [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified). It is read-only; if it finds that nothing is set up, offer this wizard afterward. + +Decide the mode from the user's first message before you write anything. Never announce the mode by name; for a preview request, say "I'll show what would change and make no changes." **In Resume mode, skip this step entirely**: do not show the intro or any part of it, and go straight to Step 1. In Setup mode, present this intro (you may lightly adapt the wording, but keep the structure). In Preview mode, replace the last line with "I'll show you every change and make none of them." + +> **Release Autopilot Setup** +> +> I'll set up Release Autopilot in a few short steps: check your account, pick capabilities, connect GitHub, map repositories, choose whose AI model account runs the agents, then set up releases, flag cleanup, and archival for the capabilities you pick. +> +> Nothing changes until you approve the full plan. + +Then go to Step 1. Skip any step that the account state already completes, and say why in one line. + +### Step 1: Read current state + +Run these reads. Do not write anything. + +1. `get-member-self`: the member's `role`, and their `email`. Keep the email: every browser step must happen while signed in to LaunchDarkly as that member (see [Step 3](#step-3-connect-github)). + - `reader` or `no_access`: switch to Preview. Tell the user: "You have read-only access, so I'll show what would change and make no changes." + - `admin` or `owner`: can make every write in this flow. + - Any other role, including `writer` and custom roles: may map repos in projects they can edit, but account settings need an admin or owner. In Step 7, mark account writes as "needs an admin" and leave them out of Step 8. +2. `get-release-autopilot-settings`: the account gates. +3. `list-release-autopilot-repo-settings`: repos that are already mapped. +4. `list-release-autopilot-github-repos`: repos on the GitHub App install (`owner/name` and `githubRepoId`), plus `installed` and `userAuthorized`. If either is false, the response includes a `githubConnectUrl`. + +Join mapped repos to the install list by repository ID to get their names. For a mapped ID that is not on the install, tell the user, for example, "One more repository (ID 999) is mapped to project default but is no longer on the GitHub install; I'll leave it alone." Do not write the field name or change that repo. If a later step fails with a generative AI eligibility error, see [references/edge-cases.md](references/edge-cases.md). + +Summarize what is already set up in two or three sentences. If the account already has a capability on or any repo mapped, ask: + +- label: "Add more", description: "Keep the current setup and add capabilities, repos, or release setup" +- label: "Change one setting", description: "Hand off to the settings skill for a single change" +- label: "Diagnose a PR", description: "A PR was not classified or auto-flagged. Hand off to the settings skill" + +In Preview mode, leave out "Change one setting", because that hand-off writes. For "Change one setting" or "Diagnose a PR", stop this skill and follow [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). + +### Step 2: Pick capabilities + +Use `AskUserQuestion` with header "Capabilities", `multiSelect: true`: + +- label: "Auto-flagging (Recommended)", description: "Reads each pull request and, when a change should ship behind a flag, opens a PR that adds the flag" +- label: "Auto-releasing", description: "After a flagged change deploys, releases it with the environment's release policy or release plan. This can ship code to users" +- label: "Auto-cleanup", description: "On a schedule, opens pull requests that remove the code for flags that are ready for removal. This removes code" +- label: "Auto-archival", description: "Archives flags that are ready to archive, after safety checks and a grace period" + +Do not ask about approval. Auto-flagging always waits for a person: it posts its proposed change on the pull request, and nothing changes until someone replies `@launchdarkly approve` (or `approve flag`) or presses Approve on the check. If the user picked auto-flagging, say that in one sentence. Never send `approvalRequired`; the backend does not read it today, so turning it off would not skip that step. + +Auto-cleanup and auto-archival are opt-in like auto-releasing: never recommend them by default. If the user picks no capability, stop and tell them there is nothing to set up. + +### Step 3: Connect GitHub + +Skip this step without mentioning GitHub if Step 1 returned `installed: true`, `userAuthorized: true`, and no `githubConnectUrl`. + +Otherwise read the URL's `flow`: + +- `flow=install`: the LaunchDarkly GitHub App is not installed on their GitHub organization. None of these capabilities can see a pull request until it is. +- `flow=authorize`: the App is installed, but this member has not authorized GitHub, so repos cannot be listed. + +Give the user the exact URL. The link acts on whatever LaunchDarkly session the browser has, so tell them to open it while signed in as the member from Step 1 (for example, "Open it while signed in to LaunchDarkly as admin@acme.example"). Then ask header "GitHub", with the URL in the question text. The options depend on the flow: + +- `flow=install`: to install, they must be a GitHub organization owner, or request the install from one. Question: "Open https://… to install the LaunchDarkly GitHub App. Done?" + - label: "Done", description: "I installed the App" + - label: "Need an org owner", description: "I can't install the App myself. Pause setup here" +- `flow=authorize`: the App is already installed, so no org owner is needed. They authorize with a GitHub account that is a member of the organization where the App is installed. Question: "Open https://… and authorize with a GitHub account in the organization that installed the App. Done?" + - label: "Done", description: "I authorized GitHub" + - label: "I can't authorize", description: "My GitHub account can't see that organization's install" + +On "Done", call `list-release-autopilot-github-repos` again. If it still returns `githubConnectUrl`, do not repeat the link. The most common cause is a browser signed in to a different LaunchDarkly account or member, so the App was installed there. Ask header "Browser account", with this connection's email in the question text (for example, "This connection is signed in as admin@acme.example. Which account is your browser signed in to?"): + +- label: "The same", description: "My browser is signed in as that member" +- label: "A different one", description: "My browser is signed in to another account or member" +- label: "Not sure", description: "I need to check" + +For "A different one" or "Not sure", tell them to check at `/profile`, sign in as the member from Step 1, open the link again, and finish there; then list again. For "The same", check the other causes in [references/github-troubleshooting.md](references/github-troubleshooting.md). On "Need an org owner", summarize the decisions so far, repeat the exact URL for the org owner, tell the user to say "continue Release Autopilot setup" after the App is installed, and stop. On "I can't authorize", pause the same way: summarize the decisions so far, tell them to sign in to GitHub with an account that is a member of that organization (or ask an org owner to add theirs), repeat the exact URL, tell them to say "continue Release Autopilot setup" once they have authorized, and stop. See the GitHub-identity cause in [references/github-troubleshooting.md](references/github-troubleshooting.md). + +Never go on to Step 4 while `list-release-autopilot-github-repos` still returns `githubConnectUrl`: an empty repo list then means GitHub is not connected, not that the repo's organization lacks the App. + +### Step 4: Map repositories + +1. **Choose repos.** If the current workspace has a `git remote` on the install list, offer it first. Use `AskUserQuestion` with header "Repos", `multiSelect: true`, and up to four repos from the install list, as `owner/name`. If the install has more than four repos, say the user can type any other `owner/name` from the list in Other. Use only repos that came back from `list-release-autopilot-github-repos`. + + If a repo the user wants is not on the list, its owner decides the fix: a listed owner means the App needs access to that repo, and a new owner means an org owner must install the App there (no `githubConnectUrl` appears for this case). See [references/edge-cases.md](references/edge-cases.md). Wait for "Done" and list again. +2. **Choose a project for each repo.** Each repo maps to one LaunchDarkly project, which is where auto-flagging creates flags and metrics for that repo. If `list-projects` is available, offer the projects as options. Otherwise ask for the project key. +3. **Check existing mappings** with the join from Step 1 (an unmapped repo returns 404, which is expected). Already mapped to the chosen project with matching capabilities: write nothing. Mapped to the chosen project with an unpicked capability on: record an override that turns it off. Mapped to a different project: never remap here; send the user to the settings skill to unmap first. Details: [references/edge-cases.md](references/edge-cases.md#existing-mappings). +4. **Say what starts when.** If the account already has a capability on, mapping a repo starts that automation in the repo as soon as Step 8 maps it. + +### Step 5: Choose the AI model account + +The agents behind these capabilities (Vega) use an AI model to read pull requests and write code changes. This step decides whose model account they use; the product calls it "inference". Most users keep the default. Before asking, tell the user in one sentence what the choice means, for example: "The agents behind auto-flagging use an AI model to read your pull requests. By default they run on LaunchDarkly's model account; you can use your own Anthropic or OpenAI key instead." Then use `AskUserQuestion` with header "AI model": + +- label: "LaunchDarkly's (default)", description: "The agents use an AI model that LaunchDarkly provides and pays for. No setup" +- label: "Bring your own key", description: "The agents use your own Anthropic or OpenAI API key, for example to keep usage on your provider contract. Your provider bills you for usage" + +For "Bring your own key", this is a browser step (no MCP tool reads or writes these settings). For each project, follow [references/byok.md](references/byok.md), then ask header "AI model": + +- label: "Done", description: "I saved my key in the Vega settings for every project" +- label: "Use LaunchDarkly's for now", description: "Skip this. I'll switch to my own key later" + +Record the answer as reported by the user, not as verified. + +### Step 6: Set up releases and cleanup + +Run each part of this step only for a capability the user picked in Step 2, and skip the step if they picked none of them. Never walk the user through a capability they did not pick: + +- Auto-releasing picked: [Release setup](#release-setup). +- Auto-cleanup picked: [Auto-cleanup](#auto-cleanup). +- Auto-archival picked: [Auto-archival](#auto-archival). + +#### Release setup + +Only if the user picked auto-releasing. Otherwise skip to [Auto-cleanup](#auto-cleanup) or [Auto-archival](#auto-archival), and do not ask about environments or look up release plans or policies. + +For each project from Step 4, choose environments: use `AskUserQuestion` with header "Environments", `multiSelect: true`, offering the project's environments (from `get-project`, or ask). Then check once per account whether it uses release plans or release policies. + +##### Check whether the account uses release plans + +Call `list-release-plans` for the first chosen environment. + +- **Release policies** if the call returns 403 `Plan does not allow this operation`, or if `list-release-plans` is not available. The 403 is account-wide, so do not call it for other environments. This is the mode for every account today: auto-releasing requires a complete release policy for each environment. +- **Release plans** if the call succeeds. Environment release plans replace release policies by general availability. + +##### Release policies (today) + +Call `list-release-policies` for the project and show the user what it returns, including any `warnings`. Record no plan writes. This skill does not change policies. + +- No warnings for the chosen environments: say that auto-releasing in this project uses those policies. +- Any warning, such as `missing_policy` or `incomplete_policy`: auto-releasing cannot work there yet. Tell the user to add or complete a release policy in the project's release settings (`/projects/{projectKey}/settings/releases` on their LaunchDarkly instance). Record a repo override that keeps auto-releasing off for each repo in that project, and say in Step 7 that they can turn it on with the settings skill once the policy is complete. + +##### Release plans (by GA) + +For each chosen environment, call `list-release-plans`. If a complete plan exists, show its name, phases, and metrics, and ask header "Plan": + +- label: "Use this plan", description: "Keep the existing plan for this environment" +- label: "Create another", description: "Add a new plan for this environment" + +To create a plan, follow [references/release-plans.md](references/release-plans.md). Record the plan and any metrics to create. Do not call `create-release-plan` or create metrics yet. + +#### Auto-cleanup + +Only if the user picked auto-cleanup. + + +Auto-cleanup is the per-project **Automated flag cleanup** setting, on the Lifecycle settings page (or the Vega settings page, for accounts where Vega has its own project settings). The MCP settings tools do not manage it yet, so never send `autoCleanup`; this is a browser step. If a later version of `get-release-autopilot-settings` returns `autoCleanup` and the update tool accepts it, set it there instead. For each project from Step 4, the link is exactly `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (or `/projects/{projectKey}/settings/vega` if that page has no "Automated flag cleanup" section). Follow [references/auto-cleanup.md](references/auto-cleanup.md), then ask header "Auto-cleanup", with the link in the question text: + +- label: "Done", description: "I turned on automated flag cleanup for every project" +- label: "Later", description: "Skip this. I'll turn it on from Lifecycle settings later" + +Record the answer as reported by the user, and list "Done" under browser steps in Step 7. + +#### Auto-archival + +Only if the user picked auto-archival. + + +Auto-archival is the per-project **Automatic archival** setting on the Lifecycle settings page. No MCP tool manages it, so this is a browser step. For each project from Step 4, the link is exactly `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance. Follow [references/auto-archival.md](references/auto-archival.md), then ask header "Auto-archival", with the link in the question text: + +- label: "Done", description: "I turned on automatic archival for every project" +- label: "Later", description: "Skip this, or the section isn't there yet. I'll turn it on from Lifecycle settings later" + +Record the answer as reported by the user, and list "Done" under browser steps in Step 7. + +### Step 7: Confirm the plan + +**STOP. Do not write anything until the user explicitly confirms.** A detailed request at the start is not approval. + +List every write in the order you will make it. For each write, state the current and the proposed state and the blast radius, using the wording rules from the settings skill's [Confirm before write](../launchdarkly-release-autopilot-settings/SKILL.md#confirm-before-write): + +1. **Metrics and release plans** (release plans only): one line per metric to create and per environment, with the plan name, steps, advancement, and metrics. +2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; this project has no release policy yet)." +3. **Account settings**: only fields that change. If auto-releasing turns on, say it can ship code. + For a member who is not an admin or owner, list account writes under "Needs an admin" instead, and say an admin can make them with the settings skill. Do not name a settings page; there is none to point to. +4. **Browser steps the user reported done**: GitHub, the AI model choice, auto-cleanup, and auto-archival, marked as reported, not verified. +5. **Follow-ups for the user**: release policies to add, metric events to instrument. + +If there are no writes, say so, list the follow-ups, and stop. + +**Preview mode:** the final message contains only the heading "Changes an admin would make", the checklist (one line per change), and any follow-ups. Do not repeat the account state, and do not ask for confirmation. For example: + +``` +- [ ] Map acme/demo to project default, with auto-releasing off for this repository +- [ ] Turn on auto-flagging for the account +``` + +**Setup mode:** use `AskUserQuestion` with header "Confirm": + +- label: "Looks good", description: "Make every change in this plan" +- label: "Change something", description: "Adjust the plan before you start" + +Only "Looks good" approves the plan. On "Change something", ask what to change, make the change, and show the full plan again. Any other answer, including one that does not match an option, is not approval: ask the confirmation again. Do not go to Step 8 until the user selects "Looks good". + +### Step 8: Apply + +Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. + +1. Release plans only: metrics recorded in Step 6, created in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). Skip its instrumentation steps. +2. Release plans only: `create-release-plan` for each new plan. +3. `update-release-autopilot-repo-settings` for each repo, with `repo: "owner/name"` and `projectKey`. For each capability the user did not pick, or that cannot work yet, send `{ "enabled": false }` for it, so the repo does not inherit it from the account. +4. `update-release-autopilot-settings` with only the confirmed fields. Never send `autoCleanup`. + +The order matters. When the plan creates release plans, they come first, so that no release can start without the plan the user chose. The account gates come last because turning a capability on starts automation in every mapped repo that inherits it. + +If a write fails with 403, the member lacks that permission. Report which write failed, list the remaining writes as a checklist for an admin, and stop. + +### Step 9: Verify + +1. `get-release-autopilot-settings`: the account gates match the plan. +2. `get-release-autopilot-repo-settings` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. +3. With release plans: `list-release-plans` for each environment lists the new plans. With release policies: report the policy state from Step 6. + +If any change was left for an admin or failed, lead with "Partly set up: waiting on an admin" and the admin checklist, and never call the setup complete. Then report a short checklist with each item marked verified, reported by the user (browser steps the user did in this flow), or already set up (steps the flow skipped because they were already done), then the follow-ups. Tell the user how to see it work; if a change is waiting on an admin, make that conditional ("Once an admin turns on auto-flagging, open a pull request…"): + +- Open a pull request in a mapped repo. Auto-flagging reads it within a few minutes and posts its proposed change on the PR; nothing changes until someone replies `@launchdarkly approve` or presses Approve on the check. +- If a PR is not classified, use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md#why-didnt-my-pr-get-classified) to diagnose it. + +## Edge Cases + +| Situation | Action | +|-----------|--------| +| Every settings tool returns 404 | The account does not have the Release Autopilot add-on enabled. Tell the user and stop | +| Member role is `reader` or `no_access` | Preview mode. End with a checklist for an admin | +| Release plans return 403 | Use release policies. Create no plans | +| Repo not on the install list | Explain the GitHub fix by owner, then list again | +| GitHub still not connected after "Done" | Likely a different browser account. Follow [references/github-troubleshooting.md](references/github-troubleshooting.md) | + +More cases: [references/edge-cases.md](references/edge-cases.md). + +## What NOT to Do + +- Don't write before Step 7's "Looks good", or at all in Preview mode. +- Don't turn on auto-releasing unless the user picked it, and never send `approvalRequired`. +- Don't map unpicked repos, iterate over the install list, or remap a repo to another project. +- Don't act in the user's browser, change release policies, or report a browser step as verified. + +## References + +- [references/release-plans.md](references/release-plans.md): creating environment release plans +- [references/modes.md](references/modes.md): Preview and Resume details +- [references/byok.md](references/byok.md): bring-your-own-key setup +- [references/auto-cleanup.md](references/auto-cleanup.md) / [references/auto-archival.md](references/auto-archival.md): turning on flag cleanup and archival +- [references/github-troubleshooting.md](references/github-troubleshooting.md): GitHub still not connected after the user finished +- [references/edge-cases.md](references/edge-cases.md): edge cases and existing mappings +- [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md): settings rules, write confirmations, and PR diagnosis +- [Vega inference settings](https://launchdarkly.com/docs/home/getting-started/vega#vega-inference-settings) +- [Guarded rollouts](https://launchdarkly.com/docs/home/releases/guarded-rollouts) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json new file mode 100644 index 00000000..32071da7 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json @@ -0,0 +1,23 @@ +{ + "name": "launchdarkly-release-autopilot-onboarding", + "description": "Step-by-step Release Autopilot onboarding: pick capabilities, connect GitHub, map repos, choose inference, review release setup", + "version": "0.2.0-experimental", + "author": "LaunchDarkly", + "repository": "https://github.com/launchdarkly/ai-tooling", + "skills": ["./"], + "tags": [ + "launchdarkly", + "release-autopilot", + "onboarding", + "github", + "auto-flagging", + "auto-releasing", + "auto-cleanup", + "auto-archival", + "release-plans", + "mcp" + ], + "requirements": { + "mcp-servers": ["@launchdarkly/mcp-server"] + } +} diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-archival.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-archival.md new file mode 100644 index 00000000..43f8a6b4 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-archival.md @@ -0,0 +1,16 @@ +# Auto-archival + +Auto-archival archives flags that are ready to archive, after archival safety checks and a grace period. It is set per project, in the **Automatic archival** section of Lifecycle settings. No MCP tool manages it. The section is still rolling out, so some accounts do not see it yet. + +For each project, give the user `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). In the "Automatic archival" section they: + +1. Turn on **Enable automatic archival**, and confirm in the "Enable automatic archival?" dialog. +2. Set the grace period before a ready flag is archived. +3. If the section shows **Require approval before archiving**, choose whether archivals need approval. +4. Click **Save**. + +"View ready to archive flags" and "View auto-archived flags" open the flags list filtered to each set, so they can check what it will do and what it did. + +If the page has no "Automatic archival" section, the feature is not available for their account yet: record it as a follow-up, not as done. + +Record the result as reported by the user. It cannot be verified. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md new file mode 100644 index 00000000..1356ddaf --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/auto-cleanup.md @@ -0,0 +1,18 @@ +# Auto-cleanup + +Auto-cleanup runs Vega on a schedule to find flags that are ready for code removal and open pull requests that remove their code. It is set per project, in the **Automated flag cleanup** section of Lifecycle settings. On accounts where Vega has its own project settings, the section is on the Vega settings page (`/projects/{projectKey}/settings/vega`) instead. The MCP settings tools do not manage it yet. + +For each project, give the user `/projects/{projectKey}/settings/lifecycle` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). If that page has no "Automated flag cleanup" section, they use `/projects/{projectKey}/settings/vega`. In the "Automated flag cleanup" section they: + +1. Check **Enable automated flag cleanups**. +2. Choose a **Schedule frequency**: 1 day, 7 days (the default), or 30 days. +3. Set **Max cleanup jobs per run** (1–20). +4. Select the **GitHub repositories** to clean up. Suggest the repos mapped in this flow. At least one is required. +5. Optionally add **Additional context** for the cleanup agent, such as where flags are defined. Vega also follows the repo's `AGENTS.md`. +6. Click **Save**. + +Cleanup jobs run with the permissions of the member who saved the settings. If that member loses access, someone else must save the settings again. + +Record the result as reported by the user. It cannot be verified. + +Docs: [Automated flag cleanup](https://launchdarkly.com/docs/home/flags/manage/flag-cleanup-vega#automated-flag-cleanup) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md new file mode 100644 index 00000000..e43b6922 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/byok.md @@ -0,0 +1,11 @@ +# Bring your own key + +Release Autopilot's agents use an AI model to read pull requests. Bring your own key is optional: a project that does not set it up uses LaunchDarkly's model account (the product calls this "LaunchDarkly managed inference"). No MCP tool reads or writes inference settings, so this is a browser step. + +For each project, give the user `/projects/{projectKey}/settings/vega` on their LaunchDarkly instance (for example `https://app.launchdarkly.com`, or their EU or federal URL). On that page they: + +1. Select bring your own key. If the account has no Anthropic or OpenAI key yet, the page links to the integration form to add one. Keys are account-wide, so later projects can reuse them. +2. Keep or change the suggested default and small models. +3. Save. LaunchDarkly checks the key with the provider before it stores the settings, and shows the cause if the check fails. + +Record the result as reported by the user. It cannot be verified. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md new file mode 100644 index 00000000..47cae1d4 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -0,0 +1,28 @@ +# Edge cases + +## Edge cases + +| Situation | Action | +|-----------|--------| +| Every Release Autopilot tool returns 404 | The account does not have the Release Autopilot add-on enabled. Tell the user and stop | +| Member role is `reader` or `no_access` | Preview mode. End with the plan as a checklist for an admin | +| Member role is `writer` or a custom role | Account writes are marked "needs an admin" and left out of Step 8; repo writes still run | +| A write returns 403 | Report which write failed, list the remaining writes for an admin, and stop | +| `githubConnectUrl` with `flow=install` | Step 3. The user needs a GitHub org owner to install the App | +| Repo not on the install list | Same owner as listed repos: the user grants the App access to it. New owner: an org owner installs the App on that organization. Then list again | +| Mapped repo ID not on the install list | Mapped-repo reads return only repository IDs, so join them to the install list. An ID with no match is a repo the App can no longer see, or one this member cannot see. Show it by ID with its project. Do not change it | +| Repo already mapped to a different project | Do not remap here. The user unmaps it with the settings skill, then maps it in this flow | +| `list-release-plans` returns 403 `Plan does not allow this operation` | Use release policies. Show them and their warnings, create no plans, and keep auto-releasing off for the repo where a policy is missing or incomplete | +| Auto-releasing picked but no metrics exist (release plans) | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | +| Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | +| Generative AI or Vega eligibility error | Release Autopilot's agents need generative AI turned on, and no MCP tool reads that setting. An Admin or Owner selects **Enable generative AI use cases** in Organization settings > Security. Stop | +| Bring your own key chosen | Browser step only. Record it as reported, never as verified | +| No writes needed | Say so, list follow-ups, and stop | + +## Existing mappings + +| Current mapping | Action | +|-----------------|--------| +| Mapped to the chosen project, effective capabilities match the user's picks | Tell the user it is already set up. Write nothing | +| Mapped to the chosen project, a capability the user did not pick is effectively on | Record a repo override that turns it off, as its own line in Step 7 | +| Mapped to a different project | Do not remap. Remapping changes where its flags are created. The user unmaps it with [`launchdarkly-release-autopilot-settings`](../../launchdarkly-release-autopilot-settings/SKILL.md), then maps it in this flow | diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/github-troubleshooting.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/github-troubleshooting.md new file mode 100644 index 00000000..08a308bd --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/github-troubleshooting.md @@ -0,0 +1,12 @@ +# GitHub still not connected after "Done" + +The connect link acts on whatever LaunchDarkly session the browser has. This connection is signed in as the member from Step 1 (`get-member-self`). If the browser is signed in to a different LaunchDarkly account, or as a different member, the user installs or authorizes the App there, and this connection still sees nothing. + +Check the causes in this order. Stop at the first that applies. + +1. **Different LaunchDarkly account or member in the browser.** SKILL.md Step 3 asks about this first. An install made from the other account stays there; that account's admin can remove it if it was a mistake. +2. **Different LaunchDarkly instance.** The browser is on another instance (for example EU or federal) than this connection. Give the link with the instance this connection uses. +3. **Authorized with the wrong GitHub account** (`flow=authorize` remains). The GitHub account they authorized cannot see the organization's install. They reconnect with a GitHub account that is a member of that organization. +4. **Installed on a different GitHub organization.** The App is now installed, but not on the organization that owns their repo. See the repo-owner check in Step 4. + +If none applies, say what you checked and stop. Do not retry the link more than once without a change. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md new file mode 100644 index 00000000..e15b50ef --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md @@ -0,0 +1,18 @@ +# Preview and Resume + +## Preview + +- Run Steps 1–7 with live reads, and make no writes. +- In Step 1, leave out the "Change one setting" option, because that hand-off writes. +- In Step 7, show the plan as a checklist under "Changes an admin would make", one line per change, and stop without asking for confirmation. +- A member whose role is `reader` or `no_access` is always in Preview. + +## Resume + +- Keep the earlier mode: a resumed Preview stays Preview. +- Skip Step 0 entirely. Re-read live state in Step 1, and say in one sentence what is already done. +- Use the earlier picks if they are still in the conversation. If not, ask them again with live state as the default. +- Writes that landed before a Step 8 failure show as already set up, so the new plan lists only what is left. +- A restated pick is an answer, never approval. Resume still stops at Step 7 and waits for "Looks good". + +Every mode is safe to run again: a re-run reads live state and proposes only the writes that are still needed. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md new file mode 100644 index 00000000..51bee1bc --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md @@ -0,0 +1,21 @@ +# Create an environment release plan + +Use this from Step 6 of the onboarding skill when the account has environment release plans. A release plan is a reusable recipe for one environment, made of phases, rollout steps, and metrics. A plan with metrics makes a release guarded, so a regression on a metric can roll it back. A plan without metrics is progressive: say so. + +Follow the `create-release-plan` tool guidance. Ask each setting as its own question, never bundled: + +1. **Randomization unit**: the context kind to roll out by, such as `user` or `account`. +2. **Rollout steps**: header "Rollout", with options such as "1% → 10% → 50% → 100%, 4h holds" or "10% → 100%, 24h hold". The user can type another schedule in Other. +3. **Advancement**: header "Advance", options "Automatic", description "Move to the next step when the hold ends and metrics are healthy", and "Manual", description "A person moves each step". +4. **Metrics**: call `list-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create. Step 8 creates them in LaunchDarkly before the plans that use them. This flow creates the metric only: if the app does not send the event yet, list instrumenting it as a follow-up, and do not edit code or `.env` files. +5. **Cleanup**: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. + +Every phase needs at least one clause. For a phase that targets everyone, use: + +```json +{ "attribute": "kind", "op": "in", "values": [""], "contextKind": "" } +``` + +`list-release-plans` hides incomplete plans that were migrated from release policies. If the user expects a plan that is missing, call it again with `includeIncomplete: true`. An incomplete plan cannot start a release, so that environment needs a new complete plan. + +Always list before creating, so the new key does not collide with an active plan. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md index ddfb6b47..97f6b624 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/README.md @@ -19,7 +19,7 @@ This skill teaches agents how to: The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot settings tools (`get-release-autopilot-settings`, `list-release-autopilot-github-repos`, `update-release-autopilot-repo-settings`, and siblings). If `list-release-autopilot-github-repos` returns `githubConnectUrl`, give the user that link. This skill does not complete GitHub OAuth itself. -Release Autopilot settings are in alpha. The endpoints behind these tools are gated by an alpha feature flag, so accounts outside the alpha get 404s. +The endpoints behind these tools require the Release Autopilot add-on, so accounts without it enabled get 404s. ## Usage @@ -37,6 +37,7 @@ Which GitHub repos are mapped for Release Autopilot, and which can I map? ## Related +- [LaunchDarkly Release Autopilot Onboarding](../launchdarkly-release-autopilot-onboarding/): step-by-step first-time setup - [LaunchDarkly Flag Create](../../feature-flags/launchdarkly-flag-create/): Create flags after Release Autopilot is mapped - [LaunchDarkly MCP Server](https://github.com/launchdarkly/mcp-server) - [LaunchDarkly Docs](https://docs.launchdarkly.com) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md index 5c9c8ab9..b7fd8566 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md @@ -1,11 +1,11 @@ --- name: launchdarkly-release-autopilot-settings -description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR." +description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, review release policies), use launchdarkly-release-autopilot-onboarding instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly - version: "1.3.0-experimental" + version: "1.4.0-experimental" --- # LaunchDarkly Release Autopilot Settings @@ -24,7 +24,7 @@ This skill requires the remotely hosted LaunchDarkly MCP server. If these tools are missing, stop. Do not invent REST calls, numeric GitHub ids, or "fixes" by guessing settings. -Release Autopilot settings are in alpha: the underlying endpoints are gated by an alpha feature flag. If every Release Autopilot tool returns 404, the account is not in the alpha — tell the user rather than retrying or falling back to REST. +The underlying endpoints require the Release Autopilot add-on. If every Release Autopilot tool returns 404, the account does not have the Release Autopilot add-on enabled — tell the user "Your account does not have the Release Autopilot add-on enabled," rather than retrying or falling back to REST. ## Core Principles @@ -73,12 +73,15 @@ After they confirm, apply **only** the fields in the proposal. Then verify (do n ## Choose a workflow -- **"Why didn't Release Autopilot classify / auto-flag my PR?"** (or similar) → [Why didn't my PR get classified?](#why-didnt-my-pr-get-classified) first. Do not write settings in that workflow. +Route on what the user asked for, not on account state. Check these in order: + +- **"Why didn't Release Autopilot classify / auto-flag my PR?"** (or similar) → [Why didn't my PR get classified?](#why-didnt-my-pr-get-classified), even on an account with nothing set up. Do not write settings in that workflow. +- **First-time setup** ("set up Release Autopilot", "onboard us", "get started") → [`launchdarkly-release-autopilot-onboarding`](../launchdarkly-release-autopilot-onboarding/SKILL.md). It gathers every decision first and confirms one plan. - Configure / map / turn on / unmap → [Configure settings](#configure-settings). ## Why didn't my PR get classified? -Use this when the user asks why a PR was not classified, not auto-flagged, or Release Autopilot ignored the PR. **Read-only.** Check in this order. Stop at the first failure and tell them how to fix it (then [Confirm before write](#confirm-before-write) if they ask you to apply the fix). +Use this when the user asks why a PR was not classified, not auto-flagged, or Release Autopilot ignored the PR. **Read-only.** Check in this order. Stop at the first failure and tell them how to fix it (then [Confirm before write](#confirm-before-write) if they ask you to apply the fix). If the answer is that auto-flagging is off for the account or the repo is not mapped, and nothing is set up yet, offer [`launchdarkly-release-autopilot-onboarding`](../launchdarkly-release-autopilot-onboarding/SKILL.md) as the way to set it up. Identify the GitHub repo (`owner/name`) from the PR URL, `git remote`, or the name they gave. If you cannot identify one repo, ask. Do not diagnose a different repo. @@ -93,7 +96,7 @@ Identify the GitHub repo (`owner/name`) from the PR URL, `git remote`, or the na If all three look fine (account on, repo mapped, repo auto-flagging effective on): - Do **not** flip settings to "try something." -- Point them at the Release Autopilot runbooks. These are internal LaunchDarkly Confluence pages (PD space) while Release Autopilot is in alpha; there is no public docs URL yet. Classification can still fail for reasons this MCP surface cannot see (GitHub App install on the wrong org, PR not in an installed repo, workflow/app permissions, classifier skip rules). +- Point them at the Release Autopilot runbooks. These are internal LaunchDarkly Confluence pages (PD space); there is no public docs URL yet. Classification can still fail for reasons this MCP surface cannot see (GitHub App install on the wrong org, PR not in an installed repo, workflow/app permissions, classifier skip rules). - [Flag Classification (ODD) Runbook](https://launchdarkly.atlassian.net/wiki/spaces/PD/pages/5377097848/WIP+Flag+Classification+ODD+Runbook) — the classifier itself: verdicts (`not-suited`, `already-flagged`), escalations, and the per-PR ledger that records why a PR got the verdict it did. - [GitHub App Runbook](https://launchdarkly.atlassian.net/wiki/spaces/PD/pages/5157388373/GitHub+App+Runbook) — the PR event never reaching Release Autopilot: install scope, permissions, webhooks. - Optional reads: `list-release-autopilot-github-repos` to confirm the repo is on the install list; `autoFlagging.approvalRequired` if they expected a PR and one exists but is waiting on approval. @@ -112,7 +115,7 @@ If `githubConnectUrl` is present, give the user that link and stop this turn. Do If an account PATCH is needed, follow [Confirm before write](#confirm-before-write), then `update-release-autopilot-settings` with **only** the confirmed fields: -- `autoFlagging.enabled` / `autoFlagging.approvalRequired` (approval is only valid here) +- `autoFlagging.enabled` / `autoFlagging.approvalRequired` (approval is only valid here). The agent backend does not read `approvalRequired` today: every auto-flagging proposal waits for `@launchdarkly approve` either way. If a user asks to turn approval off, tell them it currently has no effect before proposing the change. - `autoReleasing.enabled` Do not send `autoCleanup`. Requires permission to update account Release Autopilot settings. @@ -139,6 +142,6 @@ To unmap: [Confirm before write](#confirm-before-write), then `delete-release-au - Auto-cleanup - Completing GitHub OAuth inside the tool. Hand the user `githubConnectUrl` instead. -- Vega BYOK +- Vega inference and bring-your-own-key setup, and environment release plans. The [onboarding skill](../launchdarkly-release-autopilot-onboarding/SKILL.md) guides both. - Observability MCP (`github_repositories`) — do not require a second MCP server just to map a repo - Changing classification rules, GitHub App permissions, or org install scope via this skill diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json index 818cb907..af89cebf 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json @@ -1,7 +1,7 @@ { "name": "launchdarkly-release-autopilot-settings", "description": "Configure LaunchDarkly Release Autopilot settings via MCP, or diagnose why a PR was not classified", - "version": "1.3.0-experimental", + "version": "1.4.0-experimental", "author": "LaunchDarkly", "repository": "https://github.com/launchdarkly/ai-tooling", "skills": ["./"],