feat(release-autopilot): add onboarding wizard skill and eval suite

.claude-plugin/marketplace.json

@@ -11,7 +11,7 @@
       "name": "launchdarkly",
       "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics",
       "source": ".",
-      "version": "1.0.2",
+      "version": "1.0.3",
       "author": {
         "name": "LaunchDarkly",
         "email": "support@launchdarkly.com"

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "launchdarkly",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics",
   "author": {
     "name": "LaunchDarkly",
@@ -36,6 +36,7 @@
     "./skills/agentcontrol/online-evals",
     "./skills/agentcontrol/projects",
     "./skills/agentcontrol/tools",
-    "./skills/release-autopilot/launchdarkly-release-autopilot-settings"
+    "./skills/release-autopilot/launchdarkly-release-autopilot-settings",
+    "./skills/release-autopilot/launchdarkly-release-autopilot-onboarding"
   ]
 }

.cursor-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "launchdarkly",
-  "version": "1.0.3",
+  "version": "1.0.4",
   "description": "LaunchDarkly agent skills and mcp server for feature flag management, AgentControl, and skill authoring",
   "author": {
     "name": "LaunchDarkly",

CHANGELOG.md

@@ -4,6 +4,9 @@ All notable changes to this repository will be documented in this file.
 
 ## Unreleased
 
+- Add `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.2.0-experimental: a step-by-step wizard with setup, preview, and resume modes that asks which capabilities to set up (auto-flagging, auto-releasing, auto-cleanup, auto-archival), hands off the GitHub App install or authorize link, maps chosen repos, guides managed or bring-your-own-key Vega inference, and reviews or creates environment release plans for auto-releasing, or uses the project's release policies where plans are not available. Release policies are required for auto-releasing today; the skill will be updated to use environment release plans by GA, when plans replace policies. One confirmed plan lists every write before any change. Ships in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4.
+- `release-autopilot/launchdarkly-release-autopilot-settings` 1.4.0-experimental: route first-time setup to the onboarding skill.
+- evals: add the `release-autopilot-onboarding` suite (20 cases: confirm before writing, repo overrides, policy and plan modes, reader and writer roles, preview, resume, GitHub install and authorize paths, deprecated MCP, a failed write, auto-cleanup, auto-archival, a browser signed in to a different account, an account without the add-on, a diagnosis on an empty account). The suite keeps its tool definitions and mocks in its own folder (`tool_definitions`, `tool_mocks`). The provider gains `mock_ask_question_match` with label matching, stateful Release Autopilot mocks, and `CLAUDE_CODE_OAUTH_TOKEN` as an alternative to `ANTHROPIC_API_KEY`.
 - `onboarding/mcp-configure` 0.1.1: Claude Code loads MCP servers only at session start, so a server added mid-session needs the user to quit, resume with `claude --continue`, and authenticate from `/mcp`. The skill no longer says Claude Code needs no restart or that its OAuth prompt appears on the first tool call.
 - `experiments/launchdarkly-experiment-setup`: require the flag and variations, hypothesis, and primary metric from the user before creating anything; if the named flag does not exist, offer three choices in one message: correct the key, listing similar flags; create it themselves and then continue; or have the agent create it. For the last, propose the key, name, type, and variations, say it will be created off, follow `launchdarkly-flag-create` when that skill is installed, and wait for an explicit yes before `create-flag`. Never create a flag unless they pick that option and approve the details; add flag variations only when the user explicitly asks, and otherwise say which variations exist and ask; suggest a control from what the flag serves today, or the off variation when it is off, and ask instead when the flag is brand new (created in this conversation, or status `new`: created fewer than seven days ago and never evaluated), the targeted rule is a percentage rollout, or the flag is off and that environment is `inactive` (no one is currently being served the off variation); suggest `user` as the randomization unit only when `search-contexts` confirms it is the only context kind in use, and otherwise ask; label every suggestion with a one-line reason and let the user change it before approval; before showing a design, gather every missing piece in one message, and skip that message when nothing is missing; include a new metric's kind, basis, and measure when the user's wording has not already made them clear; show the design only when nothing is blank, then stop until the user approves it; create the metric and experiment only after the user approves a complete design they have seen; if they change anything after seeing it, show the updated design and ask once more; call `list-metric-events` only after a missing primary metric is known to be a custom event and the user has named its event key; if that event is not reaching LaunchDarkly, warn in the design, list at most 10 arriving custom event keys (preferring ones similar to the key the user gave), and let the user correct the key or approve as is; create the metric and experiment on approval even if it has not arrived; right before starting, call `list-metric-events` again for a custom-event primary metric and, if it is still not arriving, ask whether to start anyway, in the same message as the flag-off question when both apply. A metric name alone is not that key. Do not skip the flow check or block creation before the event is identified; start an iteration only after the user says to start. Omit `methodology` unless the user chooses one; the design shows statistics as LaunchDarkly default (frequentist, sequential testing) unless they chose otherwise, and a Bayesian choice is sent as `methodology` `"bayesian"`. When stopping, report significance for a frequentist experiment and probability to beat baseline or probability to be best for a Bayesian one (0.3.0).
 - `onboarding` 0.4.0: add `references/sdk/observability-plugin.md` and load it from the SDK install when the user asks for observability, session replay, error monitoring, logs, or tracing. Covers browser (React, JavaScript, Vue), Next.js, Node.js, React Native, Python, Ruby, Go, .NET, iOS, and Android. The plugin goes on the existing client's `plugins` option and reuses its key. `recipes.md` points to the new reference instead of its per-SDK one-liners. The request is honored at any point, including a follow-up after onboarding finished; session replay defaults to strict privacy; an SDK too old for `plugins` gets a question before any major upgrade. Adds three onboarding evals for these paths.

README.md

@@ -28,6 +28,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor
 
 | Skill | Description |
 |-------|-------------|
+| `release-autopilot/launchdarkly-release-autopilot-onboarding` | Step-by-step setup: pick capabilities, connect the GitHub App, map repos, choose inference, review release setup |
 | `release-autopilot/launchdarkly-release-autopilot-settings` | Turn on GitHub App auto-flagging / auto-releasing, map `owner/name` repos, or diagnose why a PR was not classified (confirm before every write) |
 
 ### AgentControl

evals/.env.example

@@ -6,6 +6,11 @@
 # Request access via #ask-app-eng to get an API key.
 ANTHROPIC_API_KEY=
 
+# ALTERNATIVE to ANTHROPIC_API_KEY for the agent: a Claude subscription token.
+# Run `claude setup-token` in your own terminal and paste the token here.
+# llm-rubric assertions still need RUBRIC_MODEL to reach a grader.
+# CLAUDE_CODE_OAUTH_TOKEN=
+
 # OPTIONAL: the system-under-test model. Stays on Claude because that's what
 # real users will run skills against. Defaults to claude-sonnet-4-6 if unset.
 # AGENT_MODEL=claude-sonnet-4-6

evals/README.md

@@ -35,6 +35,7 @@ npm run eval:configs-variations       # agentcontrol/configs-variations
 npm run eval:flag-create             # feature-flags/launchdarkly-flag-create
 npm run eval:onboarding              # onboarding
 npm run eval:experiment-orchestration # aws-devops-agent/experiment-orchestration
+npm run eval:release-autopilot-onboarding  # release-autopilot/launchdarkly-release-autopilot-onboarding
 
 # Quick smoke check — first test case only (~15-20s, ~$0.05)
 npm run eval:configs-create:single
@@ -44,6 +45,7 @@ npm run eval:configs-variations:single
 npm run eval:flag-create:single
 npm run eval:onboarding:single
 npm run eval:experiment-orchestration:single
+npm run eval:release-autopilot-onboarding:single
 
 # Aggregate and CI operations
 npm run eval:all                     # Run every suite and rebuild ../eval-scores.json

evals/package.json

@@ -25,6 +25,8 @@
     "eval:flag-and-release-change:single": "promptfoo eval -c shared/defaults.yaml -c flag-and-release-change/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1",
     "eval:onboarding": "promptfoo eval -c shared/defaults.yaml -c onboarding/promptfooconfig.yaml --env-file .env --no-cache -o onboarding/results.json",
     "eval:onboarding:single": "promptfoo eval -c shared/defaults.yaml -c onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1",
+    "eval:release-autopilot-onboarding": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache -o release-autopilot-onboarding/results.json",
+    "eval:release-autopilot-onboarding:single": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1",
     "eval:experiment-guide": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache -o experiment-guide/results.json",
     "eval:experiment-guide:single": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1",
     "eval:experiment-orchestration": "promptfoo eval -c shared/defaults.yaml -c experiment-orchestration/promptfooconfig.yaml --env-file .env --no-cache -o experiment-orchestration/results.json",

evals/providers/_mock.js

@@ -126,6 +126,9 @@ function createMockState() {
   return {
     configs: {},      // configKey -> config object
     flags: {},        // flagKey -> flag object
+    raSettings: null, // Release Autopilot account settings written this run
+    raRepos: {},      // owner/name -> Release Autopilot repo settings written this run
+    raPlans: {},      // environmentKey -> release plans created this run
   };
 }
 
@@ -134,9 +137,57 @@ function createMockState() {
  * `state` is the per-invocation map from `createMockState()`.
  * `templates` is the full parsed mocks/tool-responses.json.
  */
+function mergeDeep(base, patch) {
+  const out = { ...(base || {}) };
+  for (const [k, v] of Object.entries(patch || {})) {
+    out[k] = v && typeof v === "object" && !Array.isArray(v) ? mergeDeep(out[k], v) : v;
+  }
+  return out;
+}
+
+// Release Autopilot settings and release plans: writes are recorded so that
+// later reads (the skill's verify step) reflect them instead of the template.
+function renderReleaseAutopilot(template, input, toolName, state, replacements) {
+  const args = input || {};
+  if (toolName === "update-release-autopilot-settings") {
+    state.raSettings = mergeDeep(state.raSettings, args);
+    return { ...state.raSettings };
+  }
+  if (toolName === "get-release-autopilot-settings" && state.raSettings) {
+    return mergeDeep(walk(template, replacements), state.raSettings);
+  }
+  if (toolName === "update-release-autopilot-repo-settings" && args.repo) {
+    const { repo, githubRepoId, ...rest } = args;
+    state.raRepos[repo] = mergeDeep(state.raRepos[repo] || { repo }, rest);
+    return { ...state.raRepos[repo] };
+  }
+  if (toolName === "get-release-autopilot-repo-settings" && args.repo && state.raRepos[args.repo]) {
+    return { ...state.raRepos[args.repo] };
+  }
+  if (toolName === "create-release-plan" && args.environmentKey) {
+    const plan = { key: args.key, name: args.name, environmentKey: args.environmentKey, progressiveRelease: args.progressiveRelease };
+    (state.raPlans[args.environmentKey] = state.raPlans[args.environmentKey] || []).push(plan);
+    return plan;
+  }
+  if (toolName === "list-release-plans" && state.raPlans[args.environmentKey]) {
+    const items = state.raPlans[args.environmentKey];
+    return { items, totalCount: items.length };
+  }
+  return undefined;
+}
+
 function renderMockResponse(template, input, toolName, state) {
   const replacements = buildReplacements(input);
 
+  // An error template on a write (for example a 403 override) wins over
+  // stateful handling, so the failed write is not recorded. Reads still use
+  // state: the unmapped-repo read template is itself a 404 error.
+  const failedWrite = /^(update|create|delete)-/.test(toolName) && template && template.error;
+  if (state && state.raRepos && !failedWrite) {
+    const ra = renderReleaseAutopilot(template, input, toolName, state, replacements);
+    if (ra !== undefined) return ra;
+  }
+
   // ---------- stateful reads ----------
   if (toolName === "get-ai-config") {
     const key = input.configKey || input.key;

evals/providers/claude-skill-agent-sdk.js

@@ -67,6 +67,15 @@
  *                                 successive `ask-question` calls; an empty
  *                                 array is a dismissed question (no selection).
  *                                 Calls past the end get the first option.
+ *   mock_ask_question_match     - when true, each `ask-question` call takes the first
+ *                                 unused scripted answer whose entries all match its
+ *                                 options (by id or label), instead of the next one in
+ *                                 order. Entries match an option by id, or else by
+ *                                 label (case-insensitive substring), so tests can
+ *                                 steer by label when the agent picks the ids. Use
+ *                                 for flows that may skip questions. A call with no match gets
+ *                                 its first option and is recorded with
+ *                                 `scripted: false`
  *   mock_files                  - optional object mapping relative file paths to
  *                                 string content, written into the isolated cwd
  *                                 before the agent runs
@@ -80,6 +89,8 @@
  * Environment variables:
  *   AGENT_MODEL          - SUT model (default claude-sonnet-4-6).
  *   ANTHROPIC_API_KEY    - Auth for the SDK's child Claude Code process.
+ *   CLAUDE_CODE_OAUTH_TOKEN - Alternative auth from `claude setup-token`, for
+ *                          running evals on a Claude subscription without an API key.
  *   SKILL_EVAL_DEBUG=1   - Dump every SDK message to a temp file for inspection.
  */
 
@@ -107,6 +118,45 @@ const toolDefs = JSON.parse(
   fs.readFileSync(path.join(__dirname, "../tools/definitions.json"), "utf-8"),
 );
 
+/**
+ * Resolve a scripted answer to an option id. Exact id wins; otherwise the
+ * first option whose label contains the answer (case-insensitive). Unmatched
+ * answers pass through unchanged.
+ */
+function matchOption(want, options) {
+  const list = Array.isArray(options) ? options : [];
+  if (list.some((o) => o.id === want)) return want;
+  const needle = String(want).toLowerCase();
+  const hit = list.find((o) => String(o.label || "").toLowerCase().includes(needle));
+  return hit ? hit.id : want;
+}
+
+/**
+ * Choose the scripted answer for the `idx`-th ask-question call. In order
+ * mode, take answers[idx] as written. In match mode, take the first unused
+ * answer whose entries all resolve to the call's options (by id or label),
+ * mark it used, and return the matched ids. Returns undefined when no
+ * scripted answer applies.
+ */
+function pickAnswer(answers, idx, options, matchMode, used) {
+  const resolves = (ans) =>
+    Array.isArray(ans) &&
+    ans.length > 0 &&
+    ans.every((want) => (options || []).some((o) => o.id === matchOption(want, options)));
+  let chosen;
+  if (matchMode) {
+    const hit = answers.findIndex((ans, i) => !used.has(i) && resolves(ans));
+    if (hit === -1) return undefined;
+    used.add(hit);
+    chosen = answers[hit];
+    return chosen.map((want) => matchOption(want, options));
+  }
+  // Order mode is unchanged: the scripted answer is returned as written,
+  // and an explicit [] is a dismissed question (no selection).
+  chosen = answers[idx];
+  return Array.isArray(chosen) ? chosen : undefined;
+}
+
 const mockTemplates = JSON.parse(
   fs.readFileSync(path.join(__dirname, "../mocks/tool-responses.json"), "utf-8"),
 );
@@ -286,9 +336,14 @@ class ClaudeSkillAgentSdk {
   }
 
   async callApi(_prompt, context) {
-    const apiKey = process.env.ANTHROPIC_API_KEY;
-    if (!apiKey) {
-      return { error: "ANTHROPIC_API_KEY environment variable is not set" };
+    // Either an API key or a Claude subscription token (`claude setup-token`)
+    // authenticates the SDK's child Claude Code process. The isolated config
+    // dir hides the interactive /login, so one of these must be in the env.
+    if (!process.env.ANTHROPIC_API_KEY && !process.env.CLAUDE_CODE_OAUTH_TOKEN) {
+      return {
+        error:
+          "Set ANTHROPIC_API_KEY, or CLAUDE_CODE_OAUTH_TOKEN from `claude setup-token`",
+      };
     }
 
     const userRequest =
@@ -313,6 +368,8 @@ class ClaudeSkillAgentSdk {
       : [];
     const mockFiles = context?.vars?.mock_files || {};
     const mockOverrides = context?.vars?.mock_overrides || {};
+    const matchAnswers = context?.vars?.mock_ask_question_match === true;
+    const usedAnswers = new Set();
 
     let userMessage = userRequest;
     if (codebaseContext) {
@@ -349,7 +406,11 @@ class ClaudeSkillAgentSdk {
               const template = this.mockTemplates[def.name];
               let mock;
               if (override !== undefined) {
-                mock = renderOverride(override, args);
+                // Release Autopilot tools keep their recorded state under an
+                // override (an error override on a write is not recorded).
+                mock = /release-autopilot|release-plan/.test(def.name)
+                  ? renderMockResponse(override, args, def.name, mockState)
+                  : renderOverride(override, args);
               } else if (template) {
                 mock = renderMockResponse(template, args, def.name, mockState);
               } else {
@@ -399,11 +460,11 @@ class ClaudeSkillAgentSdk {
             },
             async (args) => {
               const idx = askQuestionCallCount++;
-              const override = askQuestionAnswers[idx];
+              const picked = pickAnswer(askQuestionAnswers, idx, args.options, matchAnswers, usedAnswers);
+              const scripted = picked !== undefined;
               let selected;
-              // An explicit [] is a dismissed question: no selection.
-              if (Array.isArray(override)) {
-                selected = override;
+              if (scripted) {
+                selected = picked;
               } else if (Array.isArray(args.options) && args.options.length > 0) {
                 selected = [args.options[0].id];
               } else {
@@ -415,6 +476,8 @@ class ClaudeSkillAgentSdk {
                 tool: "ask-question",
                 arguments: args,
                 turn: currentTurn,
+                scripted,
+                selected,
                 mock_response_preview: previewMock(response),
               });
 
@@ -684,3 +747,5 @@ module.exports.filterToolDefs = filterToolDefs;
 module.exports.mergeToolDefs = mergeToolDefs;
 module.exports.skillInstructions = skillInstructions;
 module.exports.overrideFor = overrideFor;
+module.exports.matchOption = matchOption;
+module.exports.pickAnswer = pickAnswer;