Repository navigation
Conversation
… and mapping, clear opt-out overrides Follow-ups from review on #229: - Bring your own key, auto-cleanup, and auto-archival change settings in the browser, so they now happen only after 'Looks good'; Preview lists them and never asks for them. - GitHub runs only for auto-flagging, auto-releasing, or auto-cleanup, and repo mapping only for auto-flagging or auto-releasing; auto-archival alone picks projects directly. - A picked capability that a repo override keeps off is turned back on, and verification checks the effective value. - Evals: mappings must name the exact project, the policy case must take the 403-then-policies path, and plan mode must write the repo mapping. New cases for archival-only, preview with cleanup, and the repo override. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
review Step 8 now sends enabled:true for a picked capability a repo override kept off (unless it cannot work yet), and Step 7 describes that write. Preview with GitHub disconnected lists mapping as a later step instead of contradicting the no-mapping-without-GitHub rule. Resume re-lists pending browser steps after the confirmation. Evals detect browser-step and repo-picker questions by content, not only by link, require the override case to read the repo back, and add a bring-your-own-key case (24 total). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
| !/github/i.test(String((t.arguments && t.arguments.prompt) || "")); | ||
|
|
||
| // Browser steps that change settings come only after "Looks good". | ||
| module.exports.browserStepsAfterConfirm = (output) => { |
There was a problem hiding this comment.
blocking: a rejected confirmation still counts as approval.
this treats showing the confirmation question as approval, regardless of which option the user selected. In the archival-only case there is no MCP write, so no other assertion checks the answer; a trajectory that selects “Change something” and then asks the user to change Lifecycle settings still passes. Could we require the selected option to be “Looks good” before the first browser step?
|
|
||
| // Policy mode: exactly one release-plans lookup (the 403 is account-wide) and | ||
| // at least one release-policies lookup. | ||
| module.exports.policyLookupPath = (output) => { |
There was a problem hiding this comment.
blocking: the policy lookup can target the wrong project
this proves that a policy lookup happened, but not that it queried the requested project. The policy-mode case is for default, and a list-release-policies call with no projectKey or with another project still passes. Could we require the expected projectKey here (and validate the plans lookup arguments too) so the eval protects the actual release setup?
There was a problem hiding this comment.
also is it possible to future proof for environment release plans?
|
|
||
| Skip this step without mentioning GitHub if Step 1 returned `installed: true`, `userAuthorized: true`, and no `githubConnectUrl`. | ||
|
|
||
| **Preview mode:** do not ask the user to install or authorize. List "Connect GitHub" in the Step 7 checklist with the link. Repo mapping cannot be planned until GitHub is connected, so skip Step 4 and any release setup that depends on it; list "After GitHub is connected: map the repos" in the checklist, and continue with the steps that do not need GitHub. |
There was a problem hiding this comment.
blocking: Preview skips required auto-release setup when GitHub is disconnected
when GitHub is disconnected, Preview skips release setup but only adds a future repo-mapping item to the checklist. For auto-releasing, the admin still needs to choose environments and validate or create their release policies/plans before enabling it, so the promised checklist is incomplete. Could we add that deferred release-setup item after mapping when auto-releasing is selected?
- Browser steps count as approved only after a 'Looks good' answer, not after the confirmation question is merely shown. - Release setup lookups must name the requested project and one of its environments; the check accepts either path, release policies (403 then a policy lookup for the project) or environment release plans, so it keeps working at GA. - Preview with GitHub disconnected and auto-releasing picked also lists the deferred release setup (environments and release policies or plans) after repo mapping. New eval case (25 total). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
|
||
| // Index of the last confirm question before `limit` whose answer was "Looks good", | ||
| // or -1. Showing the confirm question is not approval; only that answer is. | ||
| function approvedConfirmBefore(steps, limit) { |
There was a problem hiding this comment.
approvedConfirmBefore walks backward until it finds any “Looks good.” It does not stop at the latest confirmation. An older approval followed by “Change something,” then an automatic-archival browser question, still returns pass: true. The newer answer revoked approval, so the eval can stay green while the agent continues a production-setting step.
Inspect the last confirmation before the browser step and require that selected option to be “Looks good.”
| const steps = traj(output); | ||
| const plans = steps.filter((t) => t.tool === "list-release-plans"); | ||
| if (!plans.length) return result(false, "never checked release plans"); | ||
| const badPlan = plans.find((t) => (t.arguments || {}).projectKey !== projectKey || !envs.includes((t.arguments || {}).environmentKey)); |
There was a problem hiding this comment.
The successful release-plan path accepts any lookup whose environment is test or production. It does not require the environment selected in the case. The plan-mode case selects Production, but a lookup for Test returns pass: true. A multi-environment run also passes after checking only one environment.
Give each assertion the selected environment set and require one successful plan lookup for every selected environment. Keep the one-call account-wide behavior only for the 403 policy path.
Summary
Follow-ups to the review on #229.
Skill (
launchdarkly-release-autopilot-onboarding0.3.0-experimental)enabled: true), unless it cannot work yet, and Step 9 checks the effective value before calling the setup complete.Evals (
evals/release-autopilot-onboarding, 25 cases)projectKey.acme/demo→defaultmapping after the release plan.Testing
python3 scripts/validate_skills.pyandtests/test_*.pypasscd evals && npm test: 125 passnpm run eval:release-autopilot-onboarding: 25/25 pass; the new and changed cases passed three runs each🤖 Generated with Claude Code
via LD Research 🤖