Repository navigation
feat(aws-devops-agent): add the experiment-orchestration skill and agent prompt - #230
Merged
Merged
Conversation
…ent prompt Add a new AWS DevOps Agent category. It holds the experiment-orchestration skill and the system prompt of the experiment-orchestrator custom agent. ldcli aws-devops-agent setup installs both in an agent space. The files are the same as the copies that ldcli embeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Andrewjeska
approved these changes
Oct 28, 2026
Andrewjeska
left a comment
Andrewjeska
left a comment
There was a problem hiding this comment.
I trust that this is tested with the agent, and you ran claude with adversarial mode to check the skill is written well. Because that would be what I would have done for review anyways.
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
ari-launchdarkly
requested changes
Oct 28, 2026
ari-launchdarkly
left a comment
ari-launchdarkly
left a comment
Contributor
There was a problem hiding this comment.
This is really cool. A couple things I'd love clarity on are:
- What happens if the LDCLI isn't installed?
- Can we really have an experiment and a guarded rollout at the same time?
- What's the deployment step about?
- There are a lot of dependent assertions here that we don't have stability on, could we add some evals to make sure that this works across different models and agents?
…etadata The description promised a coding agent and GitHub Actions, which an agent space has only when GitHub is connected, and led with guarded rollouts, which not every plan has. Add license, compatibility, and version like the other skills. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the skill - Check for GitHub access before Step 2; without it, or with read-only access, hand the code change, merge, and deploy to the user. - Step 4 uses the AWS DevOps Agent release readiness review and never skips it. A person merges, since the GitHub integration documents no merge action. - Approvals: call the change tool, and create an approval request only when it returns requiresApproval. - Guarded rollout: set regressionThreshold, rolloutWeight units, and updateDefaultRuleOnCompletion; keep stages below 50%. - Add a regular-rollout fallback for plans without guarded rollouts. - Stopping an iteration serves the winner to the experiment's rule, not always everyone. - Unattended runs stop at the first write, since every MCP write needs approval. - Remove wording from another harness in Step 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Execution mode lists the planned handoffs to the user, not only failures. - Without GitHub, the user's review counts as the review, and the deploy step hands off too. - Ask for regression thresholds in Step 1 so the rollout doesn't stall. - Regular rollout: look up the true/false variation indexes, and watch in a session where someone can approve the revert. - Remove leftover "auto-promotes to 100%" wording and label the retry section as guarded-rollout only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Drop "mandatory, not optional" so no other step reads as optional. - Address the coding-agent handoff to the agent, not the operator. - Open Step 5 with why deployment matters before the experiment. - State that the experiment and the rollout never run at the same time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Link the ldcli install instructions, and point to the manual install when ldcli isn't installed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… samples Step 7 checks results repeatedly, which inflates false positives in a fixed-horizon frequentist test. Create the experiment with sequential testing or the Bayesian methodology, and require at least 100 exposures per variation before acting on a significant result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
A "50/50 split allocated to 10%" maps to no create-experiment field, and an agent sending 50 and 50 runs the experiment on all traffic. Treatment allocationPercent is a share of the rule's traffic (WeightsForFlag in gonfalon converts it directly), so 10% split 50/50 is 5 + 5, with the rest served notInExperimentVariationId. - Run the experiment on the fallthrough, the rule both rollouts act on. - Name every required create-experiment field: hypothesis, treatment parameters, metrics, primarySingleMetricKey. - Drop the separate manual split; starting the iteration writes it. - Drop the Bayesian option: Step 7 decides on significance. - Stopping takes the winning treatment's _id, not a variation ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inside a numbered list, "Step 6" reads like an item of the same list. Name
the step ("Step 6: Experiment Phase"), and call references to the list's
own items "item N above".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skill required an error-rate guardrail but never found or created one. - Step 1 picks the randomization unit and finds the success and error-rate metrics, preferring LaunchDarkly's built-in telemetry error-rate metric. Guarded rollouts reject metrics without the rollout's unit. - Step 3 creates a missing metric (error rate as occurrence, the share of contexts with an error, like the built-in metric) or adds the track() call for a silent one, rather than duplicating it. - Step 6 smoke-tests the new instrumentation with a capped wait, and doesn't wait for error or treatment-only events that can't arrive yet. - Rename "Safety Rules (the non-negotiables)" and stop hardcoding error-rate in the report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first change to a flag or experiment in an environment calls the tool directly. If LaunchDarkly says it needs approval, the agent says the change wasn't made, asks who should review, and submits an approval request. Later changes to that entity go straight to an approval request, so the user doesn't approve a write that LaunchDarkly will refuse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `mock_overrides` test var: replace a tool's response for one test, as an object, a per-call sequence, or by an argument's value (`by_arg`). An override never hides a flag the agent created or changed in the run. - `exclude_mcp_tools` suite option: hide older duplicate tools a skill's real MCP server doesn't offer. - `inline_skill` suite option: put SKILL.md in the agent's instructions. With builtins off, the skill listed in the SDK agent's `skills` never reached the transcript, and force_skill_invocation interrupted every MCP call when the agent's first action was one. - Mocks reflect the agent's writes: create-flag keeps the sent `defaults`, and toggle-flag reports and stores the requested state. - Add 9 hosted MCP tools (guarded rollouts, rollouts, approvals, experiment results, metric events) and bring create-flag, toggle-flag, list-metrics, create-metric, and create-experiment in line with the hosted schemas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven cases for the AWS DevOps Agent skill: asking before acting, the no-GitHub handoff, the 5 + 5 fallthrough experiment, the LaunchDarkly approval path, the guarded rollout after a win, the regular-rollout fallback on plans without guarded rollouts, and a losing experiment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…answered goal Evals caught the agent applying an approval request that list-approval-requests still showed pending, and creating a flag and metric after its goal question went unanswered. - A pending request means stop and tell the user; don't re-check in the same turn. - Make no changes until questions 1-2 are answered; ask again or stop instead of assuming an answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An explicit empty entry in mock_ask_question_answers now returns no selection instead of the first option, so the vague-goal case can model a user who never answers. No other suite passes an empty entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ging other suites - Mark the 9 new tools opt_in; a suite sees them only through include_mcp_tools, so other suites keep the 35 tools they had. - Relax create-experiment back to the top-level required fields it had, and describe the full iteration shape in its description instead, so a call another suite makes isn't rejected before it's recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tools - Step 3 creates a multivariate flag when the hypothesis has several versions, with the control first and both defaults on it. - Step 6 splits 10% of traffic evenly across all variations (10 / N), sends a complete frequentist analysisConfig, and adds Benjamini-Hochberg with its required scope for two or more treatments. - Step 7 decides proven / loser / inconclusive across several treatments, always stops with the control treatment as winner, and rolls out the winning variation. - An agent space can block tools it marks destructive (likely toggle-flag and start-experiment-iteration), failing silently. After a failed or empty write, read the state back; if unchanged, tell the user and give a direct LaunchDarkly link to make the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…files Other suites now see exactly main's tool definitions, mocks, and mock behavior. - Move the 9 added tools, the 5 changed definitions, and the new mocks into experiment-orchestration/tools.json and mocks.json, loaded through new `tool_definitions` and `tool_mocks` suite options. This replaces the opt_in / include_mcp_tools mechanism. - Revert the shared create-flag and toggle-flag mock changes: a stored toggle bypassed flag-drift's per-environment get-flag. - Add multipleComparisonCorrectionScope to this suite's create-experiment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Several versions: a multivariate flag with the control first and both defaults on it. - Three variations: 3.33% each within the 10%, the control outside the experiment, and Benjamini-Hochberg with its scope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4ef63aa. Configure here.
ari-launchdarkly
approved these changes
Oct 29, 2026
ari-launchdarkly
left a comment
ari-launchdarkly
left a comment
Contributor
There was a problem hiding this comment.
Those cursor comments might be worth addressing
…ll metrics to the rollout Addresses Cursor Bugbot: Proven and Loser could both match on a multivariate run, and the release-policy step could drop the error-rate guardrail from the guarded rollout. A treatment with a guardrail regression and no winner now stops the experiment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…; create-flag defaults fallback Lists more tools an agent space may block as destructive. Any write the agent can't complete is handed to the user with the link, what to click, and every value. A plan limit counts only when LaunchDarkly's error says so, so a silently blocked start-guarded-rollout no longer falls back to an unguarded rollout. If create-flag rejects defaults, set the off variation and default rule to the control right after creating the flag, then read it back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cchu-ld
enabled auto-merge (squash)
October 29, 2026 16:52
cchu-ld
disabled auto-merge
October 29, 2026 17:08
cchu-ld
added a commit
to launchdarkly/ldcli
that referenced
this pull request
Oct 29, 2026
The sync now pins 887d4cb, the merge commit of launchdarkly/ai-tooling#230. SKILL.md changed, so its hash is in shippedSkillHashes. The agent prompt did not change. The skill test now compares the description with the front matter of the embedded SKILL.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cchu-ld
added a commit
to launchdarkly/ldcli
that referenced
this pull request
Oct 29, 2026
…t setup (#972) * feat: add the experimentation skill to the agent space in setup Setup now adds the experiment-orchestration skill to the agent space. The skill tells the agent how to move a goal through an experiment, a guarded rollout, and a report. The skill text comes from the setup guide. If the space has a skill with the same name and other instructions, setup does not change it. Pass --replace-skill to replace the instructions. The status and agent types of the skill stay the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: make the skill step optional and update copies from older versions Setup now does the skill step after the MCP server step. If the skill step fails, setup shows a warning and puts the error in skillError. The other steps still run. If setup cannot read the skills of the space, the plan skips the skill. Setup keeps the SHA-256 of each SKILL.md that it sent. If a space has one of these texts, setup updates it without --replace-skill. Other text is a customer edit. The plan keeps the version of the skill asset. If the version changes before the write, setup does not change the skill. The plan line and the help text say that every agent type in the space can load the skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep the AWS message in skill warnings and report the skill action The skill warning now keeps the full AWS error, so it names the denied action. If AWS denies access, the warning also names the permissions that the skill step needs. The warning shows one time, at the end of the run. An empty SKILL.md now gets an update without --replace-skill. A skill with no SKILL.md gets its own plan line. A SKILL.md that AWS sends as bytes is compared as text. The setup result now has skillAction, so a JSON run shows if setup kept, updated, or left the skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: report a plan change as a skipped skill and warn on stderr in JSON runs If the skill changes between the plan and the write, skillAction is now skipped, because setup tried no write. The warning for a denied GetAssetFile now names the skill one time. The plan line for a skipped skill starts with "Skip skill". A JSON run now also writes the skill warning to stderr. If someone deletes the skill after the list, the plan now creates a new skill. Before, the plan said that the skill had no SKILL.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: let the experiment skill resume across runs and wait 2 weeks for a result A scheduled run of the agent has no memory of the run that started the experiment. The skill now keeps an experiment record, tags the flags that it owns, and resumes in-flight work before it starts a new goal. The experiment now runs at least 2 weeks before a decision, not 30 minutes. A result that is not a win stops the iteration with control as the winner and turns the flag off. Guarded rollout stages stay at or below 50%, as LaunchDarkly requires. If no coding agent is available, the skill waits for a person to confirm the deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: shape the experiment skill for a chat run with approved writes In AWS DevOps Agent, every LaunchDarkly write needs a person to approve it in chat, and no coding agent or CI/CD is connected. The skill now runs as two chats: one that starts an experiment and one that checks on experiments. A person implements and deploys the change, and the agent stops at each wait. - The flag serves false when it is on, so turning it on before the experiment starts does not give everyone the treatment. - The experiment uses 10% of traffic through 5% and 5% treatments and notInExperimentVariationId, with sequential testing. - The agent shows the draft experiment and waits for a confirmation before it starts it, as the create-experiment tool asks. - The record in the flag description holds the project and environment. - The release review, merge, and deploy steps are gone. The allowlist adds update-flag-settings, for the record, and create-metric. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: add missing MCP tools to an existing allowlist on request, and keep the win out of the 100% path Stopping an experiment iteration serves the chosen winner to everyone that the rule matches. The skill now stops a won experiment with control as the winner, records the win first, makes sure that the default rule serves control, and then ramps the treatment with the guarded rollout. A space that reuses an MCP server kept its old allowlist, so the tools that the skill now uses were missing. Setup now names the missing tools in the plan and warns. With --add-mcp-tools it adds them. It rewrites the allowlist only if every tool on it is an ldcli tool, because AWS replaces the whole configuration and does not send back each tool's classification, and an unclassified tool runs without approval. The run skips the rewrite if the allowlist changed after the plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: address review feedback on the experiment skill - The report template is in a fenced markdown block, so its headings stay out of the skill's outline. - The report compares control with treatment, not before with after. - "Design waiting for confirmation" is a record state, so a later chat resumes the draft experiment instead of making another one. - The skill says why the 2-week minimum holds with sequential testing on: sequential testing lets a guardrail regression end the experiment early, and the minimum covers weekly cycles and novelty effects. - The report recommends a flag cleanup for a winner too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: add the experiment-orchestrator custom agent to the agent space in setup Setup now creates a custom agent that runs the experiment-orchestration skill with the LaunchDarkly MCP tools that the space allows. The tool IDs are "<server name>_<tool name>", the format that AWS uses for a custom agent made in the console. An update adds the ldcli tools and skill to the agent and keeps the ones that the customer added. If someone edited the prompt, setup leaves the agent unless --replace-agent is passed. The agent is optional: if its step fails, or the MCP server or the skill is missing, setup warns and does the other steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: show the skill and the custom agent in status status now reports the experiment-orchestration skill and the experiment-orchestrator custom agent of each agent space: whether each is the ldcli version, an older one that setup updates, edited, without its file, or missing. For the agent, it also gives the number of tools and whether the skill is attached. An agent with the ldcli prompt but without the skill or an allowed tool is outdated, because setup updates it. Like the operator app, an asset that status cannot read does not stop it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: shape the custom agent prompt for chat runs and read-only runs The skill now runs in chat, where a person approves each write. Outside chat, for example in a scheduled run, every write fails. In chat, the agent follows the skill. Outside chat, it only reads: it reports, for each experiment in flight, what is due and the message to send in chat, and it never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: name the custom agent's tools after the server's name in AWS Since #968, setup registers a server for each agent space, named LaunchDarkly-<agent space ID>, and a replace can rename a server. The agent's tool IDs used the fixed name LaunchDarkly, so they would not match. The run now reads the name of the registered server, and the plan shows the name that the server will have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: add the experiment-orchestrator custom agent to the agent space in setup Setup now creates a custom agent that runs the experiment-orchestration skill with the LaunchDarkly MCP tools that the space allows. The tool IDs are "<server name>_<tool name>", the format that AWS uses for a custom agent made in the console. An update adds the ldcli tools and skill to the agent and keeps the ones that the customer added. If someone edited the prompt, setup leaves the agent unless --replace-agent is passed. The agent is optional: if its step fails, or the MCP server or the skill is missing, setup warns and does the other steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: show the skill and the custom agent in status status now reports the experiment-orchestration skill and the experiment-orchestrator custom agent of each agent space: whether each is the ldcli version, an older one that setup updates, edited, without its file, or missing. For the agent, it also gives the number of tools and whether the skill is attached. An agent with the ldcli prompt but without the skill or an allowed tool is outdated, because setup updates it. Like the operator app, an asset that status cannot read does not stop it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: shape the custom agent prompt for chat runs and read-only runs The skill now runs in chat, where a person approves each write. Outside chat, for example in a scheduled run, every write fails. In chat, the agent follows the skill. Outside chat, it only reads: it reports, for each experiment in flight, what is due and the message to send in chat, and it never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: name the custom agent's tools after the server's name in AWS Since #968, setup registers a server for each agent space, named LaunchDarkly-<agent space ID>, and a replace can rename a server. The agent's tool IDs used the fixed name LaunchDarkly, so they would not match. The run now reads the name of the registered server, and the plan shows the name that the server will have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: make the custom agent prompt a thin layer on the experiment skill The prompt now loads the experiment-orchestration skill from #970 and follows it as is. It adds only the LaunchDarkly MCP tools and one rule: never start a goal that a person did not ask for. The old prompt never shipped, so its hash is replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: shorten the custom agent prompt to one paragraph The prompt now has one paragraph that loads the experiment-orchestration skill and follows it. The help text for the custom agent is shorter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: remove --replace-agent from aws-devops-agent setup Setup never overwrites an edited prompt of the custom agent. The plan and the log say that setup leaves the agent, and that a person can delete the agent in the AWS console and run setup again to get the ldcli version. Status no longer names a flag for an edited agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: leave the skill and the custom agent out of status Status does not show the experiment-orchestration skill or the experiment-orchestrator custom agent. Status is the same as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: remove comments that repeat the code The comments that stay give a reason that the code does not show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: remove dead code from the custom agent step Setup always connects the MCP server before the agent step, so the check for a missing server is gone. planAgent no longer takes the options, and the unused AllowedTools test helper is gone. The message for a missing permission now names aidevops:ListAssociations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: pass the MCP server name to the custom agent step The MCP step knows the name of the server, so it keeps the name in the result. The agent step no longer lists the servers again, and the OAuth poll test is the same as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: give the custom agent the tools that --add-mcp-tools adds The plan compared the agent with the allowlist before the MCP step. With --add-mcp-tools, the MCP step adds tools first, so the plan kept the agent without them. The plan now expects those tools, so setup updates the agent in the same run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: sync the AWS DevOps Agent files from ai-tooling at a fixed commit The experiment-orchestration skill and the custom agent prompt now come from launchdarkly/ai-tooling. scripts/sync-aws-devops-agent.sh downloads them at the commit in AI_TOOLING_COMMIT. go generate and make aws-devops-agent-sync run it. ldcli embeds the files, so they stay committed. CI fails if they differ from that commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: sync the AWS DevOps Agent files from the ai-tooling merge commit The sync now pins 887d4cb, the merge commit of launchdarkly/ai-tooling#230. SKILL.md changed, so its hash is in shippedSkillHashes. The agent prompt did not change. The skill test now compares the description with the front matter of the embedded SKILL.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: say that setting up the skill and the custom agent is optional "The agent is optional" can mean that a space does not need the agent. The help text now says that the setup step is optional. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: allow the MCP tools of the whole experiment flow, and add them to existing spaces The allowlist now has the tools that the new SKILL.md names and that were missing: list-metric-events and update-off-variation. It also has tools for the rest of that work that the skill does not name: update-experiment, save-and-start-experiment-iteration, create-automated-rollout-config, get-flag-health, get-flag-unique-context-exposures, and search-contexts. Setup now adds the missing ldcli tools to the allowlist of an existing space without --add-mcp-tools, so spaces from older ldcli versions get the new tools. The flag still parses, has no effect, and is deprecated. A new test fails when SKILL.md names a tool that the allowlist does not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: download the AWS DevOps Agent skill and prompt from ai-tooling main Setup no longer embeds SKILL.md and the custom agent prompt. The plan downloads both from the main branch of launchdarkly/ai-tooling, and the run writes the texts that the plan compared. If the download fails, setup skips the skill and custom agent steps and warns. A space text counts as ours when it is empty, matches a frozen hash of a released ldcli, or equals an earlier version of the file on ai-tooling main. Setup reads that history only when nothing else matches, and skips the step if it cannot read it. The plan and the result warn when the skill names MCP tools that this ldcli version does not allow. The sync and check scripts, their Makefile targets, and the CI step are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: drop the unknown-tool warning and the ai-tooling history check A future GitHub Action will compare the skill on ai-tooling main with the allowlist and open a pull request when they differ, so setup no longer warns about tools that the skill names. Setup also no longer reads the history of the files on ai-tooling main. A space whose text is the current main text or a text that a released ldcli embedded gets the new text. Other text counts as an edit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: skip an existing skill or agent that differs from ai-tooling main Setup no longer keeps the hashes of the texts that older ldcli versions embedded. If the skill in the space is not the latest SKILL.md on ai-tooling main, the plan says so and setup skips it. --replace-skill updates it to the latest version. A custom agent whose prompt differs is skipped the same way, with the hint to delete it and run setup again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: trim comments and dead code in the skill and custom agent steps Sources.File drops the ref, which is always main, and GitHubSources drops the nil-client fallback. planSkill is one function again, and skillText is inlined. Comments that repeat the code, or that describe the embedded files and the hash lists, are gone. The allowlist comment no longer says that the skill keeps its record in the flag description. Tests named for edits are named for a different text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: clarify why setup downloads the skill and prompt once Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: restore comments that the earlier cleanup only rephrased Comments whose code did not change get their wording from main back, and skillText and two test names return. Comments now change only where the behavior did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds an AWS DevOps Agent category with two files that
ldcli aws-devops-agent setupinstalls in an agent space:skills/aws-devops-agent/experiment-orchestration/SKILL.md: theexperiment-orchestrationskill. It takes a product goal through a LaunchDarkly experiment and, if the experiment wins, a guarded rollout.skills/aws-devops-agent/agents/experiment-orchestrator.md: the system prompt of theexperiment-orchestratorcustom agent. The agent runs the skill with the LaunchDarkly MCP tools.The skill is not in the Claude Code, Cursor, or Codex plugin manifests, because it is for AWS DevOps Agent. It uses GitHub through the AWS DevOps Agent GitHub integration when GitHub is connected to the agent space; without it, the skill hands the code change, merge, and deploy to a person. On LaunchDarkly plans without guarded rollouts, it uses a regular rollout.
There are overlap with existing experimentation skills - but the prompt is different enough for the AWS DevOps agent that we should keep it separate for this use case.
python3 scripts/validate_skills.py, the unit tests, andgenerate_catalog.py --checkpass.🤖 Generated with Claude Code
Note
Overview
Adds an AWS DevOps Agent package: the
experiment-orchestrationskill (8-step flow from goal clarification through a fixed 10% experiment to guarded or regular rollout) plus theexperiment-orchestratorcustom-agent system prompt. Documents install vialdcli aws-devops-agent setupin the root README andskills/aws-devops-agent/README.md, and registers the skill inskills.json.Ships a promptfoo eval suite (
evals/experiment-orchestration/) with scenario tests (vague goals, no GitHub handoff, multivariate flags, 10% allocation, approvals, win/loss/plan-limit branches), suite-specific MCP tool definitions and mocks, and npm scripts wired through the eval manifest.Extends the Claude skill eval provider so suites can
inline_skill(embedSKILL.mdin instructions), merge or hide MCP tools per suite, and drive branch-specific behavior with per-testmock_overrides(includingby_argresponses). Adds unit tests for those helpers.Reviewed by Cursor Bugbot for commit dd04f43. Bugbot is set up for automated code reviews on this repo. Configure here.