Repository navigation
Conversation
…and start Score the skill's gates for a complete design, suggested versus asked fields, a missing event, and not starting or changing a flag until the user says so. Co-authored-by: Cursor <cursoragent@cursor.com>
Invoke the skill in the same turn as the request, and grade lookups and forbidden writes instead of ask-question answers that never arrive. Co-authored-by: Cursor <cursoragent@cursor.com>
…up-evals Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # evals/README.md # evals/package.json # evals/providers/claude-skill-agent-sdk.js # evals/scripts/_manifest.js
…atalog The four tools this suite added, and their default templates, now load only for this suite. Sentinel responses stay in the shared mock renderer. Co-authored-by: Cursor <cursoragent@cursor.com>
cromeroLD
enabled auto-merge (squash)
October 30, 2026 19:31
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
launchdarkly-experiment-setupwith 8 cases: complete design, suggested control and unit, ask instead of suggesting, missing pieces, missing event, start with the flag off and the event missing, missing flag, and a running experiment change. Each case has a weight-3 tool check (required lookups, and no writes: no create, start, or flag change) and a weight-1 wording rubric, so the tool check decides pass or fail. Lookup checks accept equivalent alternatives. Write checks stay strict.inline_skilloption, the same way experiment-orchestration does, withforce_skill_invocationoff. A forced invocation sends a separate turn that interrupts this skill's early lookups. No provider changes.get-flag-status-across-envs,list-metric-events,search-contexts,save-and-start-experiment-iteration) are defined only for this suite viatool_definitionsandtool_mocks, so other suites don't see them. Their input schemas are copied from the hosted LaunchDarkly MCP. Argument-based test data for existing tools stays in_mock.js, keyed on sentinel values no other suite sends, with unit tests.expose_ask_questionis off. No case scripts an answer, and an unscripted call auto-selects the first option._mock.jsand their unit tests, plus registering the suite in the eval README,package.json, and_manifest.js.Testing
npm test: 129/129.npm run eval:experiment-setuptwice: 8/8 both times (eval-FCE-2026-10-30T19:00:29,eval-QnX-2026-10-30T19:01:35). Every tool check and rubric scored 1.eval:flag-drift4/4 (eval-0Qs-2026-10-30T19:02:45),eval:flag-command3/3 (eval-1oY-2026-10-30T19:04:00),eval:experiment-guide:singlepass (eval-clB-2026-10-30T19:04:22),eval:experiment-orchestration:singlepass (eval-taQ-2026-10-30T19:06:30).origin/main, including experiment-orchestration.Notes
Follow-ups (not in this PR):
create-experimentdescription says Bayesian is the default. This skill omitsmethodology, and the product default is frequentist. The description was left unchanged.