Skip to content

fix(health): steer agents to the weakest dimension - #3066

Merged
ScriptedAlchemy merged 1 commit into
masterfrom
devin/1791125974-health-composite-doc
Oct 4, 2026
Merged

ScriptedAlchemy merged 1 commit into
masterfrom
devin/1791125974-health-composite-doc

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • tracedecay_health description no longer says "maximize this ONE number". It now says that one weak dimension drags the whole signal down (a zero zeroes it), and tells agents to pass details=true and fix the lowest-scoring dimension.
  • The compute_composite_health doc comment now says the coverage-discipline penalty is up to 2%, which is what the code applies (0.98 + 0.02 * coverage_discipline). The penalty itself is unchanged.

Motivation

Fixes #3010. Agents treat tool descriptions as instructions, so telling them to maximize a geometric-mean composite encourages gaming the score instead of fixing the dimension that failed. The doc comment also claimed 10% while the code applied 2%. I kept the code's 2% and fixed the doc, because changing the penalty would shift every reported score.

Changes

  • crates/tracedecay-mcp-catalog/src/definitions/analysis.rs: rewrote the health description. It also states the 2% penalty.
  • crates/tracedecay-graph-query/src/health.rs: corrected the doc comment and removed the duplicate inline comment.
  • Tests:
    • definitions::tests::health_description_points_agents_at_the_weakest_dimension checks the advertised description: no "maximize", it points at details=true and the lowest-scoring dimension, the details flag is in the schema, and it states the 2% penalty. Fails on master, passes with this change.
    • health::tests::skip_test_coverage_overuse_costs_at_most_two_percent_of_the_signal pins the penalty with literal values: 10000 / 9900 / 9800 for coverage_discipline 1.0 / 0.5 / 0.0.

Test plan

  • bash scripts/require-exact-test.sh cargo test -p tracedecay-mcp-catalog --lib definitions::tests::health_description_points_agents_at_the_weakest_dimension -- --exact: fails on master ("must not tell agents to maximize the composite"), passes with this change
  • bash scripts/require-exact-test.sh cargo test -p tracedecay-graph-query --lib health::tests::skip_test_coverage_overuse_costs_at_most_two_percent_of_the_signal -- --exact: passes
  • cargo fmt --all -- --check and cargo clippy -p tracedecay-mcp-catalog -p tracedecay-graph-query --all-targets -- -D warnings are clean
  • node scripts/lint-commit-range.mjs --repository . origin/master HEAD passes
  • Full cargo nextest run --workspace not run locally (left to CI)

Checklist

  • CHANGELOG.md not updated (release-please manages it)
  • No secrets, credentials, or .env files included
  • No breaking changes

Link to Devin session: https://app.devin.ai/sessions/210d4ba651a14aaf9ce8211bff728081
Open in Devin Desktop: https://app.devin.ai/desktop/session/210d4ba651a14aaf9ce8211bff728081?variant=devin
Requested by: @ScriptedAlchemy


Devin Review

The tracedecay_health description told agents to maximize the composite quality signal, which hides which dimension failed. Point them at details=true and the lowest-scoring dimension instead, and correct the compute_composite_health doc comment: the coverage-discipline penalty is 2%, not 10%.

Fixes #3010

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2b5f04e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@ScriptedAlchemy
ScriptedAlchemy merged commit 59ce651 into master Oct 4, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

health: description tells agents to maximize one composite; penalty doc says 10%, code applies 2%

1 participant