Skip to content

refactor(eval): address on-demand review feedback - #2007

Open
jariy17 wants to merge 2 commits into
aws:refactorfrom
jariy17:followup/eval-ondemand-review-feedback
Open

refactor(eval): address on-demand review feedback#2007
jariy17 wants to merge 2 commits into
aws:refactorfrom
jariy17:followup/eval-ondemand-review-feedback

Conversation

@jariy17

@jariy17 jariy17 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

  • Missing runtime telemetry now throws ResourceNotFoundError.
  • Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
  • Malformed telemetry produces at most one warning per command.
  • Repeated validation tests now use test.each.
  • Redundant and drift-prone comments were removed.

Decisions

  1. Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
  2. Keep results flat: context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
  3. Defer a generic poller: extract one when another production caller needs the same behavior.
  4. Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

  • Build, typecheck, lint, changed-file formatting, and focused tests pass.
  • Full suite: 1,373 tests pass.

@github-actions github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 14, 2026
@codecov-commenter

codecov-commenter commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.01%. Comparing base (a9d34be) to head (1779d53).
⚠️ Report is 2 commits behind head on refactor.

Additional details and impacted files
@@             Coverage Diff              @@
##           refactor    #2007      +/-   ##
============================================
+ Coverage     96.96%   97.01%   +0.04%     
============================================
  Files           366      366              
  Lines         21179    21191      +12     
============================================
+ Hits          20536    20558      +22     
+ Misses          643      633      -10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 14, 2026

@Hweinstock Hweinstock left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

return { core, stdout: io.stdout(), stderr: io.stderr() };
}

async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

@Hweinstock

Copy link
Copy Markdown
Contributor

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants