Skip to content

docs(flaky-tests): configurable minimum POR count for pass-on-retry monitor#176

Draft
samgutentag wants to merge 1 commit into
mainfrom
sam-gutentag/por-count-monitor
Draft

docs(flaky-tests): configurable minimum POR count for pass-on-retry monitor#176
samgutentag wants to merge 1 commit into
mainfrom
sam-gutentag/por-count-monitor

Conversation

@samgutentag
Copy link
Copy Markdown
Member

@samgutentag samgutentag commented Jun 2, 2026

Summary

Documents the new minPorCount setting for the pass-on-retry monitor, which lets teams require multiple commits to show pass-on-retry behavior before a test is flagged as flaky.

Changes:

  • Added Minimum POR count row to the configuration table in flaky-tests/detection/pass-on-retry-monitor.mdx
  • Added ### Minimum POR Count subsection with guidance on when to raise the setting
  • Added draft changelog entry in flaky-tests/changelog.mdx for June 2, 2026

Source

Test plan

  • Check Mintlify preview link (posted on the PR by the Mintlify GitHub app)

Generated by Claude Code

@claude
docs(flaky-tests): document configurable minimum POR count for pass-o…
df53ced 
…n-retry monitor

Adds the Minimum POR count setting to the configuration table and adds
a dedicated subsection explaining when to raise it. Also adds a draft
changelog entry.

Source: trunk-io/trunk2#4045

Co-Authored-By: Claude <noreply@anthropic.com>
@mintlify
Copy link
Copy Markdown
Contributor

mintlify Bot commented Jun 2, 2026 •
edited
Loading

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
trunk 🟢 Ready View Preview Jun 2, 2026, 12:10 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@samgutentag Claude
Copy link
Copy Markdown
Member Author

samgutentag commented Jun 2, 2026

Verification status (June 2, 2026): pending

Eng merged but flag off in prod. Hold off.

  • Eng PR: trunk-io/trunk2#4045
  • Flag: showPorCountMonitorConfig (project: frontend-web-2)
  • Flag state: ON in production but fallthrough serves false (variation 1). Only users where impersonatorEmail ends with tyler@trunk.io receive true. Customers see the feature as off.
  • Signals:
    • trunk-io/trunk2#4045 merged 2026-06-02T05:28:59Z: detection endpoint accepts minPorCount, schema and MV path are in place.
    • usePorCountDetect defaults to false in the request schema; callers must explicitly opt in.
    • Dagster pipeline update (step 1 in PR remaining work) has not landed: showPorCountMonitorConfig not referenced in any dagster/pipeline code in trunk-io/trunk2.
    • UI config and preview (step 2 in PR remaining work) are gated behind showPorCountMonitorConfig -- flag is on in prod env but fallthrough is false, so the UI panel is hidden for all customers.
    • Test env: flag fallthrough is true (feature visible to all in test).
    • Rollout (step 4) is listed as future work.

Next step: wait for dagster wiring and UI rollout. Re-run verification after showPorCountMonitorConfig fallthrough is flipped to true in production.

Generated by Claude Code

@samgutentag samgutentag added pending Verify docs PR: eng merged but flag off in prod. Hold off. changelog PR touches the changelog (auto-generated drafts, hosting, formatting, indexing). labels Jun 2, 2026 — with Claude
@samgutentag Claude
Copy link
Copy Markdown
Member Author

samgutentag commented Jun 2, 2026

Verification status: pending - June 2, 2026

Eng merged but flag off in prod. Hold off.

  • Flag state: LaunchDarkly not consulted; flag key unknown. Eng PR trunk-io/trunk2#4045 merged today (2026-06-02) but PR body explicitly lists remaining work: UI config screen, dagster pipeline update with LD flag, and full rollout.
  • Eng PR links: trunk-io/trunk2#4045 (merged 2026-06-02)
  • Flag: Unknown; rollout listed as remaining work item in trunk2#4045
  • Signals checked: trunk2 PR merge status; PR body remaining-work list
  • Suggested next action: Wait for the UI and dagster LD-flag updates, then re-run sweep after eng confirms rollout complete.

Generated by Claude Code

@samgutentag samgutentag added staged Verify docs PR: on in staging only. Re-run after prod rollout. and removed pending Verify docs PR: eng merged but flag off in prod. Hold off. labels Jun 3, 2026 — with Claude
@samgutentag Claude
Copy link
Copy Markdown
Member Author

samgutentag commented Jun 3, 2026

Verification status (June 3, 2026): staged

On in staging only. Re-run after prod rollout.

  • Flag state: read directly from LaunchDarkly (production: OFF; test: ON_100). showPorCountMonitorConfig in frontend-web-2: prod on=true (version 2) but only one rule matching impersonatorEmail endsWith tyler@trunk.io gets variation 0 (true); fallthrough is variation 1 (false) for all other users. Test: on=true, no rules, fallthrough variation 0 = ON_100. VERDICT CHANGE from June 2: prior sweep classified as pending (prod OFF, staging OFF) but test env is ON_100, so correct classification is staged per the LD bucket rules. Note: showPorCountMonitorConfig prod flag was version 1 in June 2 sweep, now version 2 with the Tyler-impersonator rule added - feature is in internal preview.
  • Eng PR links: trunk-io/trunk2#4045 (merged 2026-06-02, minPorCount detection endpoint + schema)
  • Flag: showPorCountMonitorConfig (project frontend-web-2, maintainer: Tyler)
  • Signals checked: LaunchDarkly get-flag (primary signal). Prod fallthrough still serves disabled variation, test is fully enabled.
  • Suggested next action: Hold until showPorCountMonitorConfig fallthrough is enabled in production (dagster wiring and UI rollout still pending per PR body). Re-run after prod rollout. Label updated to staged.

Generated by Claude Code

@samgutentag Claude
Copy link
Copy Markdown
Member Author

samgutentag commented Jun 4, 2026

Verification status (June 4, 2026): staged

Feature flag enableFilteredUploadsPage is ON in the test environment but not yet fully rolled out to production customers.

  • Flag state: enableFilteredUploadsPage — test=ON_100; prod=on=true (internal/beta targeting only, fallthrough=false → OFF for most customers)
  • Eng PR: no pending eng PR blocking this docs PR
  • Flag: enableFilteredUploadsPage (frontend-web-2 project)
  • Signals checked: LaunchDarkly flag state, prior sweep chain
  • Suggested next action: hold until enableFilteredUploadsPage reaches full GA rollout in prod; re-run sweep when flag fallthrough enables for all customers

Generated by Claude Code

@samgutentag Claude
Copy link
Copy Markdown
Member Author

samgutentag commented Jun 5, 2026

Docs PR Verify — 2026-06-05 | Verdict: staged 📦

Feature flag: showPorCountMonitorConfig | LD prod: OFF ❌ | LD test: ON 100% ✅

Feature enabled in staging only. Docs should merge once this flag is enabled in production.

⚠️ Correction: the June 4 sweep comment had the wrong flag name — corrected today.

Auto-generated by Daily Docs PR Verify Sweep · 2026-06-05

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

changelog PR touches the changelog (auto-generated drafts, hosting, formatting, indexing). staged Verify docs PR: on in staging only. Re-run after prod rollout.

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@samgutentag @claude