You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Candidate-first cost budgets and supported-platform evidence before default switch #1832
Child of #1813. Use the manual platform cost workflow from PR #1825 (source SHA and machine metadata) to measure candidate-only end-to-end latency and peak RSS against baseline on typical and large parts on CI-class Linux, macOS, and Windows; include the 18-core development host where available but do not use it as a substitute for supported platforms. Measure median and p95, fallback frequency under an interim policy, and memory-constrained concurrency. Set explicit numerical latency/RSS/fallback budgets before any default switch, record runs and source SHA, and fail the rollout gate if the budgets or evidence are missing. Do not treat the fixed 15-part quality score as a cost or safety result.
Cost-gate status: the manual annotation-layout-cost.yml workflow is in stacked PR #1825 (with completed-case artifact hardening in draft #1836), but is not present on default main and is not listed as dispatchable by gh workflow list --all. No Linux/macOS/Windows platform matrix run or numerical latency/RSS/fallback budget has been accepted. The local candidate-first fixed15 trial on an 8 GiB/4-core Linux container is diagnostic only; its median/p95 candidate process times were ~46.21/278.21 s and peak RSS ~537.27/877.68 MiB, with a large-part tail. The ongoing additional STEP holdout has a baseline outlier beyond 40 minutes. Keep the default switch blocked until supported-platform runs, machine metadata, representative large-CAD numbers, and predeclared numerical budgets exist.
Cost-gate dependency audit (2026-09-25): PR #1825 contains .github/workflows/annotation-layout-cost.yml and its exact-head ci-ok is green, but it is still stacked/open, not on main. GitHub currently returns 404 for gh run list --workflow annotation-layout-cost.yml because the dispatchable workflow is absent from the default branch. Therefore there is no supported-platform candidate-only cost run to compare against numerical budgets yet. The saved v4 Linux fixed15 evidence (source 4ab713e5) reports candidate process median 46.21 s, p95 278.21 s, baseline median 39.94 s, p95 274.54 s, and candidate peak RSS median 537.27 MiB/p95 877.68 MiB; fallback rate is null, not zero. Those are exploratory host/corpus figures, not a Linux/macOS/Windows budget pass or default-switch authorization. The workflow must become dispatchable through an in-order merge (or another approved CI execution route) before this gate can be closed; no passed CAD task was rerun for this audit.
Pre-rollout numerical stop criteria are now in stacked draft PR #1855: ≤1.25× baseline median/p95 process time and RSS on each supported platform and typical/large cohort, ≤5% measured interim fallback, ≤1.25× total latency including fallback, and a 75%-RAM concurrency cap. The only existing cost observation is saved v4 Linux fixed15; macOS/Windows, large/user CAD, and fallback frequency are still missing, so this is a budget definition, not a gate pass. No CAD work was rerun.
Budget definition checkpoint: stacked PR #1855 is now exact-head CI green and ready for review at c5d87ad. Its 15-row saved cost extract was checked against the original full Linux report; no CAD run was repeated for it. The numerical limits are precommitted, but macOS/Windows, large/user CAD, and actual fallback observations remain missing, so #1832 is still open and no default switch is authorized.
Cost-gate unblock: #1860 adds an opt-in pull_request:labeled trigger to the existing three-platform fixed15 cost workflow, since workflow_dispatch is unavailable while the file exists only on the stack. The annotation-layout-cost label was applied once; run https://github.com/pzfreo/draftwright/actions/runs/36198994169 is queued at exact PR head 0ba17cc. The workflow records the checked-out source SHA separately from the PR event/merge SHA and keeps --jobs 1. No cost or rollout pass is claimed yet; large/user CAD and fallback-rate gates remain separate.
First supported-platform cost artifact completed (Linux job of opt-in run 36198994169, source SHA 0ba17cc; provenance sidecar also records synthetic PR event SHA separately). Fixed-15 candidate-first and offline baseline, 15/15 workers each, no failures: candidate/baseline process median 17.85/16.78 s = 1.064x, p95 144.30/133.86 s = 1.078x; peak RSS median 536.14/537.44 MiB = 0.998x, p95 884.87/884.42 MiB = 1.001x. Thus this Linux fixed-15 cohort meets the documented 1.25x time/RSS ratios. It is not the full cost gate: Windows/macOS jobs are still running, large/user-part cohorts and configured-concurrency RAM bound are not established, and interim fallback rate/total latency are unavailable because candidate-preview has no automatic fallback. Quality: 12 metric wins/3 ties and 15/15 semantic parity at this exact head, but independent safety admission remains false and 0/15 pass all current provisional checks. I downloaded the completed artifact once; no CAD task was restarted.
The single opt-in cost run 36198994169 has now completed SUCCESS on Linux, macOS, and Windows. All three artifacts record exact PR head 0ba17cc separately from the synthetic PR event SHA; 15/15 candidate and offline-baseline workers completed per platform. Fixed-15 candidate/baseline process-time ratios (median, p95): Linux 1.064x/1.078x; macOS 1.018x/1.027x; Windows 1.011x/1.139x. Peak-RSS ratios (median, p95): Linux 0.998x/1.001x; macOS 0.998x/1.139x; Windows 0.995x/0.936x. Each is within the documented 1.25x per-platform fixed-15 ratios. Every platform reports 12 metric wins, 3 ties, 15/15 semantic parity, and zero production selections; current independent safety passes 0/15. This is a completed supported-platform fixed-15 cost measurement, not the full production-cost gate: large/user-part cohorts, runner RAM/concurrency bound, interim fallback frequency, and fallback-inclusive latency remain unproven. No workers were rerun.
Progress: #1873 records the already-completed three-platform fixed15 cost run 36198994169 directly from its saved artifacts, and bounds future opt-in CAD workers to 900 s (timeouts remain failed cases). No CAD workers were rerun. On source 0ba17cc, all 15 pairs completed on Linux, macOS, and Windows, with candidate median/p95 process-time and RSS ratios below the predeclared 1.25x ceiling. This is not a full cost or rollout pass: large/user CAD, configured-concurrency RAM, interim fallback frequency, fallback-inclusive latency, and current-head safety admission remain open.
New supported-platform real-part cost cohort is PR #1878 (source 80becbb), using the existing FTC09 AP242 one-case manifest under an explicit annotation-layout-cost-ftc09 opt-in label. The first three-platform run is GitHub Actions 36330010666, currently queued; each isolated worker remains bounded to 900 s and the artifact will record manifest SHA-256 and source-head provenance. This does not rerun the passed fixed15 cohort. FTC09 is a distinct PMI-rich public part, not user-supplied large CAD, so #1832 still needs a representative large/user model and concurrency/total-latency evidence. No cost conclusion until the run completes and artifacts are reviewed.
FTC09 three-platform run https://github.com/pzfreo/draftwright/actions/runs/36330010666 completed on exact source 80becbb (manifest SHA-256 9012b99e4b6719f0357d5eeee61c1c8c17f33502aa10ca86bfec86f9615c4efa). I reviewed the saved per-platform artifacts; no worker was rerun. All Linux/macOS/Windows workers finished, but all three comparisons exited 1 because candidate-preview loses the same baseline carriers: PMI source geometric_tolerance:0:1:4:32 (pmi_dropped introduced) and the envelope width.length dimension labelled 190.5. Coverage totals alone remain equal, illustrating why exact carrier parity matters. Candidate/baseline process-time ratios are 0.972x Linux, 0.895x macOS, 0.933x Windows; peak-RSS ratios are 0.971x, 0.825x, 0.961x. These are one-case samples, not meaningful p95 or a rollout pass. PR #1878's opt-in cohort correctly stays red; investigate the candidate-specific loss using these artifacts before any rerun. Fixed15 platform evidence remains valid and is not being repeated. User-supplied large-CAD and concurrency/total-latency gates remain open.
FTC09 supported-platform observation: one opt-in run at source b5a9ced passed on Linux/macOS/Windows with candidate-first semantic parity and an offline quality tie. Candidate/baseline process ratios were 0.975× / 0.760× / 0.939×; peak-RSS ratios 0.971× / 1.026× / 0.960×. All are single samples, not representative median/p95, concurrency, fallback-inclusive, or large user-CAD evidence. Independent safety still fails eight shared checks. PR #1878 records artifact provenance; follow-up bf11fd3 fixes a Windows CRLF-only manifest-hash discrepancy without rerunning the already-passed CAD cohort. The #1832 rollout cost gate remains open.
The opt-in FTC09 cohort workflow in PR #1878 is now squash-merged to main as b7ea911 after exact-head CI passed. Its sole three-platform CAD run remains 36332797155 at source b5a9ced; the later committed-blob manifest-hash correction was provenance-only and did not repeat CAD. PR #1881 is adding the reviewed results to the versioned research ledger. The broader #1832 cost gate remains open.
Closing this pre-default cost evidence issue with an explicit accepted evidence limit, not claiming its original large-cohort/concurrency checklist was fully completed. The predeclared numerical budgets are in docs/research/1813-candidate-cost-budget.md. The saved fixed15 Linux/macOS/Windows median and p95 candidate ratios are within the 1.25x time/RSS ceilings, and the opt-in large FTC09 three-platform sample was favourable, but that large-case observation is one sample and configured concurrency has not been benchmarked. No automatic fallback exists, so there is no fallback rate to measure. Treat any measured candidate-specific latency/RSS regression as a focused bug; #1813 retains the reversible rollout decision and monitoring responsibility. This closure accepts the remaining sampling risk rather than silently calling missing data a pass.
Activity
pzfreo commentedon Sep 25, 2026
Cost-gate status: the manual
annotation-layout-cost.ymlworkflow is in stacked PR #1825 (with completed-case artifact hardening in draft #1836), but is not present on defaultmainand is not listed as dispatchable bygh workflow list --all. No Linux/macOS/Windows platform matrix run or numerical latency/RSS/fallback budget has been accepted. The local candidate-first fixed15 trial on an 8 GiB/4-core Linux container is diagnostic only; its median/p95 candidate process times were ~46.21/278.21 s and peak RSS ~537.27/877.68 MiB, with a large-part tail. The ongoing additional STEP holdout has a baseline outlier beyond 40 minutes. Keep the default switch blocked until supported-platform runs, machine metadata, representative large-CAD numbers, and predeclared numerical budgets exist.pzfreo commentedon Sep 25, 2026
Cost-gate dependency audit (2026-09-25): PR #1825 contains
.github/workflows/annotation-layout-cost.ymland its exact-headci-okis green, but it is still stacked/open, not onmain. GitHub currently returns 404 forgh run list --workflow annotation-layout-cost.ymlbecause the dispatchable workflow is absent from the default branch. Therefore there is no supported-platform candidate-only cost run to compare against numerical budgets yet. The saved v4 Linux fixed15 evidence (source4ab713e5) reports candidate process median 46.21 s, p95 278.21 s, baseline median 39.94 s, p95 274.54 s, and candidate peak RSS median 537.27 MiB/p95 877.68 MiB; fallback rate isnull, not zero. Those are exploratory host/corpus figures, not a Linux/macOS/Windows budget pass or default-switch authorization. The workflow must become dispatchable through an in-order merge (or another approved CI execution route) before this gate can be closed; no passed CAD task was rerun for this audit.pzfreo commentedon Sep 25, 2026
Pre-rollout numerical stop criteria are now in stacked draft PR #1855: ≤1.25× baseline median/p95 process time and RSS on each supported platform and typical/large cohort, ≤5% measured interim fallback, ≤1.25× total latency including fallback, and a 75%-RAM concurrency cap. The only existing cost observation is saved v4 Linux fixed15; macOS/Windows, large/user CAD, and fallback frequency are still missing, so this is a budget definition, not a gate pass. No CAD work was rerun.
pzfreo commentedon Sep 25, 2026
Budget definition checkpoint: stacked PR #1855 is now exact-head CI green and ready for review at c5d87ad. Its 15-row saved cost extract was checked against the original full Linux report; no CAD run was repeated for it. The numerical limits are precommitted, but macOS/Windows, large/user CAD, and actual fallback observations remain missing, so #1832 is still open and no default switch is authorized.
pzfreo commentedon Sep 25, 2026
Cost-gate unblock: #1860 adds an opt-in pull_request:labeled trigger to the existing three-platform fixed15 cost workflow, since workflow_dispatch is unavailable while the file exists only on the stack. The annotation-layout-cost label was applied once; run https://github.com/pzfreo/draftwright/actions/runs/36198994169 is queued at exact PR head 0ba17cc. The workflow records the checked-out source SHA separately from the PR event/merge SHA and keeps --jobs 1. No cost or rollout pass is claimed yet; large/user CAD and fallback-rate gates remain separate.
pzfreo commentedon Sep 25, 2026
First supported-platform cost artifact completed (Linux job of opt-in run 36198994169, source SHA 0ba17cc; provenance sidecar also records synthetic PR event SHA separately). Fixed-15 candidate-first and offline baseline, 15/15 workers each, no failures: candidate/baseline process median 17.85/16.78 s = 1.064x, p95 144.30/133.86 s = 1.078x; peak RSS median 536.14/537.44 MiB = 0.998x, p95 884.87/884.42 MiB = 1.001x. Thus this Linux fixed-15 cohort meets the documented 1.25x time/RSS ratios. It is not the full cost gate: Windows/macOS jobs are still running, large/user-part cohorts and configured-concurrency RAM bound are not established, and interim fallback rate/total latency are unavailable because candidate-preview has no automatic fallback. Quality: 12 metric wins/3 ties and 15/15 semantic parity at this exact head, but independent safety admission remains false and 0/15 pass all current provisional checks. I downloaded the completed artifact once; no CAD task was restarted.
pzfreo commentedon Sep 26, 2026
The single opt-in cost run 36198994169 has now completed SUCCESS on Linux, macOS, and Windows. All three artifacts record exact PR head 0ba17cc separately from the synthetic PR event SHA; 15/15 candidate and offline-baseline workers completed per platform. Fixed-15 candidate/baseline process-time ratios (median, p95): Linux 1.064x/1.078x; macOS 1.018x/1.027x; Windows 1.011x/1.139x. Peak-RSS ratios (median, p95): Linux 0.998x/1.001x; macOS 0.998x/1.139x; Windows 0.995x/0.936x. Each is within the documented 1.25x per-platform fixed-15 ratios. Every platform reports 12 metric wins, 3 ties, 15/15 semantic parity, and zero production selections; current independent safety passes 0/15. This is a completed supported-platform fixed-15 cost measurement, not the full production-cost gate: large/user-part cohorts, runner RAM/concurrency bound, interim fallback frequency, and fallback-inclusive latency remain unproven. No workers were rerun.
pzfreo commentedon Sep 27, 2026
Progress: #1873 records the already-completed three-platform fixed15 cost run 36198994169 directly from its saved artifacts, and bounds future opt-in CAD workers to 900 s (timeouts remain failed cases). No CAD workers were rerun. On source 0ba17cc, all 15 pairs completed on Linux, macOS, and Windows, with candidate median/p95 process-time and RSS ratios below the predeclared 1.25x ceiling. This is not a full cost or rollout pass: large/user CAD, configured-concurrency RAM, interim fallback frequency, fallback-inclusive latency, and current-head safety admission remain open.
pzfreo commentedon Sep 27, 2026
New supported-platform real-part cost cohort is PR #1878 (source 80becbb), using the existing FTC09 AP242 one-case manifest under an explicit
annotation-layout-cost-ftc09opt-in label. The first three-platform run is GitHub Actions 36330010666, currently queued; each isolated worker remains bounded to 900 s and the artifact will record manifest SHA-256 and source-head provenance. This does not rerun the passed fixed15 cohort. FTC09 is a distinct PMI-rich public part, not user-supplied large CAD, so #1832 still needs a representative large/user model and concurrency/total-latency evidence. No cost conclusion until the run completes and artifacts are reviewed.pzfreo commentedon Sep 27, 2026
FTC09 three-platform run https://github.com/pzfreo/draftwright/actions/runs/36330010666 completed on exact source 80becbb (manifest SHA-256 9012b99e4b6719f0357d5eeee61c1c8c17f33502aa10ca86bfec86f9615c4efa). I reviewed the saved per-platform artifacts; no worker was rerun. All Linux/macOS/Windows workers finished, but all three comparisons exited 1 because candidate-preview loses the same baseline carriers: PMI source geometric_tolerance:0:1:4:32 (pmi_dropped introduced) and the envelope width.length dimension labelled 190.5. Coverage totals alone remain equal, illustrating why exact carrier parity matters. Candidate/baseline process-time ratios are 0.972x Linux, 0.895x macOS, 0.933x Windows; peak-RSS ratios are 0.971x, 0.825x, 0.961x. These are one-case samples, not meaningful p95 or a rollout pass. PR #1878's opt-in cohort correctly stays red; investigate the candidate-specific loss using these artifacts before any rerun. Fixed15 platform evidence remains valid and is not being repeated. User-supplied large-CAD and concurrency/total-latency gates remain open.
pzfreo commentedon Sep 27, 2026
FTC09 supported-platform observation: one opt-in run at source b5a9ced passed on Linux/macOS/Windows with candidate-first semantic parity and an offline quality tie. Candidate/baseline process ratios were 0.975× / 0.760× / 0.939×; peak-RSS ratios 0.971× / 1.026× / 0.960×. All are single samples, not representative median/p95, concurrency, fallback-inclusive, or large user-CAD evidence. Independent safety still fails eight shared checks. PR #1878 records artifact provenance; follow-up bf11fd3 fixes a Windows CRLF-only manifest-hash discrepancy without rerunning the already-passed CAD cohort. The #1832 rollout cost gate remains open.
pzfreo commentedon Sep 27, 2026
The opt-in FTC09 cohort workflow in PR #1878 is now squash-merged to main as b7ea911 after exact-head CI passed. Its sole three-platform CAD run remains 36332797155 at source b5a9ced; the later committed-blob manifest-hash correction was provenance-only and did not repeat CAD. PR #1881 is adding the reviewed results to the versioned research ledger. The broader #1832 cost gate remains open.
pzfreo commentedon Sep 27, 2026
Closing this pre-default cost evidence issue with an explicit accepted evidence limit, not claiming its original large-cohort/concurrency checklist was fully completed. The predeclared numerical budgets are in docs/research/1813-candidate-cost-budget.md. The saved fixed15 Linux/macOS/Windows median and p95 candidate ratios are within the 1.25x time/RSS ceilings, and the opt-in large FTC09 three-platform sample was favourable, but that large-case observation is one sample and configured concurrency has not been benchmarked. No automatic fallback exists, so there is no fallback rate to measure. Treat any measured candidate-specific latency/RSS regression as a focused bug; #1813 retains the reversible rollout decision and monitoring responsibility. This closure accepts the remaining sampling risk rather than silently calling missing data a pass.