Enumerate CDEB candidates without selecting them - #536
Merged
Conversation
CommitLore — record lintTrailers: clean — 1 commit in Active constraints for the paths this PR touchesLimits (25)
Ruled out (64)
Warnings (17)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
The benchmark cannot fill a short corpus by lowering the bar or manufacturing records. It needs an auditable answer about the records currently available instead: every record is retained in deterministic YAML, cutoff and provenance facts are recorded as rejections, and task-qualification calls that history cannot settle stay explicitly undecided. The enumerator calls the compiled shipping query engine rather than parsing history itself. It can therefore report the same records the product sees while separately applying the frozen snapshot reachability test, the product-repo exclusion, reconstructed provenance, and an explicit fork-author authorization boundary. The schema now permits rejected and undecided rows rather than silently encoding them as passing booleans. Limit: wrong-path viability, deterministic oracle feasibility, code disclosure, bounded implementation, and unproven ordinary or benchmark authorship cannot be decided from history and remain undecided for human review Blast: module Undo: easy Certainty: firm Verified: npm ci; package and bench typechecks; seven focused candidate-registry tests; two builds with no dist diff; CDEB and benchmark verifiers; candidate schema validation; local history census Record-Id: r-cdeb10reg
MongLong0214
force-pushed
the
cdeb-10-registry
branch
from
August 11, 2026 08:57
6a5b77c to
48bd5a8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The enumeration half of CDEB-10. No corpus selection, no sealing, no prompts, no evaluators — those are the rest of the ticket and they come after this reports something worth selecting from.
Why now, when the corpus is not ready
Measured across the authorized repositories,
Ruled-outcounts are 49, 6, 6 and 0 against six tasks required per repository. §3.2 rule 5 is explicit about the alternatives: when candidates are short the study stops, rather than lowering the bar or creating records to fill it.So the useful thing to build is not a corpus — it is the instrument that says how far away one is. This turns waiting into measuring.
It reports why, not just how many
"12 eligible" tells you nothing about whether the bar is holding or quietly drifting. The census names every disqualification:
Rejected candidates are preserved with their reason (§3.2 rule 1), not dropped.
What it refuses to decide
Four §3.2 fields are human calls — whether the wrong path is functionally viable, whether a deterministic oracle is possible, whether the current code already reveals the reason, whether the implementation is bounded. They are marked
undecidedand never counted as eligible.No model decides them and no heuristic guesses them. A registry that promotes its own guesses would produce exactly the number we want to see, which is the failure mode worth designing against here. Every candidate today is
human_review_required, and that is the honest state.Current census on the authorized repositories
32 against a requirement of 30 — but distributed 23/5/4/0, so three of four repositories cannot meet the per-repository quota. The total is not the constraint; the distribution is.
Side finding
invalid_record_identityis non-zero — 3 in this repository, 2 in agent-operator-score. MalformedRecord-Idvalues exist in real history. Separate from #535 (nothing mints one at all), and worth handling together.7 cases pass; both typechecks clean; two builds leave
distunchanged; both verifiers pass. Reads records through the shipping query engine rather than a reimplementation.