Name the authorization the corpus rests on, and the gap it leaves - #534
Merged
Conversation
CDEB reads real decisions out of real repositories and publishes what it finds, so §3.3 requires ownership and authorship to be disclosed and CDEB-10 requires a privacy authorization to exist before the corpus is sealed. This is that authorization, and each repository's freeze entry points at it. A public repository is not consent to be studied. Visibility and authorization are different things, and the second one has to be named by the person who holds it rather than inferred from the first. The owner granted both the use of the source data and the transparent publication of the result. Two boundaries are written down rather than left to be discovered. The owner cannot authorize on behalf of upstream contributors, and one repository is a fork whose recent history is mostly other people's commits, so a task drawn from it has to rest on a record the owner authored. And the CommitLore repository stays excluded whatever the authorization says, because that exclusion is methodological rather than a permission. The arithmetic is stated instead of deferred: §3.3 wants five repositories, the exclusion leaves four authorized, and §3.2 rule 5 already says what may not be done about a shortfall. Limit: this records that authorization was granted, not that every candidate drawn under it was authored by the grantor -- the candidate registry checks that per record, and nothing here substitutes for it Ruled-out: treating public visibility as consent to study | a repository being readable says nothing about whether its owner agreed to be measured and named Blast: local Undo: easy Certainty: firm Verified: both bench verifiers pass and the package typecheck is clean Provenance: authored Record-Id: r-cdebauth1
CommitLore — record lintTrailers: clean — 1 commit in Limits (1)
Ruled out (1)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CDEB-10 lists privacy authorization among the things that must exist before the corpus is sealed, and §3.3 requires ownership and authorship to be disclosed. This is that record; each repository's freeze entry (§6.1
source_authorization_id) points at it.A public repository is not consent to be studied. Visibility and authorization are different things, and the second has to be named by the person who holds it rather than inferred from the first. This was raised by another session working in one of the source repositories, and it was the right challenge — the owner has now granted both the use of the source data and transparent publication of the result.
Two boundaries written down rather than discovered later
Upstream commits are not the owner's to authorize. One repository is a fork whose recent history is mostly other people's commits — 1,052 in one recent week. A task drawn from it has to rest on a record the owner authored, checked per record in the candidate registry rather than assumed.
The CommitLore repository stays excluded whatever the authorization says. That exclusion is methodological — the product's own decisions are not independent of the product being measured — so permission does not touch it.
Tier B, stated in the document
Every authorized repository is operated by the same person who authored the benchmark and the product. That is §3.3's Tier B, and the attached rule is not optional: the result must read "author-operated repositories" and must not describe itself as independent external validation.
The arithmetic, stated instead of deferred
§3.3 wants five repositories. The exclusion leaves four authorized. §3.2 rule 5 already says what may not be done about a shortfall — the study stops rather than lowering the bar or creating records to fill it.
Measured today,
Ruled-outtrailer counts across the authorized repositories are 49, 6, 6 and 0, against six tasks required per repository. The corpus is not ready, and no amendment to §3.3 fixes that — changing the composition rule while looking at which shape fits the available data is precisely the move rule 5 exists to prevent. The answer is accumulation, and the next step is a candidate registry that reports eligibility continuously so the moment it becomes reachable is a number rather than a guess.Both bench verifiers pass; package typecheck clean. Documentation only — no code path changes.