Add version and upstream license links to THIRD_PARTY_NOTICES.md - #1986
Draft
abrarshivani wants to merge 4 commits into
Draft
Add version and upstream license links to THIRD_PARTY_NOTICES.md#1986abrarshivani wants to merge 4 commits into
abrarshivani wants to merge 4 commits into
Conversation
THIRD_PARTY_NOTICES.md now carries a Version and a Location column instead of the Dependency column. Location links to the license file in the dependency's own upstream repository, pinned to the version we redistribute: | Package | Version | License | Location | |---------|---------|---------|----------| | `github.com/NVIDIA/go-nvml/pkg` | v0.13.3-1 | Apache-2.0 | [LICENSE](https://github.com/NVIDIA/go-nvml/blob/v0.13.3-1/LICENSE) | Version was dropped in 93ebcb6 because the notices were meant to identify dependencies rather than a build. That is reversed here: a notices file that does not say which version it describes cannot be matched to a release, and a link into upstream needs a ref to point at. The churn per bump is one index row and two bullets. Every URL was verified by fetching it and comparing its sha256 against the copy under vendor/. A URL that does not match is never written, so no link is dead and none points at the wrong license. 96 URLs across 72 modules. vendor/ gives the module, the version and the license file names for free, but not the upstream repository: cyphar.com/go-pathrs lives at github.com/cyphar/go-pathrs, k8s.io/api at github.com/kubernetes/api. Two committed maps carry that instead, both machine-generated. hack/module-repos.tsv maps module to repository, resolved from the Go module proxy's Origin, then the go-import meta tag that go get itself uses, then the github.com/<org>/<repo> path shape. It is keyed by module and not by version, so a bump does not invalidate it. hack/license-urls.tsv maps module, version and license path to a verified URL. A row is written only when the bytes at that URL hash identically to the vendored copy. Probing for a 200 is not enough: it cannot tell a correct link from one that returns 200 for the wrong license. Both are produced out of band by 'make third-party-notices-repos' and 'make third-party-notices-urls', which need network. 'make third-party-notices' reads them offline, so 'make check-third-party-notices' stays hermetic. License files are now enumerated from vendor/ rather than from the go-licenses save output, because that output keeps only the one file it classifies as the license per package and drops the rest. That recovers nine PATENTS files, the LICENSE.libyaml that go.yaml.in/yaml/v2 ships alongside its Apache-2.0 LICENSE, and several AUTHORS and NOTICE files. hack/license-overrides.tsv corrects the License column where go-licenses under-reports it. Three modules ship one file holding two licenses, so they read Apache-2.0 / MIT from the override. It is curated by hand rather than detected, because scanning license text cannot tell BSD-2-Clause from BSD-3-Clause and a wrong addition is worse than an omission. Generation fails if an override names a package no longer in the index, so it cannot rot unnoticed. third-party-notices-links.yaml re-verifies every URL weekly. Links are proven correct when written, but upstream can retag or archive a repository afterwards and no offline gate can see that. A version change now needs two commands, because a verified URL contains the version: make third-party-notices-urls # network make third-party-notices # offline Dependabot cannot do the first on its own; its bump job needs wiring, or a human runs it. That is the direct cost of requiring every link to be verified rather than derived. 'make test-tools' runs the new bash suites, 54 assertions across two files, and is part of CHECK_TARGETS so it runs in CI. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
This was referenced Aug 27, 2026
Draft
go.googlesource.com rate-limits the per-file loop the verifier drives, returning 503 and 429 under rapid sequential requests. remote_sha fetched each blob with a single curl, so a rate-limited response made that license file count as unverified and, because the gate is fail-closed, killed the run. Both codes were observed against golang.org/x and google.golang.org/protobuf while regenerating release notices, and both succeeded on a later attempt. The weekly link check drives the same loop, so it would have reported link rot that had not happened. The retry distinguishes a transient failure from a permanent one. curl -f collapses every failure into exit 22, so the status is read explicitly and a 404 still fails on the first request: this loops over several candidate refs and paths per file, and retrying a genuine miss would multiply the runtime of the whole run. fetch_retry moves into license-url-lib.sh as http_fetch_to_file so both resolvers share one retry and status policy. It writes to a file rather than returning the body, because command substitution strips trailing newlines, which would change the sha256 of every license file that ends in one. Successive requests to one host are also spaced, so the limiter is less likely to trip at all. Re-resolving module-repos.tsv against the live proxy after the change produces a byte-identical map, so sharing the fetcher is not a behaviour change. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The header described the generator's own verification process: that every link was fetched and compared "against the copy vendored here". That is build process, not attribution, and "here" only resolves for someone reading the file inside a checkout. This document is meant to be redistributable. What a reader can act on is the guarantee itself, so keep only that: the Location links to the license file at the version listed, and serves the text reproduced below. How that is enforced belongs in the tooling and the pull request, not in the notices. The sentence about modules the commands do not link was saying again what the scope paragraph already says, so the only new part, why those modules are absent, moves up into that paragraph. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The clause restated the verification the generator already enforces, and a released notices file has nothing standing behind it: the weekly link check runs against the repository, not against a document already shipped. If upstream retags, the sentence becomes an assertion no one is checking. The license text sits directly under the link, so a reader can see the two agree without being told. What is worth saying is that the link is pinned to the listed version and points upstream rather than at our own copy, and the rest of the sentence still says that. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
VersionandLocationcolumns toTHIRD_PARTY_NOTICES.md, replacing theDependencycolumn.
Locationlinks to the license file in the dependency's own upstream repository, pinnedto the version we redistribute:
github.com/NVIDIA/go-nvml/pkgThis is the same change as NVIDIA/gpu-operator#2794, applied here.
Version was dropped in 93ebcb6 (#1955) on the grounds that the notices identify dependencies
and their licenses, not an exact build. That is reversed here, and I want to flag it rather than
bury it. Two reasons: a notices file that does not say which version it describes cannot be
matched to a release, and a link into upstream needs a ref to point at. The churn is real but
small, one index row and two bullets per bump.
Every URL in the file was verified by fetching it and comparing its sha256 against the copy under
vendor/. A URL that does not match is never written, so there are no dead links and none pointat the wrong license. 96 URLs across 72 modules.
That includes the secondary license files a module ships alongside its main one, which are easy
to lose: nine
golang.org/x/*andgoogle.golang.org/protobufmodules each carryPATENTS, andgo.yaml.in/yaml/v2carriesLICENSE.libyaml(MIT, for the embedded libyaml port) on top of itsApache-2.0
LICENSE. Twelve license files that were previously dropped are now reproduced.What to review
Hand-written, 963 lines:
hack/verify-license-urls.shhack/generate-third-party-notices_test.shhack/resolve-module-repos.shhack/license-url-lib.shhack/license-url-lib_test.sh.github/workflows/third-party-notices-links.yamlhack/test-helpers.shhack/generate-third-party-notices.shhack/license-overrides.tsvMakefile.github/workflows/third-party-notices-check.yamlGenerated, do not read:
THIRD_PARTY_NOTICES.md,hack/license-urls.tsv(101),hack/module-repos.tsv(76).hack/verify-license-urls.shis the one to read. Everything else supports it.How it works
vendor/gives the module, the version and the license file names for free, but it does notrecord the upstream repository.
cyphar.com/go-pathrsactually lives atgithub.com/cyphar/go-pathrs,k8s.io/apiatgithub.com/kubernetes/api. Scraping that out ofvendored sources is wrong more often than right, so two committed maps carry it instead, both
machine-generated:
hack/module-repos.tsvmaps module to repository. Resolution is the Go module proxy'sOrigin,then the
go-importmeta tag thatgo getitself uses, then thegithub.com/<org>/<repo>pathshape. Nothing is hand-written. It is keyed by module and not by version, so a bump does not
invalidate it.
hack/license-urls.tsvmaps module, version and license path to a verified URL. A row is writtenonly when the bytes at that URL hash identically to the vendored copy. Probing for a 200 is not
enough: it cannot tell a correct link from one that returns 200 for the wrong license.
Both are produced out of band by
make third-party-notices-reposandmake third-party-notices-urls, which need network.make third-party-noticesreads them offline, somake check-third-party-noticesstays hermetic and cannot flap on a proxy that withholdsOrigin.Scope is unchanged: the verifier reuses the generator's own collection, so it covers the packages
go-licensesattributes to./cmd/.... Modules vendored only fortests/or for build toolingare not redistributed and are not listed.
License files are now enumerated from
vendor/rather than from thego-licenses saveoutput,because that output keeps only the one file it classifies as the license per package and drops
the rest.
hack/license-overrides.tsvcorrects the License column where go-licenses under-reports it.Three modules ship one file holding two licenses:
gopkg.in/yaml.v3andgo.yaml.in/yaml/v3each carry a full-text MIT section plus a short-form Apache-2.0 grant, and
go.yaml.in/yaml/v2carries
LICENSE(Apache-2.0) alongsideLICENSE.libyaml(MIT). go-licenses classifies each asa single license, so all three read
Apache-2.0 / MITfrom the override instead. It is curatedby hand rather than detected, because scanning license text cannot tell BSD-2-Clause from
BSD-3-Clause and a wrong addition to this file is worse than an omission. Generation fails if an
override names a package that is no longer in the index, so the file cannot rot unnoticed.
.github/workflows/third-party-notices-links.yamlre-verifies every URL weekly. Links are provencorrect when written, but upstream can retag or archive a repository afterwards and no offline
gate can see that.
Note for dependency bumps
A version change now needs two commands, because a verified URL contains the version:
Dependabot cannot do the first on its own. Its bump job needs wiring, or a human runs it. This is
the direct cost of requiring every link to be verified rather than derived.
Testing
make test-toolsruns the new bash suites, 54 assertions across two files, and is now part ofCHECK_TARGETSso it runs in CI.Verified by hand:
HEADor branch refsvendor/modules.txtgo-nvmlrows fromhack/license-urls.tsvmakesmake third-party-noticesexit non-zero naming that module, and alicense-overrides.tsvrowfor a package not in the index does the same