fix: license check ignores transitive dependencies in dependency reports

[amaechler]

Description

Since #20148 (Apache parent 25 → 39), check-licenses.py only reads the direct compile dependencies from each module's dependencies.html. Transitive dependencies are skipped entirely, so the packaging-check job compares 68 reported dependencies instead of ~420 against licenses.yaml. A version bump without a matching licenses.yaml update now passes CI. I noticed that on #20520 which bumps at.yawk.lz4:lz4-java to 1.11.4 without touching licenses.yaml, and the check stays green.

Transitive dependencies need to be covered: the binary tarball bundles them (lib/ and pull-deps both resolve them), and ASF policy treats bundled transitive dependencies the same as first-order ones for LICENSE and NOTICE (release policy).

More details from Claude analysis

Fixed the dependency report parser

The newer maven-project-info-reports-plugin output marks sections with <a id="..."> anchors. The parsing path added in #20148 had two problems:

• It only matched Project_Dependencies_compile, not Project_Transitive_Dependencies_compile.
• It only looked for the anchor in state none. After the first table, the <h2>test</h2> heading moved the state machine into the legacy h2_end state, where it waited for an <h3> that never comes.

The anchor check now runs in any state, matches both compile anchors, and jumps straight to waiting for the table. This also removes the two intermediate modern_compile_* states. I also added a guard: the parser counts the compile tables it enters and finishes, and raises if any are left unfinished, so a future format change fails loudly instead of silently shrinking coverage.

Three license names that the reports use but the alias table did not know (Go License, MIT license, The BSD 2-Clause License) are now mapped, so the license-name comparison runs for re2j, animal-sniffer-annotations and stax2-api.

Fixed the licenses.yaml drift the check now finds

With the parser fixed, master has 15 dependencies that are reported but not registered:

• Maven resolver upgrade from build(deps): upgrade org.apache.maven.resolver from 1.3.1 to 2.0.23 #20386, only partly recorded: Apache Maven 3.6.0 → 3.9.16, new entries for maven-model, maven-model-builder, maven-resolver-provider, maven-resolver-named-locks and maven-resolver-transport-file (notice text taken from each jar's META-INF/NOTICE), and plexus-interpolation 1.25 → 1.29.
• Confluent in druid-protobuf-extensions: 8.3.1 → 8.3.2 and kafka-clients 8.3.1-ccs → 8.3.2-ccs.
• joda-time 2.14.4, jackson-jq 1.6.5, slf4j-api 2.0.20, auto-value-annotations 1.11.1.

Verification

On reports generated locally from current master with generate-license-dependency-reports.py:

	Before	After
Reported dependencies	68	423
Unchecked (warning only)	588	237
Clean licenses.yaml	passes	passes
lz4-java registered at the wrong version	passes	fails, naming lz4-java

The parsed rows match every compile-table row in all 31 reports, including the ones with a Classifier column. The binary LICENSE and NOTICE generators still run cleanly on the updated licenses.yaml.

Not in this PR

• Runtime-scope dependencies (Project_*_runtime tables) ship in the distribution but have never been checked, also before build(deps): bump org.apache:apache from 25 to 39 #20148. Adding them reports 28 more missing or outdated entries (for example log4j-slf4j2-impl, hibernate-validator, janino, hadoop-client-runtime). That needs per-artifact license and notice review, so I will send it as a separate PR.
• About 40 licenses.yaml entries are registered at an older version than the reports show (for example Guava 32.1.3-jre, the Confluent 6.2.15 entries, kafka-clients 6.2.12-ccs and 5.5.1-ccs under druid-avro-extensions). They only produce warnings. Some may ship from hadoop-dependencies/, so removing them needs a check against the built tarball.

---

This PR has:

• been self-reviewed.
• added or updated version, license, or notice information in licenses.yaml

[FrankChen021]

🟢 Approval recommended

No actionable issues found in this review. The parser now recognizes both direct and transitive compile tables and fails if a recognized compile table is left unfinished; the registry updates cover the newly checked artifacts and the added license aliases normalize the report names used by those artifacts.

Reviewed 2 of 2 changed files.

Validation: git diff --check passed; Python syntax compilation, a parser smoke test covering direct and transitive tables including a classifier column, and a Ruby YAML parse check all passed. No full build or test suite was run.

---

This is an automated review by Codex GPT-5.6 Luna(Max)

[FrankChen021]

Thanks for the fix.

I think this problem was there before #20148 . Previously I noticed this in some PRs raised by the dependabot, but I didn't deep dive it as the automation I used to triage these PRs now can help us fix this license mismatch, for example #20510