Add more details to THIRD_PARTY.md

DEVELOPER_GUIDE.md

@@ -136,9 +136,8 @@ Always commit both `pyproject.toml` and `uv.lock` together.
 
 ### Third-party license attribution
 
-`THIRD_PARTY.md` lists the runtime dependencies with their licenses, authors and URLs. Regenerate
-it whenever the dependencies change, and before a release, so the `NOTICE` attribution can be
-checked against it:
+`THIRD_PARTY.md` inventories the runtime dependencies with their licenses, authors and URLs.
+Regenerate it whenever the dependencies change:
 
 ```bash
 make licenses
@@ -148,6 +147,19 @@ The target runs [pip-licenses](https://github.com/raimon49/pip-licenses) in an i
 environment that contains only the runtime dependencies, so the develop and test extras are not
 listed. Commit the updated `THIRD_PARTY.md` together with `pyproject.toml` and `uv.lock`.
 
+Only the table below the `GENERATED-TABLE-BELOW` marker is replaced. Everything above it is
+maintained by hand and records what the generated table cannot: corrections where a package
+declares only a coarse classifier, the elections made for dual-licensed packages, the Category B
+dependencies, and which dependencies ship a `NOTICE` of their own. **When a dependency is added,
+removed or upgraded, review those sections too** — a new dependency may need a correction or an
+election, and a dropped one leaves a stale entry behind.
+
+`THIRD_PARTY.md` is informational and is not the licensing compliance artifact. The source
+release, the sdist and the PyPI wheel resolve these packages with pip rather than bundling them,
+so per the ASF [licensing howto](https://infra.apache.org/licensing-howto.html) they need no
+dependency entries in `LICENSE` or `NOTICE`. The Docker image does bundle them and therefore needs
+an aggregated `LICENSE`; `THIRD_PARTY.md` is the input for assembling it.
+
 ### Avoiding secrets in commits
 
 Install [git-secrets](https://github.com/awslabs/git-secrets) to prevent

Makefile

@@ -44,7 +44,7 @@ install: build
 	rm -rf dist
 
 clean:
-	rm -rf .benchmarks .eggs .tox .benchmark_it .cache build dist *.egg-info logs junit-py*.xml *.whl NOTICE.txt
+	rm -rf .benchmarks .eggs .tox .benchmark_it .cache build dist *.egg-info logs junit-py*.xml *.whl NOTICE.txt THIRD_PARTY.table THIRD_PARTY.next
 
 # Avoid conflicts between .pyc/pycache related files created by local Python interpreters and other interpreters in Docker
 python-caches-clean:
@@ -74,10 +74,21 @@ coverage: develop
 	uv run coverage run -m pytest tests/
 	uv run coverage html
 
-# Regenerates THIRD_PARTY.md from the runtime dependencies. --isolated runs in a fresh environment
+# Regenerates the dependency table in THIRD_PARTY.md from the runtime dependencies, leaving the
+# hand-maintained sections above the marker untouched. --isolated runs in a fresh environment
 # without the develop/test extras, so tools like pytest and ruff stay out of the list.
+# --with-system keeps setuptools and wheel, which pip-licenses otherwise drops even though wheel
+# is a declared runtime dependency; pip-licenses and its own dependencies are ignored instead.
 licenses: check-uv
-	uv run --isolated --with pip-licenses pip-licenses --with-authors --with-urls --format=markdown --ignore-packages solr-orbit --output-file THIRD_PARTY.md
+	@grep -q GENERATED-TABLE-BELOW THIRD_PARTY.md || { echo "THIRD_PARTY.md has no GENERATED-TABLE-BELOW marker; refusing to overwrite it" >&2; exit 1; }
+	uv run --isolated --with pip-licenses pip-licenses --with-authors --with-urls --with-system \
+	    --format=markdown --ignore-packages solr-orbit pip-licenses prettytable wcwidth \
+	    --output-file THIRD_PARTY.table
+	@awk '{ print } index($$0, "GENERATED-TABLE-BELOW") { exit }' THIRD_PARTY.md > THIRD_PARTY.next
+	@echo >> THIRD_PARTY.next
+	@cat THIRD_PARTY.table >> THIRD_PARTY.next
+	@mv THIRD_PARTY.next THIRD_PARTY.md
+	@rm -f THIRD_PARTY.table
 
 release-checks:
 	./release-checks.sh $(release_version) $(next_version)