Bring the release docs in line with the automated flow

[MattFisher]

Why

An audit of the release docs after v1.10.0 went out through the new flow, and after inspect-evals-lint 0.10.0 and inspect_dataset 0.5.0 published through it. The repos' own docs are accurate. These gaps and inaccuracies are all in the template.

What changes

• RELEASING.md is now generated for every project, not only PyPI libraries. The PyPI setup, the publish step and the publish failures appear only where they apply (3a51c55, from the review):

  • It gains an If a step fails section: re-running Prepare release, CI held for approval, re-running Release on merge, and a failed publish. A failed publish means re-dispatching publish.yml, or bumping if the upload already landed, since PyPI never takes a version twice.
  • "Sanity checks before tagging" becomes "before releasing", since tagging is now automatic.

• template/changelog.d/TEMPLATE.md says Prepare release collects fragments.

• README:

  • The manual update command is now uvx copier update --trust.
  • "Keeping projects up to date" lists the release workflows among those that moving v1 delivers.
  • "Releasing this template" covers recovery, including starting bump-v1 by hand on the newest tag.

• CHANGELOG preamble and bump-v1.yml comments. Both said copier reads the template's version with git describe --tags. It reads it through dunamai, which takes the commit's newest tag by date: an annotated tag's date is when it was tagged, and a lightweight tag's is its commit's. I checked this in copier 9.18.2's _template.py and dunamai's tag sort. The advice stands (annotate release tags, keep v1 lightweight); only the reason changes. The preamble also describes releases going through Prepare template release.

• One fragment.

• Review fixes (3a51c55):

  • the recovery steps are rewritten against the workflow code;
  • the bump-v1 command passes -R;
  • every copier update command has --trust;
  • the reusable workflows' permissions and the v1 pin list are corrected;
  • the dunamai tie rule is stated exactly.

Testing

All three variants render, and their own hooks pass on RELEASING.md, README.md and .copier-answers.yml: an app, a PyPI library, and a library without PyPI. Only the PyPI library gets the publishing parts. Template CI's render step, which now expects RELEASING.md for apps too, and the rendered projects' full hook runs pass locally. The fragment doesn't quote a scriv marker, so Prepare template release will accept it.

🤖 Generated with Claude Code

[MattFisher]

Fresh-eyes review of the release docs. The direction is right and most claims hold. The dunamai correction is accurate. A few recovery steps would mislead a maintainer, though, and some likely failures aren't covered. Most important first.

Findings on this PR

1. A re-dispatched publish can't pick up a fix made in the repo. template/{% if publish_to_pypi %}RELEASING.md{% endif %}.jinja line 32. gh workflow run publish.yml --ref vX.Y.Z runs publish.yml as it is at the tag, and builds the code at the tag. That works for a PyPI-side fix, like the trusted publisher example. It does not work if the fix is in publish.yml, pyproject.toml or the build. Fix: "If the fix is on PyPI or in the pypi environment, re-run the failed publish run, or run gh workflow run publish.yml --ref vX.Y.Z. If the fix is in the repo, merge it and cut a new release." Also, bumping needs a new fragment first. Prepare release refuses when changelog.d/ has no entries (prepare-release.yml line 111).

2. The bump-v1 dispatch command targets the wrong repo from this repo's usual checkout. README.md line 254. A clone with an upstream remote makes gh resolve to MattFisher/python-project-template. I checked: gh repo view in a fresh clone of this PR answers MattFisher/python-project-template, and that fork has a Bump v1 workflow too. Fix: gh workflow run bump-v1.yml -R Generality-Labs/python-project-template --ref vX.Y.Z. The same line says "the generated projects' RELEASING.md notes apply here too". This repo has no RELEASING.md, only the .jinja source, and its publish bullet doesn't apply here. Fix: link the template file, or list the two bullets that apply (Prepare release, Release on merge).

3. Failure modes a maintainer is likely to hit that the new section misses. RELEASING.md.jinja lines 27-32.

• The Actions setting is off. This is the most likely first-run failure, and the workflow's own comment names it (prepare-release.yml lines 206-207). The branch is pushed, then gh pr create fails. Fix: turn the setting on and re-run, since the force-push replaces the branch.
• Release on merge refuses, and a re-run refuses again. Three checks are tied to the merge commit: the branch name, pyproject.toml disagreeing with the branch, and a missing ## [X.Y.Z] section (release-on-merge.yml lines 64-86). A stale tag on another commit is the same (lines 100-104). Line 31 says "Re-run" and stops. Fix: say a re-run only helps a transient error. For a stale tag that never reached PyPI, delete it and re-run. Otherwise fix main and cut a new release.
• Publishing waits rather than fails. Line 13 suggests adding a required reviewer to pypi. Then publish sits at "Waiting" until someone clicks Review deployments on the run. Fix: one bullet.
• main moved while the release PR was open. The merge commit gets the tag, so a PR merged in between ships in this release. Its fragment stays in changelog.d/ and is described in the next release. Fix: "Merge the release PR soon. If other PRs land first, close it and run Prepare release again."
• The refusal list in line 29 is partial. It also refuses a run from another branch, no fragments, a fragment with entries but no ### heading, and a quoted scriv marker (prepare-release.yml lines 68-111). "Bump further" also fits only the existing-tag case. With a release PR open, a bigger bump opens a second release PR next to the first. Fix: "Merge or close the open pull request. If the tag exists, choose a larger bump."

4. Projects without RELEASING.md get no recovery notes. Apps and unpublished libraries get prepare-release.yml and release-on-merge.yml, but RELEASING.md only comes with publish_to_pypi. Their only release doc is template/README.md.jinja line 14. A project that deleted RELEASING.md also never gets it back. I tested this: I rendered v1.10.0, deleted RELEASING.md, and ran copier update --vcs-ref=HEAD to this PR. Only TEMPLATE.md and the answers file changed. Fix: put the first three bullets in the generated README too, or link them.

5. dunamai tie wording. bump-v1.yml lines 151-152 say "they tie, dunamai can answer v1". On a tie it always answers v1. dunamai sorts by (topo distance, date) with a stable sort over git for-each-ref output, which is in refname order, and refs/tags/v1 sorts before refs/tags/v1.X.Y. I confirmed this with dunamai 1.26.2 (copier 9.18.2) in a scratch repo:

• v1 and v1.10.0 both lightweight: v1.
• Release tag annotated later, v1 lightweight: v1.10.0.
• Both annotated, v1 newer: v1.

Fix: "With both lightweight they tie, and dunamai answers v1". The rest of the explanation is right. copier's Template.version calls Version.from_git(pattern=Pattern.DefaultUnprefixed). best_date() uses the tagger date when there is one, and otherwise the commit date. v1.10.0 has a real tagger date, from the API annotation in bump-v1.

6. CHANGELOG preamble. CHANGELOG.md line 9 still lists only python-ci.yml@v1 and node-ci.yml@v1. The README now adds the release workflows. Both miss template-update.yml@v1, which the scaffolded caller also pins (README.md lines 224-226 too). Line 10 says bump-v1 annotates the tag "if publishing left it lightweight". In the automated flow, release-on-merge always creates a lightweight tag (release-on-merge.yml line 107), so bump-v1 annotates on every release. Fix: say so, and keep the UI case as the second reason.

7. Hand paths.

• README.md line 256: a release published by hand starts bump-v1. But bump-v1 refuses unless CHANGELOG.md at that commit has the ## [X.Y.Z] section. By then the tag and release already exist. Fix: "Collect the fragments by hand (uvx --from scriv scriv collect --version X.Y.Z) and merge that first."
• RELEASING.md.jinja line 25: the by-hand path creates no GitHub release. inspect_dataset's README adds gh release create vX.Y.Z, which is worth copying. Also, if the PR's branch is named release/vX.Y.Z, Release on merge tags and publishes on its own, and the manual tag push is rejected. Fix: say to use another branch name, or to skip the manual tag.

8. Small, pre-existing, nearby.

• README.md line 152: "the reusable workflows declare none of their own". That's false for prepare-release.yml, which declares job-level contents: write and pull-requests: write (lines 51-53).
• README.md line 246 ends in a colon, but a paragraph follows instead of a list.
• uvx copier update without --trust is still in template/.github/workflows/{% if use_template_update %}template-update.yml{% endif %}.jinja line 8, which is scaffolded into every project, and in the copier.yml line 117 comment.

9. Changelog fragment: fine. It quotes no scriv marker (grep finds none). scriv collect --version 1.10.1 with scriv 1.8.0 renders both bullets under ### Fixed, so auto picks patch. Optional: it doesn't mention the new recovery paragraph in "Releasing this template". "Explain why release tags must be annotated correctly" would read better as "give the right reason release tags are annotated".

Verified OK

• Re-running Prepare release is safe. The version is recomputed from unchanged main, and the push is --force.
• Release CI is held for approval. The v1.10.0 release PR (Release v1.10.0 #9) was opened by app/github-actions, and Template CI ran on it. A push from a person starts CI normally.
• Release on merge skips a tag already on the merge commit and an existing release, and refuses a tag on another commit.
• publish.yml has workflow_dispatch and refuses any ref that isn't a v* tag. PyPI rejects a re-upload of an existing file (known PyPI behaviour, not tested here).
• The bump-v1 dispatch guard accepts only ^v1\.[0-9]+\.[0-9]+$ when it is the newest by sort -V. Note that a dispatch on a branch skips the job silently, through the job if, rather than failing.
• Two concurrent Prepare release runs share the prepare-release concurrency group in both the template and scaffolded callers. The second waits, then refuses because the PR is open.
• A release PR closed without merging leaves its branch behind. That's harmless, since the next run force-pushes it, and Release on merge skips unmerged PRs.
• The Actions PR setting is on for this repo, inspect-evals-lint and inspect_dataset (can_approve_pull_request_reviews: true).
• A rendered library (copier copy --vcs-ref=HEAD) gives RELEASING.md, README.md and TEMPLATE.md as in the diff.

Follow-ups in other repos (not this PR)

Generality-Labs/inspect-evals-lint (main at 7790a8c)

• RELEASING.md line 27 still says "Sanity checks before tagging" and has no "If a step fails" section. The next template-update PR after this release brings both. I tested copier update from v1.10.0 to this PR over lint's customised RELEASING.md, and it merges cleanly and keeps lint's step 2.
• RELEASING.md stops at PyPI. inspect-evals-actions (pyproject.toml) and inspect-evals-template pin this package. If bumping those pins is part of a release, add it as a last step.

Generality-Labs/inspect_dataset (main at 6cb54ee)

• It has no RELEASING.md and never had one, though publish_to_pypi: true. As tested above, copier update won't add it, so the failure notes will never arrive. Add them to README "Releasing", or restore RELEASING.md.
• README "Releasing" (line 210) skips creating the pypi GitHub environment. It also doesn't mention the Actions setting, which is only in the Development paragraph (line 227).
• Line 214: "auto picks minor for new or changed features, patch for fixes only". It also picks minor for Deprecated and Removed, and never picks major. Use the template's wording.

Posted by Claude Code on Matt's behalf.

[MattFisher]

Thanks. Actioned in 3a51c55:

1. Failed publish. Rewritten. Re-dispatching publish.yml rebuilds the workflow and code as they are at the tag, so it only helps when the cause is outside the repo. A fix in the repo needs a new release with a fragment, and so does a part-way upload.

2. The bump-v1 command passes -R Generality-Labs/python-project-template, with the reason. The pointer goes to template/RELEASING.md.jinja.

3. Missing failure modes. Added:

   • the Actions setting, including when it's greyed out at org level;
   • every refusal Prepare release can give;
   • the four release-on-merge refusals tied to the merge commit, which a re-run can't clear;
   • a publish waiting on an environment reviewer;
   • main moving under an open release PR.

   "Bump further" now applies only to an existing tag.

4. RELEASING.md for every project. The file is now template/RELEASING.md.jinja, with the PyPI parts conditional. Template CI expects it for apps too. A project that deleted it won't get it back from copier update, as you found. The fragment says so, and inspect_dataset will get one directly.

5. dunamai tie. It now says the tie breaks by name and picks v1.

6. Preamble. The pin list adds template-update.yml@v1 and the release workflows. bump-v1's comment says it annotates every release.

7. Hand paths. A hand-published template release needs its changelog collected and merged first. The generated manual path includes gh release create.

8. Nearby fixes. The README's permissions sentence is corrected, the stray colon is gone, and the answers header and template-update comment have --trust.

The follow-ups for inspect-evals-lint and inspect_dataset will be separate PRs after this releases.

Posted by Claude Code on Matt's behalf.