Keep scaffolded changelogs as scriv fragments

[MattFisher]

Why

Scaffolded projects keep a hand-edited ## [Unreleased] section in CHANGELOG.md. Every PR adds lines at the same place, so any two open PRs conflict as soon as one merges. inspect-evals-lint has seven open PRs that each conflict with all the others on this one file.

inspect_evals solves this with scriv: one fragment file per PR, collected into the changelog at release time. This brings the same setup into the template.

What changes in a scaffolded project

• pyproject.toml:
  • scriv>=1.8 in the dev group.
  • [tool.scriv] with Keep a Changelog categories and the existing ## [X.Y.Z] - YYYY-MM-DD heading format.
  • The version is read from src/<pkg>/__init__.py for libraries and from pyproject.toml for apps, so scriv collect needs no arguments.
• changelog.d/TEMPLATE.md: every category, commented out. uv run scriv create copies it.
• CHANGELOG.md: <!-- scriv-insert-here --> replaces ## [Unreleased].
• RELEASING.md step 2 becomes uv run scriv collect. The README's Development section says how to add a fragment.

CHANGELOG.md is added to _skip_if_exists. A project's changelog is its own once scaffolded. Without the skip, any template change to it would 3-way merge into a long project history and conflict. The downside is that copier update won't add the marker to existing projects. The template CHANGELOG's upgrade note covers the one manual step.

Testing

• Template CI renders both variants, collects a fragment with each scaffold's own config, and checks the heading, the bullet, the deleted fragment and the kept TEMPLATE.md. The whole step passes locally.
• In a rendered library, scriv create produces a fragment from the template, and the collected CHANGELOG.md and TEMPLATE.md pass mdformat with the scaffold's .mdformat.toml.

Rollout

After this is released (v1.9.0), inspect-evals-lint and inspect_dataset adopt it by copier update in their own PRs. Those PRs also move their current [Unreleased] entries into a fragment.

Unrelated, noticed while here

Under set -e, a command negated with ! never fails the script, so the existing ! grep -q … checks in template-ci.yml (typos, frontend, rulesets) can't fail the step. test ! -e is unaffected. The new check uses ! grep … || exit 1. The existing ones are unchanged here.

🤖 Generated with Claude Code

[MattFisher]

Review of 399a60e

The core change works. I found one real problem for a named adopter, one undocumented side effect of _skip_if_exists, and some smaller doc gaps. Ranked most important first.

1. The library branch reads the wrong version for inspect_dataset

Where: template/pyproject.toml.jinja lines 57-58.

What: Every project_kind: library gets version = "literal: src/<pkg>/__init__.py: __version__". That assumes the hatch layout, where __init__.py is the source of truth. inspect_dataset is answered as a library, but it builds with uv_build and keeps a static version = "0.4.0" in pyproject.toml. Its src/inspect_dataset/__init__.py still says __version__ = "0.2.0".

Failure scenario: inspect_dataset adopts this by copier update, as the Rollout section plans. At its next release, uv run scriv collect writes ## [0.2.0] - <date> above the existing 0.4.0 history. Nothing errors, so it is easy to tag and publish with the wrong heading.

Fix: In the inspect_dataset adoption PR, set its [tool.scriv] version to literal: pyproject.toml: project.version (copier keeps that local edit on later updates) and fix or remove the stale __version__. In the template, add one line to the upgrade note: check that [tool.scriv] version points at the file your build backend actually reads. inspect-evals-lint is fine. It uses [tool.hatch.version] path = "src/inspect_evals_lint/__init__.py".

How verified: Ran: in a rendered library with __version__ set to 0.2.0, scriv collect wrote ## [0.2.0]. Read: inspect_dataset's pyproject.toml, __init__.py and .copier-answers.yml on its main branch.

2. _skip_if_exists brings back a deleted CHANGELOG.md on every update

Where: copier.yml lines 7-8, and the template CHANGELOG.md line 20 ("copier update no longer touches a project's changelog").

What: Copier documents _skip_if_exists as "skip if it exists, but always be present". If the file is missing at update time, copier creates it again. Without the skip, a deleted file stays deleted.

Failure scenario: A project removes CHANGELOG.md, or renames it to docs/changelog.md. Every copier update creates a fresh CHANGELOG.md again. With template-update.yml on, the weekly PR proposes it again each time someone closes it.

Fix: Say it in the template CHANGELOG entry. For example: "copier update leaves an existing CHANGELOG.md alone, but creates one if it is missing." A project that keeps its changelog elsewhere can run copier update --exclude CHANGELOG.md by hand, but the shared template-update.yml takes no such input, so the scheduled run would keep proposing it. I don't think the skip itself is wrong. Your reason for it holds.

How verified: Ran: scaffolded a library at 95fcf12 (pre-PR main), deleted CHANGELOG.md, committed, then ran copier update --vcs-ref=HEAD. With this PR, CHANGELOG.md came back as an untracked file with the scriv marker. With a local commit that drops _skip_if_exists, the same update left it deleted.

Also ran: a project with a hand-edited changelog (an [Unreleased] entry plus two releases) and an untouched scaffold changelog. In both, update left CHANGELOG.md byte-for-byte unchanged and added [tool.scriv] and changelog.d/TEMPLATE.md cleanly. So the PR's main claim holds.

3. The manual upgrade step fails loudly, but with a confusing message

Where: template CHANGELOG.md line 20 (the upgrade note).

What: The weekly template-update.yml PR brings in the config but not the note. If a maintainer merges it and skips the manual marker step, scriv collect fails at release time with Entry 'Changelog' is not a valid version! If scriv should ignore this heading, add 'scriv-end-here' somewhere before it. It does not point at the missing marker, and its hint (scriv-end-here) is the wrong fix here. The good news: it fails rather than misplacing the entry.

Fix: Quote that error in the upgrade note, so whoever hits it can find the fix by searching. Following the note as written works: I replaced ## [Unreleased] with the marker, moved the entry into a fragment, bumped the version, and scriv collect put a new ## [0.3.0] above the old ## [0.2.0] and ## [0.1.0].

How verified: Ran, on the updated projects from finding 2.

4. Apps and unpublished libraries are never told about scriv collect

Where: template/README.md.jinja line 14.

What: Only RELEASING.md names uv run scriv collect, and it is rendered only when publish_to_pypi is true. An app's README says "Fragments are collected into CHANGELOG.md at release time" and stops there.

Failure scenario: An app maintainer finds fragments piling up in changelog.d/ and no command for releasing them.

Fix: End that README paragraph with "Run uv run scriv collect at release time to collect them". For apps, mention that it reads project.version, so bump that first.

How verified: Read.

5. Smaller points

• A fragment that scriv create made and nobody edited is all comments. Collecting it writes a bare ## [0.1.0] - <date> heading with nothing under it. Ran. This is scriv's normal behaviour and not worth code. It may be worth one clause in the README ("delete the file if the change needs no entry").
• skip_fragments = "[A-Z]*" (template/pyproject.toml.jinja line 52) silently skips any fragment whose name starts with a capital letter. A hand-named changelog.d/Fix-thing.md stayed in place and never reached the changelog. Ran. scriv create names files by date, so this only bites hand-made files. It matches inspect_evals, so I'd leave it. "TEMPLATE.md" would be stricter, if you prefer.
• The CI check runs uvx --from 'scriv>=1.8', so it tracks the newest scriv rather than the version a scaffold locks. That is probably what you want for catching breakage early. Just noting it.

Checked and fine

• Both rendered pyproject.toml files parse with tomllib. The {% raw %}...{% endraw +%} line renders the Jinja title template literally, and there are no stray blank lines around [tool.scriv] in either variant. Ran.
• scriv collect with scriv 1.8.0 writes ## [0.1.0] - YYYY-MM-DD with category subheadings in both variants, deletes the fragment and keeps TEMPLATE.md. Ran.
• scriv create in both variants copies TEMPLATE.md verbatim. Uncommenting two sections and collecting gives a clean entry. Ran.
• copier copy into an empty directory writes CHANGELOG.md with the marker. Ran.
• The scaffold's full pre-commit stack, including mdformat 1.0.0 with mdformat-gfm and .mdformat.toml, passes on both variants after a collect, with TEMPLATE.md present. Ran. CI also covers this for the app, since the hooks step runs on /tmp/out-app after the scriv loop.
• The new CI block under set -euo pipefail passes on fresh renders and exits 1 when ## [Unreleased] is present. ! grep -q ... || exit 1 is correct. Ran.
• The "Unrelated" note is right. bash -c 'set -euo pipefail; ! true; echo reached' prints reached and exits 0. So the existing bare ! grep -q checks can never fail the step. Ran. Worth a follow-up PR.
• CI is green on 399a60e.

Posted by Claude Code on Matt's behalf.

[MattFisher]

Thanks. Actioned in 4ed2dea:

1. Version source. inspect_dataset's adoption PR (Keep the changelog as scriv fragments inspect_dataset#55) reads pyproject.toml. The upgrade note now says to check that [tool.scriv] version reads the file the build backend reads.
2. _skip_if_exists recreates a missing CHANGELOG.md. Now documented in the CHANGELOG entry. I kept the skip.
3. The confusing error. The upgrade note quotes Entry 'Changelog' is not a valid version! and says its scriv-end-here hint is not the fix.
4. Apps never told about scriv collect. The scaffold README now names it for every project kind, and says to bump pyproject.toml first for apps.
5. Unedited fragment. The README says to delete a fragment the change doesn't need.

Also added, from the reviews of the two adoption PRs: compact_fragments = true. Without it, two fragments in one category leave a blank line that makes the list loose, and mdformat then fails the release commit. Template CI now collects two same-category fragments and checks they form one tight list. The full CI step passes locally.

Left as is: skip_fragments = "[A-Z]*" (matches inspect_evals, and scriv create names files by date) and the unpinned scriv>=1.8 in CI (deliberate, to catch new scriv releases early). The bare ! grep checks stay as a separate follow-up.

Posted by Claude Code on Matt's behalf.