Skip to content

compat-gen skips old fixture versions in CI and stops at the first failure #10754

Open

Description

@connortsui20

Problem or motivation

The backward-compatibility harness (vortex-test/compat-gen) has three gaps:

  • No workflow runs compat.py check --mode all on its own. compat-validation.yml defaults to last (inputs.mode || 'last'), and its only caller, publish-dry-runs.yml, passes last. So once a newer release publishes a fixture under the same name, CI stops reading the files of older releases. For example, after Tracking Issue: Fix Canonical Decimal #10701 lands, the old decimal forms that Add compat fixtures for old decimal forms #10753 adds exist only under the versions published before it.
  • One failing fixture stops the whole check. check.rs returns early on the first fixture write error, and assert_arrays_eq! panics instead of recording a failure.
  • The README ("Git worktree workflow") says that compat.py publish --git-ref <ref> builds the binary from a worktree at that ref. The script only uses the ref to pick the version (scripts/compat.py:394, :416) and always builds the current tree.

Proposed solution

  • Run --mode all before each release, or on a schedule.
  • Record failures per fixture, and report them together at the end.
  • Either make --git-ref build from the ref, or correct the README.

Additional context

Found while planning #10701. Research notes: #10633.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureA feature request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions