Skip to content

Fix order-dependent unit test failures under pytest 9.2 - #567

Open
dkliban wants to merge 2 commits into
pulp:mainfrom
dkliban:fix-566-pytest92-default-storage-location
Open

dkliban wants to merge 2 commits into
pulp:mainfrom
dkliban:fix-566-pytest92-default-storage-location

Conversation

@dkliban

@dkliban dkliban commented Oct 30, 2026

Copy link
Copy Markdown
Member

Problem

The nightly test (s3) job started failing on 2026-10-28 (run 40624725849):

test_path_index_integration.py::test_four_processes_share_segment_downloads FAILED
test_path_index_integration.py::test_cache_pressure_releases_idle_mappings FAILED
E   AttributeError: 'S3Storage' object has no attribute 'location'

Only the transitive dependency pytest changed between the last green run and the first red one (9.1.1 → 9.2.0); pulpcore, django and django-storages were identical.

Root cause

pytest 9.2.0 includes pytest#10644: monkeypatch.setattr() now undoes a patch of an attribute that is not in the target instance's own __dict__ with delattr instead of restoring the original value.

test_fresh_namespace_rebuilds_incompatible_profile does monkeypatch.setattr(domain.get_storage(), "location", ...) on the default domain. Domain.get_storage() for default returns Django's process-global default_storage — a LazyObject. location is stored in the wrapped S3Storage.__dict__ (set in __init__), but not in the LazyObject's own __dict__, so pytest 9.2 tears the patch down with delattr, which proxies through and permanently deletes location from the shared S3Storage. Unit tests run sequentially, so the next tests that call artifact.save() raise in S3Storage._normalize_name. The failure is collection-order dependent, which is why only a subset of tests fail.

Fix

Add pulp_maven/tests/unit/conftest.py with an autouse fixture that rebuilds default_storage before (and after) each test, so no test can inherit process-global storage state mutated by another. Resetting at setup is what makes it order-independent: monkeypatch's finalizer runs after an autouse fixture's, so a teardown-only reset would be masked.

Verification

Reproduced and fixed against the real suite with a moto-backed S3 default_storage:

pytest fix result
9.2.0 no 2 failed, 15 passed (reproduces CI)
9.2.0 yes 216 passed, 6 skipped, 0 failed
8.4.2 yes 216 passed, 6 skipped, 0 failed (no regression)

Closes #566

🤖 Generated with Claude Code

The s3 nightly unit suite began failing with
``AttributeError: 'S3Storage' object has no attribute 'location'``.

``test_fresh_namespace_rebuilds_incompatible_profile`` does
``monkeypatch.setattr(domain.get_storage(), "location", ...)`` on the default
domain, which mutates Django's process-global ``default_storage`` (a
``LazyObject``). ``location`` lives in the wrapped ``S3Storage.__dict__`` but not
in the ``LazyObject``'s own ``__dict__``, so pytest 9.2
(pytest-dev/pytest#10644) undoes the patch with ``delattr`` instead of restoring
the value, deleting ``location`` from the shared storage. Since unit tests run
sequentially, the next tests that call ``artifact.save()`` fail -- a
collection-order dependent failure.

Add ``pulp_maven/tests/unit/conftest.py`` with an autouse fixture that rebuilds
``default_storage`` before each test so storage state cannot leak between tests.

Closes: pulp#566
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Defense-in-depth on top of the unit conftest reset: rather than mutating
``location`` on the shared ``default_storage`` instance (which the default domain
returns and which pytest 9.2 fails to restore), patch ``Domain.get_storage`` to
return a relocated read-only view of the storage. The patch is restored cleanly
on teardown and never corrupts the shared instance, so the test no longer depends
on collection order even without the conftest safety net.

Ref: pulp#566
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dkliban

dkliban commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

Added a second commit with defense-in-depth on the root-cause test itself.

test_fresh_namespace_rebuilds_incompatible_profile was the sole leaker: it mutated location on the shared process-global default_storage via monkeypatch, which pytest 9.2 then failed to restore. It now patches Domain.get_storage to return a relocated read-only view of the storage instead — visible to the domain reloaded inside build_path_index, restored cleanly on teardown, and never corrupting the shared instance.

Verified against the real suite (moto-backed S3 default_storage, pytest 9.2.0):

  • With the conftest removed, the hardened test_path_index_integration.py passes on its own (19 passed) — confirming that test was the only leaker.
  • Full change together: 236 passed, 6 skipped, 4 subtests passed (the 234 that passed in CI plus the 2 that were failing).

The conftest reset stays as a general safety net so a future test can't reintroduce this class of leak.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nightly s3 unit tests fail under pytest 9.2.0: 'S3Storage' object has no attribute 'location'

2 participants