Skip to content

[IMPROVEMENT] Zstd: Contract and test coverage for ZstdBlobStoreDAO atop file and memory implementations - #3199

Merged
chibenwa merged 1 commit into
apache:masterfrom
HesandaLiyanage:file-zstd-blobstore
Sep 24, 2026
Merged

chibenwa merged 1 commit into
apache:masterfrom
HesandaLiyanage:file-zstd-blobstore

Conversation

@HesandaLiyanage

Copy link
Copy Markdown
Contributor

Context & Motivation

Following the discussion on #3193 regarding ZstdBlobStoreDAO running atop non-S3 storage layers, @chibenwa noted:

"ZstdBlobStoreDAO should work out of the box atop file implem. If this is not the case a separated PR is welcomed."

Prior to this change:

  1. ZstdBlobStoreDAOTest in server/blob/blob-zstd was hardcoded to S3BlobStoreDAO and required spinning up a Docker AWS S3 container (DockerAwsS3Extension).
  2. There was zero test coverage proving that ZstdBlobStoreDAO operates correctly atop FileBlobStoreDAO (or MemoryBlobStoreDAO).
  3. Running Zstd compression tests locally always incurred Docker container startup overhead.
  4. BlobStoreModulesChooserTest in james-server-guice-distributed tested S3 and Cassandra module resolution, but lacked test coverage for file() and postgres() implementations.

Solution Overview

  1. Extracted ZstdBlobStoreDAOContract:

    • Created org.apache.james.blob.zstd.ZstdBlobStoreDAOContract extending BlobStoreDAOContract and MetadataAwareBlobStoreDAOContract.
    • Encapsulates all Zstd-specific contract assertions (compression threshold, min ratio, custom caller metadata preservation, decompression, corrupt payload handling, metrics verification, and multi-file roundtrips).
  2. Added ZstdFileBlobStoreDAOTest:

    • Tests ZstdBlobStoreDAO directly atop FileBlobStoreDAO backed by FileSystemImpl.forTesting().
    • Confirms that FileBlobStoreDAO's user-defined extended attributes (UserDefinedFileAttributeView) correctly store and propagate ContentEncoding.ZSTD and content-original-size metadata out of the box.
  3. Added ZstdMemoryBlobStoreDAOTest:

    • Provides ultra-fast, containerless contract verification of ZstdBlobStoreDAO atop MemoryBlobStoreDAO.
  4. Refactored ZstdBlobStoreDAOTest:

    • ZstdBlobStoreDAOTest now implements ZstdBlobStoreDAOContract for S3BlobStoreDAO (retaining full Docker S3 integration coverage while reducing boilerplate from 454 lines down to 88 lines).
  5. Enhanced BlobStoreModulesChooserTest:

    • Added test assertions verifying BlobStoreModulesChooser correctly selects FileBlobStoreDAODeclarationModule and PostgresBlobStoreDAODeclarationModule.

Verification & Tests

  • server/blob/blob-zstd:
    • ZstdFileBlobStoreDAOTest: 109 / 109 tests passed (100%)
    • ZstdMemoryBlobStoreDAOTest: 109 / 109 tests passed (100%)
    • ZstdBlobStoreDAOTest (S3 + Testcontainers): 110 / 110 tests passed (100%)
    • Total: 328 tests passed, 0 failures.
  • server/container/guice/distributed:
    • BlobStoreModulesChooserTest: 8 / 8 tests passed (100%)
  • Checkstyle: 0 violations across all modified modules.
  • Sortpom: Clean pom ordering verified.

…top file and memory implementations

Following PMC feedback regarding ZstdBlobStoreDAO working out of the box atop the file implementation:

- Extract ZstdBlobStoreDAOContract interface extending BlobStoreDAOContract and MetadataAwareBlobStoreDAOContract.
- Add ZstdFileBlobStoreDAOTest verifying ZstdBlobStoreDAO works out of the box atop FileBlobStoreDAO.
- Add ZstdMemoryBlobStoreDAOTest for lightweight in-memory contract verification without requiring Docker.
- Refactor ZstdBlobStoreDAOTest to implement ZstdBlobStoreDAOContract against S3BlobStoreDAO.
- Add File and Postgres BlobStore module choosing tests in BlobStoreModulesChooserTest.

@chibenwa chibenwa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Just need ICLA reception)

@chibenwa

Copy link
Copy Markdown
Contributor

"ZstdBlobStoreDAO should work out of the box atop file implem. If this is not the case a separated PR is welcomed."

My concern was not necessarily about the DAO compatibility (which is possible through the BlobStoreDAOMetadataContract) but more about guice wiring.

But those tests do not hurt.

@HesandaLiyanage

HesandaLiyanage commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

(Just need ICLA reception)

So i have to sign this https://www.apache.org/licenses/icla.pdf and then send to secretary@apache.org right ?

@chibenwa

chibenwa commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

So i have to sign this https://www.apache.org/licenses/icla.pdf and then send to secretary@apache.org right ?

Yes

Fill with PDF for automate reading and sign a separated paper.

notify project james add this in the email body so that I get notified

@HesandaLiyanage

Copy link
Copy Markdown
Contributor Author

notify project james add this in the email body so that I get notified

Done

@chibenwa
chibenwa merged commit bf10035 into apache:master Sep 24, 2026
1 check passed
@chibenwa

Copy link
Copy Markdown
Contributor

Thanks a lot for this first contribution.

Wishing you to be able to contribute many more!

Best regard,

Benoit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants