From eadb532d7ea20d509015168608230eb4261ff9d0 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:20:17 +0530 Subject: [PATCH 01/24] JAMES-4231 ADR: Architecture Decision Record for S3 Object Compaction Document the context, architecture, chunk format, crash safety invariants, and garbage collection strategy for autonomous S3 object compaction. --- src/adr/0076-s3-object-compaction.md | 110 +++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 src/adr/0076-s3-object-compaction.md diff --git a/src/adr/0076-s3-object-compaction.md b/src/adr/0076-s3-object-compaction.md new file mode 100644 index 00000000000..2e3216583a2 --- /dev/null +++ b/src/adr/0076-s3-object-compaction.md @@ -0,0 +1,110 @@ +# 76. S3 Object Compaction + +Date: 2026-09-20 + +## Status + +Implemented (JAMES-4231) + +## Context + +High email ingest volumes in Apache James deployments utilizing S3/MinIO object storage result in +hundreds of millions of small S3 objects (average mail body/header size ~20–50KB). While S3 provides +virtually limitless capacity, operating at this granularity introduces substantial challenges: +- High S3 API request costs (PUT/GET/LIST pricing per 1,000 operations). +- High metadata overhead and slow bucket listing during storage maintenance and garbage collection. +- Reduced overall object storage throughput due to connection setup and small payload overheads. + +Apache James already features generation-aware blob IDs (`GenerationAwareBlobId`) and Bloom filter +garbage collection (`0049-deduplicated-blobs-gs-with-bloom-filters.md`). However, once a generation +becomes immutable, standalone small objects remain permanently fragmented across S3. + +## Decision + +We introduce an autonomous S3 object compaction engine (`server/blob/blob-compaction`) and chunked +blob store DAO (`ChunkedBlobStoreDAO`) that packs multiple small standalone blobs into large, immutable +chunk objects (~100MB target size) with slot-level virtual addressing and ranged read support. + +### 1. Chunk File Format (`ChunkFormat`) + +A chunk object is a single binary object stored in S3 containing multiple concatenated blob slots and +a trailer footer: + +``` ++-------------------------------------------------------------------------------+ +| Chunk Header (16 bytes: magic 0x4A43484B, version 1, slot count, flags) | ++-------------------------------------------------------------------------------+ +| Slot 0: [Slot Header: compressed size, uncompressed size, flags][Payload] | ++-------------------------------------------------------------------------------+ +| Slot 1: [Slot Header: compressed size, uncompressed size, flags][Payload] | ++-------------------------------------------------------------------------------+ +| ... | ++-------------------------------------------------------------------------------+ +| Slot N-1: [Slot Header: compressed size, uncompressed size, flags][Payload] | ++-------------------------------------------------------------------------------+ +| Footer: [Slot 0 Offset (8B), Size (4B), Flags (4B)]...[Slot N-1 Offset, ...] | ++-------------------------------------------------------------------------------+ +| Footer Metadata Trailer (4 bytes: footer size) + End Magic (0x454F4643) | ++-------------------------------------------------------------------------------+ +``` + +Key format invariants: +- **Per-Slot Zstandard Compression:** Each slot is independently compressed using Zstandard + (`content-encoding=zstd\n`), allowing individual slot reads via HTTP byte-range requests without + decompressing or buffering the full 100MB chunk. +- **Suffix-Indexed Footer:** The chunk footer is located at the tail of the object. Inspecting chunk + metadata or slot allocations requires only a single 64KB HTTP ranged read (`readRange(..., -65536, -1)`). + +### 2. Virtual Slot Addressing (`ChunkId`) + +Compacted slots are addressed via synthetic blob IDs formatted as: +``` +__chunk~~ +``` +- **Transparent DAO Interception:** `ChunkedBlobStoreDAO` acts as a decorator around the underlying + `BlobStoreDAO`. Requests for standalone blobs pass through to the delegate. Requests for virtual + slot IDs parse the offset and limit, translating them into exact HTTP ranged reads + (`readRange(bucket, chunkId, offset, offset + limit - 1)`). +- **Format Invariance:** Virtual slot IDs conform to `ChunkMarker` regex rules + (`^\\d+_\\d+_chunk[A-Za-z0-9_-]{16,}(~\\d+~\\d+)?$`), ensuring seamless interoperability with + Bloom filter GC without treating chunks as unreferenced blobs. + +### 3. Compaction Algorithms & Crash Safety + +Compaction runs as background distributed tasks (`BlobCompactionTask`) managed via WebAdmin endpoints: +- **Initial Compaction (`initialCompact`):** + 1. Windowed candidate scanning in batches of 1,000 blobs to bound heap usage. + 2. Candidate payloads packed into a new chunk object and written to S3 raw storage. + 3. Source-of-truth metadata tables updated in Cassandra (`messageV3`, `messageIdToImapUid`, + `messageIdTable`) pointing old blob IDs to new slot refs. + 4. Original standalone blobs deleted from S3 only for candidates whose metadata references + were successfully updated. +- **GC Compaction (`gcCompact`):** + 1. Inspect existing chunks using footer-only ranged reads (metadata-only). + 2. Orphan chunks (100% dead slots) deleted immediately with zero payload reads. + 3. Chunks with dead slot ratios exceeding thresholds rewritten to discard dead slots. + 4. Small adjacent chunks merged into target-sized chunks. Surviving slots are streamed + one-by-one via HTTP ranged reads, bounding GC heap consumption to $O(\text{maxSlotSize})$. + +### 4. Configuration Guards and Layering + +Because client-side AES encryption (`CryptoConfig`) is incompatible with arbitrary HTTP byte-range +slicing without custom IV handling, compaction automatically disables itself when encryption or +whole-blob compression is configured on the target bucket, logging an informative warning. + +## Consequences + +### Positive +- **Object Count Reduction:** Decreases S3 object count by 10x to 100x for historical email generations. +- **Cost Reduction:** Drastically reduces S3 LIST and GET request volume and monthly storage request fees. +- **Low Read Latency:** Reading compacted emails requires only a single HTTP byte-range GET request, + matching the latency of standalone object reads. +- **Crash Safety:** Step ordering guarantees no dangling references; failed runs leave either intact + original blobs or orphan chunks that are automatically reclaimed on the next GC run. + +### Negative / Trade-offs +- Compacting historical generations consumes temporary I/O to read candidates and write chunks. +- Cassandra reference updates introduce a transient inconsistency window during process crashes + that is healed by `CassandraBlobIdRepairer`. +- Materializes generation live reference multimaps in memory during compaction passes (~200MB heap + per 1M references); future iterations can introduce partition-paged reference lookups. From c3a6b84cc754e82eec414175ff46d7c6bacd1d85 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:20:23 +0530 Subject: [PATCH 02/24] JAMES-4231 Blob API: Add range read contract and ChunkMarker support - Add BlobStoreDAO.readRange(bucket, blobId, offset, endOffset) returning a Mono with total object size and requested slice. - Define default readRange fallback using readBytes. - Add ChunkMarker utility for discriminating compaction chunk references. - Add contract tests for readRange in ReadSaveBlobStoreDAOContract. --- .../apache/james/blob/api/BlobStoreDAO.java | 49 ++++++++++++++ .../apache/james/blob/api/ChunkMarker.java | 48 ++++++++++++++ .../api/ReadSaveBlobStoreDAOContract.java | 66 +++++++++++++++++++ 3 files changed, 163 insertions(+) create mode 100644 server/blob/blob-api/src/main/java/org/apache/james/blob/api/ChunkMarker.java diff --git a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java index 46af6ebed14..be5d6d2899e 100644 --- a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java +++ b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java @@ -39,6 +39,7 @@ import com.google.common.io.FileBackedOutputStream; import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; /** * James virtual blob store abstraction. @@ -303,4 +304,52 @@ default Publisher listBlobs(BucketName bucketName, String prefix) { return Flux.from(listBlobs(bucketName)) .filter(blobId -> blobId.asString().startsWith(prefix)); } + + record RangeByteSlice(byte[] data, long totalObjectSize) { + public static RangeByteSlice of(byte[] data, long totalObjectSize) { + return new RangeByteSlice(data, totalObjectSize); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + RangeByteSlice that = (RangeByteSlice) o; + return totalObjectSize == that.totalObjectSize && Arrays.equals(data, that.data); + } + + @Override + public int hashCode() { + int result = Objects.hash(totalObjectSize); + result = 31 * result + Arrays.hashCode(data); + return result; + } + } + + default Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + return Mono.from(readBytes(bucketName, blobId)) + .map(bytesBlob -> { + byte[] allBytes = bytesBlob.payload(); + long totalSize = allBytes.length; + if (start < 0) { + int suffixLength = (int) Math.min(totalSize, -start); + int from = (int) (totalSize - suffixLength); + byte[] slice = Arrays.copyOfRange(allBytes, from, (int) totalSize); + return RangeByteSlice.of(slice, totalSize); + } + if (start >= totalSize) { + return RangeByteSlice.of(new byte[0], totalSize); + } + long boundedEnd = Math.min(end, totalSize - 1); + if (boundedEnd < start) { + return RangeByteSlice.of(new byte[0], totalSize); + } + byte[] slice = Arrays.copyOfRange(allBytes, (int) start, (int) boundedEnd + 1); + return RangeByteSlice.of(slice, totalSize); + }); + } } diff --git a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/ChunkMarker.java b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/ChunkMarker.java new file mode 100644 index 00000000000..0efbac0a12b --- /dev/null +++ b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/ChunkMarker.java @@ -0,0 +1,48 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.api; + +import java.util.regex.Pattern; + +/** + * Utility for detecting chunk-based BlobId identifiers across Apache James modules + * without introducing circular module dependencies. + */ +public final class ChunkMarker { + public static final String CHUNK_MARKER = "_chunk"; + private static final Pattern CHUNK_PATTERN = Pattern.compile("^\\d+_\\d+_chunk[A-Za-z0-9_-]{16,}(~\\d+~\\d+)?$"); + + private ChunkMarker() { + } + + public static boolean looksLikeChunkId(String id) { + if (id == null) { + return false; + } + return CHUNK_PATTERN.matcher(id).matches(); + } + + public static boolean looksLikeChunkId(BlobId blobId) { + if (blobId == null) { + return false; + } + return looksLikeChunkId(blobId.asString()); + } +} diff --git a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java index ce99703f478..9d5340baee2 100644 --- a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java +++ b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java @@ -405,4 +405,70 @@ public int read(byte[] b, int off, int len) throws IOException { }; } + + @Test + default void readRangeShouldReturnMiddleSlice() { + byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); + Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); + + BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 3, 6).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(10); + assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("3456"); + } + + @Test + default void readRangeShouldReturnPrefix() { + byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); + Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); + + BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 0, 2).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(10); + assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("012"); + } + + @Test + default void readRangeShouldReturnSuffixWhenStartNegative() { + byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); + Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); + + BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, -4, -1).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(10); + assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("6789"); + } + + @Test + default void readRangeShouldReturnFullContent() { + byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); + Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); + + BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 0, 9).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(10); + assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("0123456789"); + } + + @Test + default void readRangeShouldCapEndAtObjectSize() { + byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); + Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); + + BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 5, 50).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(10); + assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("56789"); + } + + @Test + default void readRangeShouldReturnEmptyWhenStartBeyondSize() { + byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); + Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); + + BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 20, 30).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(10); + assertThat(slice.data()).isEmpty(); + } } From 366bcd227826ccce48e8fb3c6031aad672b50a19 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:20:30 +0530 Subject: [PATCH 03/24] JAMES-4231 S3 BlobStore: Implement byte range reads for S3BlobStoreDAO - Implement S3BlobStoreDAO.readRange using AWS S3 HTTP Range requests. - Handle S3 416 RequestedRangeNotSatisfiable when requested offset exceeds object size or range is invalid, falling back or erroring appropriately. - Add contract and MinIO integration tests for S3 range reads. --- server/blob/blob-s3/pom.xml | 5 + .../objectstorage/aws/S3BlobStoreDAO.java | 106 ++++++++++ .../aws/S3BlobStoreDAORangeReadTest.java | 126 ++++++++++++ .../aws/S3MinioBlobStoreCompactionTest.java | 189 ++++++++++++++++++ 4 files changed, 426 insertions(+) create mode 100644 server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java create mode 100644 server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java diff --git a/server/blob/blob-s3/pom.xml b/server/blob/blob-s3/pom.xml index dd4ce52ef71..6fb40e7dcfe 100644 --- a/server/blob/blob-s3/pom.xml +++ b/server/blob/blob-s3/pom.xml @@ -47,6 +47,11 @@ test-jar test + + ${james.groupId} + blob-compaction + test + ${james.groupId} blob-storage-strategy diff --git a/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java b/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java index 94392fe6d3f..a0eede2a84d 100644 --- a/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java +++ b/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java @@ -38,6 +38,7 @@ import org.apache.commons.io.IOUtils; import org.apache.james.blob.api.BlobId; import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BlobStoreDAO.RangeByteSlice; import org.apache.james.blob.api.BucketName; import org.apache.james.blob.api.ObjectNotFoundException; import org.apache.james.blob.api.ObjectStoreIOException; @@ -69,6 +70,8 @@ import software.amazon.awssdk.services.s3.model.DeleteObjectsResponse; import software.amazon.awssdk.services.s3.model.GetObjectRequest; import software.amazon.awssdk.services.s3.model.GetObjectResponse; +import software.amazon.awssdk.services.s3.model.HeadObjectRequest; +import software.amazon.awssdk.services.s3.model.HeadObjectResponse; import software.amazon.awssdk.services.s3.model.ListBucketsResponse; import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; import software.amazon.awssdk.services.s3.model.ListObjectsV2Response; @@ -245,6 +248,92 @@ public Publisher readBytes(BucketName bucketName, BlobId blobId) { .onErrorMap(e -> e.getCause() instanceof OutOfMemoryError, Throwable::getCause); } + @Override + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + BucketName resolvedBucketName = bucketNameResolver.resolve(bucketName); + String rangeHeader = formatRangeHeader(start, end); + + return getObjectRangeBytes(resolvedBucketName, blobId, rangeHeader) + .onErrorMap(NoSuchBucketException.class, e -> new ObjectNotFoundException("Bucket not found " + resolvedBucketName.asString(), e)) + .onErrorMap(NoSuchKeyException.class, e -> new ObjectNotFoundException("Blob not found " + blobId.asString() + " in bucket " + resolvedBucketName.asString(), e)) + .publishOn(Schedulers.parallel()) + .map(responseBytes -> { + long totalObjectSize = extractTotalObjectSize(responseBytes.response()); + return RangeByteSlice.of(responseBytes.asByteArrayUnsafe(), totalObjectSize); + }) + .onErrorResume(this::isRangeNotSatisfiable, e -> + headObject(resolvedBucketName, blobId) + .map(head -> RangeByteSlice.of(new byte[0], head.contentLength() != null ? head.contentLength() : 0L)) + .onErrorMap(NoSuchBucketException.class, ex -> new ObjectNotFoundException("Bucket not found " + resolvedBucketName.asString(), ex)) + .onErrorMap(NoSuchKeyException.class, ex -> new ObjectNotFoundException("Blob not found " + blobId.asString() + " in bucket " + resolvedBucketName.asString(), ex))) + .onErrorMap(e -> e.getCause() instanceof OutOfMemoryError, Throwable::getCause); + } + + private boolean isRangeNotSatisfiable(Throwable t) { + if (t instanceof S3Exception && ((S3Exception) t).statusCode() == 416) { + return true; + } + if (t.getCause() instanceof S3Exception && ((S3Exception) t.getCause()).statusCode() == 416) { + return true; + } + return false; + } + + private Mono headObject(BucketName bucketName, BlobId blobId) { + return headObjectFromStore(bucketName, blobId) + .onErrorResume(e -> e instanceof NoSuchKeyException || e instanceof NoSuchBucketException, e -> { + if (fallbackNamespace.isPresent() && bucketNameResolver.isNameSpace(bucketName)) { + return headObjectFromStore(fallbackNamespace.get(), blobId); + } + return Mono.error(e); + }); + } + + private Mono headObjectFromStore(BucketName bucketName, BlobId blobId) { + return buildHeadObjectRequestBuilder(bucketName, blobId) + .flatMap(builder -> Mono.fromFuture(() -> client.headObject(builder.build()))); + } + + private Mono buildHeadObjectRequestBuilder(BucketName bucketName, BlobId blobId) { + HeadObjectRequest.Builder baseBuilder = HeadObjectRequest.builder() + .bucket(bucketName.asString()) + .key(blobId.asString()); + + if (s3RequestOption.ssec().enable()) { + return Mono.from(s3RequestOption.ssec().sseCustomerKeyFactory().get() + .generate(bucketName, blobId)) + .map(sseCustomerKey -> baseBuilder + .sseCustomerAlgorithm(sseCustomerKey.ssecAlgorithm()) + .sseCustomerKey(sseCustomerKey.customerKey()) + .sseCustomerKeyMD5(sseCustomerKey.md5())); + } + return Mono.just(baseBuilder); + } + + private String formatRangeHeader(long start, long end) { + if (start < 0) { + return "bytes=" + start; + } + if (end < start) { + return "bytes=" + start + "-"; + } + return "bytes=" + start + "-" + end; + } + + private long extractTotalObjectSize(GetObjectResponse response) { + String contentRange = response.contentRange(); + if (contentRange != null && contentRange.contains("/")) { + String totalPart = contentRange.substring(contentRange.lastIndexOf('/') + 1).trim(); + try { + return Long.parseLong(totalPart); + } catch (NumberFormatException e) { + // fall through + } + } + Long contentLength = response.contentLength(); + return contentLength != null ? contentLength : 0L; + } + private Mono> getObjectBytes(BucketName bucketName, BlobId blobId) { return getObjectBytesFromStore(bucketName, blobId) .onErrorResume(e -> e instanceof NoSuchKeyException || e instanceof NoSuchBucketException, e -> { @@ -255,12 +344,29 @@ private Mono> getObjectBytes(BucketName bucketN }); } + private Mono> getObjectRangeBytes(BucketName bucketName, BlobId blobId, String rangeHeader) { + return getObjectRangeBytesFromStore(bucketName, blobId, rangeHeader) + .onErrorResume(e -> e instanceof NoSuchKeyException || e instanceof NoSuchBucketException, e -> { + if (fallbackNamespace.isPresent() && bucketNameResolver.isNameSpace(bucketName)) { + return getObjectRangeBytesFromStore(fallbackNamespace.get(), blobId, rangeHeader); + } + return Mono.error(e); + }); + } + private Mono> getObjectBytesFromStore(BucketName bucketName, BlobId blobId) { return buildGetObjectRequestBuilder(bucketName, blobId) .flatMap(putObjectRequest -> Mono.fromFuture(() -> client.getObject(putObjectRequest.build(), new MinimalCopyBytesResponseTransformer(configuration, blobId)))); } + private Mono> getObjectRangeBytesFromStore(BucketName bucketName, BlobId blobId, String rangeHeader) { + return buildGetObjectRequestBuilder(bucketName, blobId) + .map(builder -> builder.range(rangeHeader)) + .flatMap(putObjectRequest -> Mono.fromFuture(() -> + client.getObject(putObjectRequest.build(), new MinimalCopyBytesResponseTransformer(configuration, blobId)))); + } + private Mono buildGetObjectRequestBuilder(BucketName bucketName, BlobId blobId) { GetObjectRequest.Builder baseBuilder = GetObjectRequest.builder() .bucket(bucketName.asString()) diff --git a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java new file mode 100644 index 00000000000..2503b63653b --- /dev/null +++ b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java @@ -0,0 +1,126 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.objectstorage.aws; + +import static org.apache.james.blob.api.BlobStoreDAOFixture.TEST_BUCKET_NAME; +import static org.apache.james.blob.objectstorage.aws.JamesS3MetricPublisher.DEFAULT_S3_METRICS_PREFIX; +import static org.apache.james.blob.objectstorage.aws.S3BlobStoreConfiguration.UPLOAD_RETRY_EXCEPTION_PREDICATE; +import static org.assertj.core.api.Assertions.assertThat; + +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.Arrays; +import java.util.Optional; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.TestBlobId; +import org.apache.james.metrics.api.NoopGaugeRegistry; +import org.apache.james.metrics.tests.RecordingMetricFactory; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; + +import reactor.core.publisher.Mono; +import reactor.util.retry.Retry; + +@ExtendWith(DockerAwsS3Extension.class) +class S3BlobStoreDAORangeReadTest { + private static S3BlobStoreDAO testee; + private static S3ClientFactory s3ClientFactory; + + private BlobId blobId; + private byte[] content; + + @BeforeAll + static void setUp(DockerAwsS3Container dockerAwsS3) { + AwsS3AuthConfiguration authConfiguration = AwsS3AuthConfiguration.builder() + .endpoint(dockerAwsS3.getEndpoint()) + .accessKeyId(DockerAwsS3Container.ACCESS_KEY_ID) + .secretKey(DockerAwsS3Container.SECRET_ACCESS_KEY) + .build(); + + S3BlobStoreConfiguration s3Configuration = S3BlobStoreConfiguration.builder() + .authConfiguration(authConfiguration) + .region(dockerAwsS3.dockerAwsS3().region()) + .uploadRetrySpec(Optional.of(Retry.backoff(3, Duration.ofSeconds(1)) + .filter(UPLOAD_RETRY_EXCEPTION_PREDICATE))) + .defaultBucketName(BucketName.DEFAULT) + .build(); + + s3ClientFactory = new S3ClientFactory(s3Configuration, () -> new JamesS3MetricPublisher(new RecordingMetricFactory(), new NoopGaugeRegistry(), + DEFAULT_S3_METRICS_PREFIX)); + + testee = new S3BlobStoreDAO(s3ClientFactory, s3Configuration, new TestBlobId.Factory(), S3RequestOption.DEFAULT); + } + + @BeforeEach + void init() { + StringBuilder builder = new StringBuilder(); + for (int i = 0; i < 100; i++) { + builder.append(String.format("%010d", i)); // 100 * 10 = 1000 bytes + } + content = builder.toString().getBytes(StandardCharsets.UTF_8); + blobId = new TestBlobId.Factory().of(java.util.UUID.randomUUID().toString()); + Mono.from(testee.save(TEST_BUCKET_NAME, blobId, BlobStoreDAO.BytesBlob.of(content))).block(); + } + + @Test + void readRangeFirstBytesShouldReturnFirstBytes() { + BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 0, 99).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(1000); + assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(content, 0, 100)); + } + + @Test + void readRangeMidObjectShouldReturnMidBytes() { + BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 200, 399).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(1000); + assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(content, 200, 400)); + } + + @Test + void readRangeLastBytesNegativeShouldReturnSuffix() { + BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, -100, -1).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(1000); + assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(content, 900, 1000)); + } + + @Test + void readRangeFullObjectShouldReturnAllBytes() { + BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 0, 999).block(); + + assertThat(slice.totalObjectSize()).isEqualTo(1000); + assertThat(slice.data()).isEqualTo(content); + } + + @Test + void crossCheckAgainstFullReadShouldMatch() { + BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 150, 450).block(); + byte[] fullRead = Mono.from(testee.readBytes(TEST_BUCKET_NAME, blobId)).block().payload(); + + assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(fullRead, 150, 451)); + } +} diff --git a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java new file mode 100644 index 00000000000..d078eb79afe --- /dev/null +++ b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java @@ -0,0 +1,189 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.objectstorage.aws; + +import static org.apache.james.blob.objectstorage.aws.S3BlobStoreConfiguration.UPLOAD_RETRY_EXCEPTION_PREDICATE; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.ZonedDateTime; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.ConcurrentHashMap; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobReferenceSource; +import org.apache.james.blob.api.BlobStore; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.ObjectNotFoundException; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.blob.compaction.BlobCompactionAlgorithm; +import org.apache.james.blob.compaction.BlobCompactionTask; +import org.apache.james.blob.compaction.BlobIdUpdater; +import org.apache.james.blob.compaction.BlobReferenceMappingSource; +import org.apache.james.blob.compaction.ChunkId; +import org.apache.james.blob.compaction.ChunkedBlobStoreDAO; +import org.apache.james.blob.compaction.CompactionConfiguration; +import org.apache.james.blob.compaction.CompactionRequest; +import org.apache.james.metrics.api.NoopGaugeRegistry; +import org.apache.james.metrics.tests.RecordingMetricFactory; +import org.apache.james.server.blob.deduplication.BloomFilterGCAlgorithm; +import org.apache.james.server.blob.deduplication.DeDuplicationBlobStore; +import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; +import org.apache.james.task.Task; +import org.apache.james.utils.UpdatableTickingClock; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.util.retry.Retry; + +class S3MinioBlobStoreCompactionTest { + + private static final BucketName BUCKET = BucketName.of("compaction-test-bucket"); + private static final ZonedDateTime NOW = ZonedDateTime.parse("2020-01-01T00:00:00Z"); + + @RegisterExtension + static S3MinioExtension minioExtension = new S3MinioExtension(); + + private S3BlobStoreDAO rawStore; + private ChunkedBlobStoreDAO chunkedBlobStoreDAO; + private UpdatableTickingClock clock; + private GenerationAwareBlobId.Factory generationAwareBlobIdFactory; + private GenerationAwareBlobId.Configuration generationConfiguration; + + @BeforeEach + void setUp() { + AwsS3AuthConfiguration awsS3AuthConfiguration = minioExtension.minioDocker().getAwsS3AuthConfiguration(); + + S3BlobStoreConfiguration s3Configuration = S3BlobStoreConfiguration.builder() + .authConfiguration(awsS3AuthConfiguration) + .region(DockerAwsS3Container.REGION) + .uploadRetrySpec(Optional.of(Retry.backoff(3, Duration.ofSeconds(1)) + .filter(UPLOAD_RETRY_EXCEPTION_PREDICATE))) + .defaultBucketName(BUCKET) + .build(); + + S3ClientFactory s3ClientFactory = new S3ClientFactory(s3Configuration, new RecordingMetricFactory(), new NoopGaugeRegistry()); + PlainBlobId.Factory plainBlobIdFactory = new PlainBlobId.Factory(); + + clock = new UpdatableTickingClock(NOW.toInstant()); + generationConfiguration = GenerationAwareBlobId.Configuration.DEFAULT; + generationAwareBlobIdFactory = new GenerationAwareBlobId.Factory(clock, plainBlobIdFactory, generationConfiguration); + + rawStore = new S3BlobStoreDAO(s3ClientFactory, s3Configuration, generationAwareBlobIdFactory, S3RequestOption.DEFAULT); + chunkedBlobStoreDAO = new ChunkedBlobStoreDAO(rawStore, rawStore); + } + + @Test + void compactionShouldPackBlobsAndAllowReadsWhileGcPreservesChunks() { + BlobStore blobStore = new DeDuplicationBlobStore(rawStore, BUCKET, generationAwareBlobIdFactory); + + Map savedBlobs = new HashMap<>(); + Map currentMappings = new ConcurrentHashMap<>(); + for (int i = 0; i < 20; i++) { + byte[] content = ("blob-content-payload-" + i).getBytes(StandardCharsets.UTF_8); + BlobId blobId = Mono.from(blobStore.save(BUCKET, content, BlobStore.StoragePolicy.HIGH_PERFORMANCE)).block(); + savedBlobs.put(blobId, content); + currentMappings.put("msg-" + i, new BlobReferenceMappingSource.BlobIdMessageIdMapping(blobId, "msg-" + i)); + } + + List initialListed = Flux.from(rawStore.listBlobs(BUCKET)).collectList().block(); + assertThat(initialListed).hasSize(20); + + long targetGeneration = NOW.toInstant().getEpochSecond() / generationConfiguration.getDuration().toSeconds(); + int targetFamily = generationConfiguration.getFamily(); + + // Advance clock into the next generation so that targetGeneration is now an old generation + clock.setInstant(NOW.plusMonths(2).toInstant()); + + BlobReferenceMappingSource mappingSource = () -> Flux.fromIterable(currentMappings.values()); + Map updatedIds = new ConcurrentHashMap<>(); + BlobIdUpdater blobIdUpdater = (oldId, newId, messageIds) -> { + updatedIds.put(oldId, newId); + for (String msgId : messageIds) { + currentMappings.put(msgId, new BlobReferenceMappingSource.BlobIdMessageIdMapping(newId, msgId)); + } + return Mono.empty(); + }; + + BlobCompactionAlgorithm algorithm = new BlobCompactionAlgorithm( + chunkedBlobStoreDAO, + rawStore, + mappingSource, + blobIdUpdater); + + CompactionRequest request = CompactionRequest.builder() + .generation(targetGeneration) + .family(targetFamily) + .bucketName(BUCKET) + .configuration(CompactionConfiguration.DEFAULT) + .build(); + + BlobCompactionTask task = new BlobCompactionTask(algorithm, request, clock); + Task.Result taskResult = task.run(); + assertThat(taskResult).isEqualTo(Task.Result.COMPLETED); + + // 1. Assert: original objects are gone from rawStore + for (BlobId oldBlobId : savedBlobs.keySet()) { + assertThatThrownBy(() -> Mono.from(rawStore.readBytes(BUCKET, oldBlobId)).block()) + .isInstanceOf(ObjectNotFoundException.class); + } + + // 2. Assert: chunk object exists in rawStore + List chunkIds = Flux.from(rawStore.listBlobs(BUCKET)) + .filter(ChunkId::isChunkRef) + .collectList() + .block(); + assertThat(chunkIds).isNotEmpty(); + + // 3. Assert: every blob is readable byte-identical through ChunkedBlobStoreDAO using new slot-ref id + for (Map.Entry entry : savedBlobs.entrySet()) { + BlobId newSlotId = updatedIds.get(entry.getKey()); + assertThat(newSlotId).isNotNull(); + byte[] readBytes = Mono.from(chunkedBlobStoreDAO.readBytes(BUCKET, newSlotId)).block().payload(); + assertThat(readBytes).isEqualTo(entry.getValue()); + } + + // 4. Assert: BloomFilterGCAlgorithm after compaction does NOT delete the chunk object + BlobReferenceSource gcRefSource = () -> Flux.fromIterable(updatedIds.values()); + BloomFilterGCAlgorithm gcAlgorithm = new BloomFilterGCAlgorithm( + gcRefSource, + rawStore, + generationAwareBlobIdFactory, + generationConfiguration, + clock); + + Task.Result gcResult = Mono.from(gcAlgorithm.gc(100, 10, 0.01, BUCKET, new BloomFilterGCAlgorithm.Context(100, 0.01))).block(); + assertThat(gcResult).isEqualTo(Task.Result.COMPLETED); + + for (BlobId chunkId : chunkIds) { + assertThatCode(() -> Mono.from(rawStore.readBytes(BUCKET, chunkId)).block()) + .doesNotThrowAnyException(); + } + } +} From fa6091bd383956c116f6f6799ccbb0c9e634b6e4 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:20:36 +0530 Subject: [PATCH 04/24] JAMES-4231 Storage Strategy: Exclude compacted chunks in BloomFilter GC - Update BloomFilterGCAlgorithm to skip ChunkMarker chunk references during GC sweeps so that compacted chunks are never treated as unreferenced blobs. - Add test coverage in BloomFilterGCAlgorithmContract. --- .../deduplication/BloomFilterGCAlgorithm.java | 6 +++ .../BloomFilterGCAlgorithmContract.java | 43 +++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/server/blob/blob-storage-strategy/src/main/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithm.java b/server/blob/blob-storage-strategy/src/main/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithm.java index b7523c2af87..0b387c89b4e 100644 --- a/server/blob/blob-storage-strategy/src/main/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithm.java +++ b/server/blob/blob-storage-strategy/src/main/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithm.java @@ -34,6 +34,7 @@ import org.apache.james.blob.api.BlobReferenceSource; import org.apache.james.blob.api.BlobStoreDAO; import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.ChunkMarker; import org.apache.james.task.Task; import org.apache.james.task.Task.Result; import org.slf4j.Logger; @@ -281,6 +282,7 @@ private Mono gc(BloomFilter bloomFilter, BucketName bucket } return false; }) + .filter(blobId -> !isChunk(blobId)) .filter(blobId -> !bloomFilter.mightContain(salt + blobId.asString())) .window(deletionWindowSize) .flatMap(blobIdFlux -> handlePagedDeletion(bucketName, context, blobIdFlux), DEFAULT_CONCURRENCY) @@ -314,4 +316,8 @@ private Mono> populatedBloomFilter(int expectedBlobCou .then() .thenReturn(bloomFilter)); } + + private boolean isChunk(BlobId blobId) { + return ChunkMarker.looksLikeChunkId(blobId); + } } diff --git a/server/blob/blob-storage-strategy/src/test/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithmContract.java b/server/blob/blob-storage-strategy/src/test/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithmContract.java index 39397f1f145..ba06db89041 100644 --- a/server/blob/blob-storage-strategy/src/test/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithmContract.java +++ b/server/blob/blob-storage-strategy/src/test/java/org/apache/james/server/blob/deduplication/BloomFilterGCAlgorithmContract.java @@ -261,4 +261,47 @@ default void gcShouldHandlerErrorWhenException() { .bloomFilterAssociatedProbability(ASSOCIATED_PROBABILITY) .build()); } + + @Test + default void gcShouldNotRemoveChunkBlobsEvenWhenInOldGenerationAndUnreferenced() { + BlobStoreDAO blobStoreDAO = blobStoreDAO(); + when(BLOB_REFERENCE_SOURCE.listReferencedBlobs()).thenReturn(Flux.empty()); + + BlobId normalBlob = GENERATION_AWARE_BLOB_ID_FACTORY.of(UUID.randomUUID().toString()); + BlobId chunkBlob = BLOB_ID_FACTORY.parse("1_1_chunk1234567890abcdef"); + + Mono.from(blobStoreDAO.save(DEFAULT_BUCKET, normalBlob, BlobStoreDAO.BytesBlob.of("normal".getBytes()))).block(); + Mono.from(blobStoreDAO.save(DEFAULT_BUCKET, chunkBlob, BlobStoreDAO.BytesBlob.of("chunk".getBytes()))).block(); + + CLOCK.setInstant(NOW.plusMonths(2).toInstant()); + + Context context = new Context(EXPECTED_BLOB_COUNT, ASSOCIATED_PROBABILITY); + BloomFilterGCAlgorithm bloomFilterGCAlgorithm = bloomFilterGCAlgorithm(); + Mono.from(bloomFilterGCAlgorithm.gc(EXPECTED_BLOB_COUNT, DELETION_WINDOW_SIZE, ASSOCIATED_PROBABILITY, DEFAULT_BUCKET, context)).block(); + + assertThatThrownBy(() -> blobStoreDAO.read(DEFAULT_BUCKET, normalBlob)) + .isInstanceOf(ObjectNotFoundException.class); + + assertThat(Mono.from(blobStoreDAO.readBytes(DEFAULT_BUCKET, chunkBlob)).block().payload()) + .isEqualTo("chunk".getBytes()); + } + + @Test + default void gcShouldRemoveBlobsContainingChunkSubstringWhenNotMatchingChunkIdPattern() { + BlobStoreDAO blobStoreDAO = blobStoreDAO(); + when(BLOB_REFERENCE_SOURCE.listReferencedBlobs()).thenReturn(Flux.empty()); + + BlobId pseudoChunkBlob = GENERATION_AWARE_BLOB_ID_FACTORY.of("hash_chunk_test_value"); + + Mono.from(blobStoreDAO.save(DEFAULT_BUCKET, pseudoChunkBlob, BlobStoreDAO.BytesBlob.of("pseudo".getBytes()))).block(); + + CLOCK.setInstant(NOW.plusMonths(2).toInstant()); + + Context context = new Context(EXPECTED_BLOB_COUNT, ASSOCIATED_PROBABILITY); + BloomFilterGCAlgorithm bloomFilterGCAlgorithm = bloomFilterGCAlgorithm(); + Mono.from(bloomFilterGCAlgorithm.gc(EXPECTED_BLOB_COUNT, DELETION_WINDOW_SIZE, ASSOCIATED_PROBABILITY, DEFAULT_BUCKET, context)).block(); + + assertThatThrownBy(() -> blobStoreDAO.read(DEFAULT_BUCKET, pseudoChunkBlob)) + .isInstanceOf(ObjectNotFoundException.class); + } } From 6413283e32a195db55bf932fa1d25ddf27ac2495 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:20:41 +0530 Subject: [PATCH 05/24] JAMES-4231 Compaction Engine: Add chunk format, ChunkedBlobStoreDAO, and compaction tasks - Implement chunk packing (~100MB chunks) with per-slot Zstandard compression and 64KB suffix-indexed footers. - Introduce ChunkId and BlobSlot representations for slot addressing. - Implement ChunkedBlobStoreDAO for transparent single-request HTTP ranged reads and self-healing slot resolution via BlobIdRepairer. - Implement BlobCompactionAlgorithm with windowed candidate payload streaming (DEFAULT_CANDIDATE_BATCH_SIZE = 1000) to bound heap usage during compaction. - Implement BlobCompactionTask, DTOs, and full unit test suite. --- pom.xml | 11 + server/blob/blob-compaction/pom.xml | 160 +++++ .../compaction/BlobCompactionAlgorithm.java | 617 ++++++++++++++++++ .../compaction/BlobCompactionDTOModules.java | 35 + .../blob/compaction/BlobCompactionTask.java | 237 +++++++ ...ompactionTaskAdditionalInformationDTO.java | 144 ++++ .../compaction/BlobCompactionTaskDTO.java | 139 ++++ .../james/blob/compaction/BlobIdRepairer.java | 31 + .../james/blob/compaction/BlobIdUpdater.java | 32 + .../BlobReferenceMappingSource.java | 36 + .../james/blob/compaction/BlobSlot.java | 81 +++ .../james/blob/compaction/ChunkFooter.java | 43 ++ .../james/blob/compaction/ChunkFormat.java | 337 ++++++++++ .../apache/james/blob/compaction/ChunkId.java | 227 +++++++ .../blob/compaction/ChunkedBlobStoreDAO.java | 246 +++++++ .../compaction/CompactionConfiguration.java | 149 +++++ .../blob/compaction/CompactionRequest.java | 130 ++++ .../blob/compaction/CompactionResult.java | 161 +++++ .../BlobCompactionAlgorithmTest.java | 538 +++++++++++++++ ...ctionTaskAdditionalInformationDTOTest.java | 49 ++ .../BlobCompactionTaskSerializationTest.java | 68 ++ .../blob/compaction/ChunkFormatTest.java | 222 +++++++ .../james/blob/compaction/ChunkIdTest.java | 176 +++++ .../compaction/ChunkedBlobStoreDAOTest.java | 352 ++++++++++ .../blobCompaction.additionalInformation.json | 13 + .../resources/json/blobCompaction.task.json | 11 + server/blob/pom.xml | 1 + 27 files changed, 4246 insertions(+) create mode 100644 server/blob/blob-compaction/pom.xml create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTask.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTO.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdRepairer.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdUpdater.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFooter.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkId.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionRequest.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionResult.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTOTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskSerializationTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkIdTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java create mode 100644 server/blob/blob-compaction/src/test/resources/json/blobCompaction.additionalInformation.json create mode 100644 server/blob/blob-compaction/src/test/resources/json/blobCompaction.task.json diff --git a/pom.xml b/pom.xml index 444065f62d4..3b652a40cce 100644 --- a/pom.xml +++ b/pom.xml @@ -1144,6 +1144,17 @@ blob-common ${project.version} + + ${james.groupId} + blob-compaction + ${project.version} + + + ${james.groupId} + blob-compaction + ${project.version} + test-jar + ${james.groupId} blob-export-api diff --git a/server/blob/blob-compaction/pom.xml b/server/blob/blob-compaction/pom.xml new file mode 100644 index 00000000000..408e975ce84 --- /dev/null +++ b/server/blob/blob-compaction/pom.xml @@ -0,0 +1,160 @@ + + + + 4.0.0 + + + org.apache.james + james-server-blob + 3.10.0-SNAPSHOT + ../pom.xml + + + blob-compaction + jar + + Apache James :: Server :: Blob :: Compaction + Compacts small blobs into chunks for efficient object storage. + + + + ${james.groupId} + blob-aes + test + + + ${james.groupId} + blob-api + + + ${james.groupId} + blob-api + test-jar + test + + + ${james.groupId} + blob-memory + test + + + ${james.groupId} + blob-storage-strategy + + + ${james.groupId} + james-json + test-jar + test + + + ${james.groupId} + james-server-task-api + + + ${james.groupId} + james-server-task-json + + + ${james.groupId} + james-server-task-json + test-jar + test + + + ${james.groupId} + james-server-testing + test + + + ${james.groupId} + james-server-util + + + ${james.groupId} + metrics-tests + test + + + com.github.luben + zstd-jni + + + com.google.guava + guava + + + io.projectreactor + reactor-core + + + io.projectreactor + reactor-scala-extensions_${scala.base} + + + io.projectreactor + reactor-test + test + + + net.javacrumbs.json-unit + json-unit-assertj + test + + + org.apache.commons + commons-configuration2 + + + org.assertj + assertj-core + test + + + org.junit.jupiter + junit-jupiter-engine + test + + + org.junit.jupiter + junit-jupiter-params + test + + + org.mockito + mockito-core + test + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + + true + 1C + + + + + + diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java new file mode 100644 index 00000000000..7bd1da8e244 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -0,0 +1,617 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.Set; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobReferenceSource; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.ObjectStoreIOException; +import org.apache.james.blob.compaction.BlobReferenceMappingSource.BlobIdMessageIdMapping; +import org.apache.james.blob.compaction.ChunkFormat.BlobSlotContent; +import org.apache.james.blob.compaction.ChunkFormat.ChunkWriteResult; +import org.apache.james.blob.compaction.ChunkFormat.SlotRange; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.base.Preconditions; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.core.scheduler.Schedulers; + +/** + * Executes object compaction and garbage collection for chunked blob storage in Apache James. + * + *

Crash Safety and Liveness Invariants

+ * The compaction algorithms follow a strict step ordering to ensure crash safety and liveness: + *
    + *
  • Initial Compaction: + *
      + *
    1. Read candidates and accumulate slots into an immutable chunk.
      2. + *
    2. Save the new chunk to raw storage (unreferenced yet by source-of-truth tables).
      3. + *
    3. Update source-of-truth table references (Cassandra) to the new chunk-slot references.
      4. + *
    4. Delete original standalone objects from raw storage.
      5. + *
    + * If interrupted before step 3, the saved chunk is an orphan with 0 references and will be + * safely reclaimed by the next {@code gc-compact} run. The original blobs and references remain untouched. + * If interrupted after step 3 but before step 4, the old standalone blobs have 0 references and will be + * cleaned up by standard GC. + *
    • + *
  • GC-Compact (Rewrite / Merge / Purge): + *
      + *
    1. Read existing chunk objects and inspect slot references against the BloomFilter/mapping.
      2. + *
    2. For chunks with dead slots (or pairs of small chunks to merge), assemble a new chunk with only live slots.
      3. + *
    3. Save the new chunk to raw storage.
      4. + *
    4. Update source-of-truth table references to the new chunk slot references.
      5. + *
    5. Delete the old chunk(s) from raw storage.
      6. + *
    + * If interrupted before step 4, the newly created chunk is an orphan with no references and will be + * purged on the subsequent GC-compact pass. If interrupted after step 4, the old chunk has 0 references + * and will be purged as an orphan chunk on the next GC-compact pass. + *
    • + *
  • Orphan Chunk Purging: + * Chunks with 0 live references (100% dead slots) are identified as orphan chunks and deleted immediately. + * This guarantees self-healing and liveness by construction across process crashes. + *
    • + *
+ * + *

Memory Bounds and Operational Characteristics

+ *
    + *
  • Candidate Payload Streaming: {@link #initialCompact(CompactionRequest)} streams candidate blob identifiers + * and partitions them into windows of {@value #DEFAULT_CANDIDATE_BATCH_SIZE} blobs. Candidate payloads are fetched + * and packed chunk-by-chunk. Payloads are persisted and freed window-by-window, ensuring that candidate byte arrays + * are never held in heap for the entire generation simultaneously. Candidate payload heap usage is bounded by + * {@code O(min(candidateBatchSize * avgBlobSize, chunkTargetSize))}.
    • + *
  • Reference Mapping Memory Ceiling: {@code loadReferenceMapping()} materializes all live blob-to-messageId + * mappings for the generation into an in-memory multimap. Memory consumption is {@code O(liveGenerationReferences)} + * at approximately ~200 bytes per reference (~200MB heap for 1 million live references; ~2GB heap for 10 million). + * Because {@link BlobReferenceMappingSource} currently exposes a full stream without partition-paged query capabilities, + * this table is loaded per compaction pass. High-scale deployments exceeding tens of millions of live references per + * generation can introduce partition-paged reference lookups in future iterations.
    • + *
  • GC Compaction Memory Bounds: {@link #gcCompact(CompactionRequest)} discovers chunks by reading only trailing + * 64KB footers via HTTP ranged reads (metadata-only). Orphan chunks (100% dead slots) are deleted with 0 payload bytes read. + * During chunk purge or merge, surviving live slots are streamed individually via HTTP ranged reads, strictly bounding + * GC payload heap usage to {@code O(maxSlotSize)} (~1MB).
    • + *
+ */ +public class BlobCompactionAlgorithm { + public static final int DEFAULT_CANDIDATE_BATCH_SIZE = 1000; + private static final Logger LOGGER = LoggerFactory.getLogger(BlobCompactionAlgorithm.class); + + private final BlobStoreDAO blobStoreDAO; + private final BlobStoreDAO rawStore; + private final BlobReferenceSource referenceSource; + private final BlobReferenceMappingSource mappingSource; + private final BlobIdUpdater blobIdUpdater; + + public BlobCompactionAlgorithm(BlobStoreDAO blobStoreDAO, + BlobStoreDAO rawStore, + BlobReferenceSource referenceSource, + BlobReferenceMappingSource mappingSource, + BlobIdUpdater blobIdUpdater) { + this.blobStoreDAO = Preconditions.checkNotNull(blobStoreDAO, "'blobStoreDAO' must not be null"); + this.rawStore = Preconditions.checkNotNull(rawStore, "'rawStore' must not be null"); + this.referenceSource = Preconditions.checkNotNull(referenceSource, "'referenceSource' must not be null"); + this.mappingSource = Preconditions.checkNotNull(mappingSource, "'mappingSource' must not be null"); + this.blobIdUpdater = Preconditions.checkNotNull(blobIdUpdater, "'blobIdUpdater' must not be null"); + } + + public BlobCompactionAlgorithm(BlobStoreDAO blobStoreDAO, + BlobStoreDAO rawStore, + BlobReferenceMappingSource mappingSource, + BlobIdUpdater blobIdUpdater) { + this(blobStoreDAO, rawStore, + () -> Flux.from(mappingSource.listBlobIdMessageIdMappings()).map(BlobIdMessageIdMapping::blobId), + mappingSource, blobIdUpdater); + } + + public Mono compact(CompactionRequest request) { + return initialCompact(request) + .flatMap(initialResult -> gcCompact(request).map(initialResult::combine)); + } + + public Mono initialCompact(CompactionRequest request) { + Preconditions.checkNotNull(request, "'request' must not be null"); + + return loadReferenceMapping() + .flatMap(mapping -> { + if (mapping.isEmpty()) { + LOGGER.info("No blob references found in mapping source; skipping initial compaction for generation {}", request.generation()); + return Mono.just(CompactionResult.NONE); + } + + return Flux.from(rawStore.listBlobs(request.bucketName())) + .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) + .filter(blobId -> !ChunkId.isChunkRef(blobId)) + .filter(mapping::containsKey) + .window(DEFAULT_CANDIDATE_BATCH_SIZE) + .concatMap(windowFlux -> windowFlux + .flatMap(blobId -> Mono.from(rawStore.readBytes(request.bucketName(), blobId)) + .map(bytesBlob -> new CandidateBlob(blobId, bytesBlob.payload())) + .onErrorResume(error -> { + LOGGER.warn("Failed reading candidate blob {}", blobId.asString(), error); + return Mono.empty(); + })) + .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()) + .collectList() + .flatMap(candidates -> packAndPersistChunks(request, candidates, mapping))) + .reduce(CompactionResult.NONE, CompactionResult::combine); + }); + } + + /** + * Executes GC compaction (purge dead slots, merge small chunks, delete orphan chunks). + *

+ * Memory bound is O(maxSlotSize) (~1MB) rather than O(chunkSize) (~100MB). + * Chunk footers are read via suffix ranged reads (last 64KB) without buffering chunk payloads. + * Orphan chunks are deleted without reading any payload bytes. + * Chunks being purged or merged stream individual live slots via ranged reads. + *

+ */ + public Mono gcCompact(CompactionRequest request) { + Preconditions.checkNotNull(request, "'request' must not be null"); + + return loadReferenceMapping() + .flatMap(mapping -> Flux.from(rawStore.listBlobs(request.bucketName())) + .filter(blobId -> ChunkId.isChunkRef(blobId) && blobId.asString().indexOf('~') == -1) + .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) + .flatMap(chunkBlobId -> Mono.from(rawStore.readRange(request.bucketName(), chunkBlobId, -65536, -1)) + .map(tailSlice -> { + try { + ChunkFooter footer = ChunkFormat.readFooter(tailSlice.data(), tailSlice.totalObjectSize()); + return new ExistingChunk(chunkBlobId, tailSlice.totalObjectSize(), footer); + } catch (ObjectStoreIOException e) { + LOGGER.warn("Failed reading footer for chunk object {}", chunkBlobId.asString(), e); + return null; + } + }) + .filter(Objects::nonNull) + .onErrorResume(error -> { + LOGGER.warn("Failed reading chunk footer {}", chunkBlobId.asString(), error); + return Mono.empty(); + })) + .collectList() + .flatMap(existingChunks -> processExistingChunks(request, existingChunks, mapping))); + } + + private Mono packAndPersistChunks(CompactionRequest request, + List candidates, + Map> mapping) { + if (candidates.isEmpty()) { + return Mono.just(CompactionResult.NONE); + } + + List> batches = new ArrayList<>(); + List currentBatch = new ArrayList<>(); + long currentBatchSize = 0; + + for (CandidateBlob candidate : candidates) { + if (!currentBatch.isEmpty() && currentBatchSize + candidate.payload.length > request.configuration().chunkTargetSize()) { + batches.add(currentBatch); + currentBatch = new ArrayList<>(); + currentBatchSize = 0; + } + currentBatch.add(candidate); + currentBatchSize += candidate.payload.length; + } + if (!currentBatch.isEmpty()) { + batches.add(currentBatch); + } + + return Flux.fromIterable(batches) + .concatMap(batch -> persistChunkBatch(request, batch, mapping)) + .reduce(CompactionResult.NONE, CompactionResult::combine); + } + + private Mono persistChunkBatch(CompactionRequest request, + List batch, + Map> mapping) { + int family = request.family().orElseGet(() -> extractFamily(batch.get(0).blobId.asString())); + ChunkId chunkId = ChunkId.ofChunk(family, request.generation()); + + List slots = batch.stream() + .map(candidate -> BlobSlotContent.of(candidate.payload)) + .toList(); + + ChunkWriteResult writeResult; + try { + writeResult = ChunkFormat.writeChunk(slots); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Failed to write chunk " + chunkId.chunkId(), e)); + } + + // 1. Save chunk to raw storage + return Mono.from(rawStore.save(request.bucketName(), chunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))) + // 2. Update source-of-truth table references + .then(Flux.range(0, batch.size()) + .concatMap(i -> { + CandidateBlob candidate = batch.get(i); + SlotRange range = writeResult.slotRanges().get(i); + ChunkId slotRef = ChunkId.slotRef(chunkId, range.offset(), range.limit()); + Collection messageIds = mapping.getOrDefault(candidate.blobId, Set.of()); + return blobIdUpdater.replaceReferences(candidate.blobId, slotRef, messageIds) + .thenReturn(Optional.of(candidate)) + .onErrorResume(e -> { + LOGGER.error("Failed to update references for candidate blob {}, skipping deletion of original blob", candidate.blobId.asString(), e); + return Mono.just(Optional.empty()); + }); + }) + .flatMap(opt -> opt.map(Flux::just).orElseGet(Flux::empty)) + .collectList() + .flatMap(successfulCandidates -> Flux.fromIterable(successfulCandidates) + // 3. Delete original standalone blobs ONLY for successfully updated candidates + .concatMap(candidate -> Mono.from(rawStore.delete(request.bucketName(), candidate.blobId))) + .then() + .thenReturn(CompactionResult.builder() + .packedBlobs(successfulCandidates.size()) + .packedBytes(successfulCandidates.stream().mapToLong(c -> c.payload.length).sum()) + .chunksWritten(1) + .freedBytes(successfulCandidates.stream().mapToLong(c -> c.payload.length).sum()) + .build()))); + } + + private Mono processExistingChunks(CompactionRequest request, + List chunks, + Map> mapping) { + if (chunks.isEmpty()) { + return Mono.just(CompactionResult.NONE); + } + + List analyses = new ArrayList<>(); + for (ExistingChunk chunk : chunks) { + analyses.add(analyzeChunk(chunk, mapping)); + } + + // 1. Delete orphan chunks (100% dead slots) + List orphanChunks = analyses.stream() + .filter(a -> a.liveSlots.isEmpty()) + .toList(); + + Mono deleteOrphansMono = Flux.fromIterable(orphanChunks) + .concatMap(orphan -> Mono.from(rawStore.delete(request.bucketName(), orphan.chunk.chunkBlobId)) + .thenReturn(CompactionResult.builder() + .deadPurged(orphan.totalSlotsCount) + .freedBytes(orphan.chunk.totalChunkSize) + .build())) + .reduce(CompactionResult.NONE, CompactionResult::combine); + + List remaining = analyses.stream() + .filter(a -> !a.liveSlots.isEmpty()) + .toList(); + + // 2. Purge dead slots for chunks with dead ratio >= threshold + double purgeThreshold = Math.max(request.configuration().purgeDeadRatio(), request.configuration().gainThreshold()); + List chunksToPurge = remaining.stream() + .filter(a -> a.deadRatio() >= purgeThreshold) + .toList(); + + Mono purgeMono = Flux.fromIterable(chunksToPurge) + .concatMap(analysis -> purgeDeadSlots(request, analysis, mapping)) + .reduce(CompactionResult.NONE, CompactionResult::combine); + + // 3. Merge candidates: chunks < 50% target size + long mergeThresholdSize = (long) (request.configuration().mergeDeadRatio() * request.configuration().chunkTargetSize()); + List nonPurged = remaining.stream() + .filter(a -> a.deadRatio() < purgeThreshold) + .toList(); + + List smallChunks = nonPurged.stream() + .filter(a -> a.chunk.totalChunkSize < mergeThresholdSize) + .toList(); + + Mono mergeMono = mergeSmallChunks(request, smallChunks, mapping); + + return deleteOrphansMono + .flatMap(res1 -> purgeMono.map(res1::combine)) + .flatMap(res2 -> mergeMono.map(res2::combine)); + } + + private Mono purgeDeadSlots(CompactionRequest request, + ChunkAnalysis analysis, + Map> mapping) { + int family = extractFamily(analysis.chunk.chunkBlobId.asString()); + ChunkId newChunkId = ChunkId.ofChunk(family, request.generation()); + + return Flux.fromIterable(analysis.liveSlots) + .concatMap(liveSlot -> Mono.from(rawStore.readRange(request.bucketName(), analysis.chunk.chunkBlobId, liveSlot.offset, liveSlot.offset + liveSlot.limit - 1)) + .publishOn(Schedulers.parallel()) + .flatMap(slice -> { + try { + byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), liveSlot.offset); + return Mono.just(new LiveSlotWithContent(liveSlot, decompressed)); + } catch (ObjectStoreIOException e) { + return Mono.error(e); + } + })) + .collectList() + .flatMap(liveSlotsWithContent -> { + List liveSlotContents = liveSlotsWithContent.stream() + .map(slot -> BlobSlotContent.of(slot.decompressedContent)) + .toList(); + + ChunkWriteResult writeResult; + try { + writeResult = ChunkFormat.writeChunk(liveSlotContents); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Failed writing purged chunk " + newChunkId.chunkId(), e)); + } + + // 1. Save new chunk to raw storage + return Mono.from(rawStore.save(request.bucketName(), newChunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))) + // 2. Update references to the new slot refs + .then(Flux.range(0, liveSlotsWithContent.size()) + .concatMap(i -> { + LiveSlotWithContent liveSlot = liveSlotsWithContent.get(i); + SlotRange range = writeResult.slotRanges().get(i); + ChunkId newSlotRef = ChunkId.slotRef(newChunkId, range.offset(), range.limit()); + return blobIdUpdater.replaceReferences(liveSlot.meta.slotRef, newSlotRef, liveSlot.meta.messageIds); + }) + .then()) + // 3. Delete old chunk from raw storage + .then(Mono.from(rawStore.delete(request.bucketName(), analysis.chunk.chunkBlobId))) + .thenReturn(CompactionResult.builder() + .deadPurged(analysis.deadSlotsCount) + .freedBytes(Math.max(0, analysis.chunk.totalChunkSize - writeResult.chunkBytes().length)) + .build()); + }); + } + + private Mono mergeSmallChunks(CompactionRequest request, + List smallChunks, + Map> mapping) { + if (smallChunks.size() < 2) { + return Mono.just(CompactionResult.NONE); + } + + List pairs = new ArrayList<>(); + int i = 0; + while (i + 1 < smallChunks.size()) { + ChunkAnalysis c1 = smallChunks.get(i); + ChunkAnalysis c2 = smallChunks.get(i + 1); + if (c1.chunk.totalChunkSize + c2.chunk.totalChunkSize <= request.configuration().chunkTargetSize()) { + pairs.add(new ChunkPair(c1, c2)); + i += 2; + } else { + i++; + } + } + + return Flux.fromIterable(pairs) + .concatMap(pair -> mergePair(request, pair)) + .reduce(CompactionResult.NONE, CompactionResult::combine); + } + + private Mono mergePair(CompactionRequest request, ChunkPair pair) { + int family = extractFamily(pair.c1.chunk.chunkBlobId.asString()); + ChunkId newChunkId = ChunkId.ofChunk(family, request.generation()); + + Mono> c1Live = Flux.fromIterable(pair.c1.liveSlots) + .concatMap(slot -> Mono.from(rawStore.readRange(request.bucketName(), pair.c1.chunk.chunkBlobId, slot.offset, slot.offset + slot.limit - 1)) + .publishOn(Schedulers.parallel()) + .flatMap(slice -> { + try { + byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), slot.offset); + return Mono.just(new LiveSlotWithContent(slot, decompressed)); + } catch (ObjectStoreIOException e) { + return Mono.error(e); + } + })) + .collectList(); + + Mono> c2Live = Flux.fromIterable(pair.c2.liveSlots) + .concatMap(slot -> Mono.from(rawStore.readRange(request.bucketName(), pair.c2.chunk.chunkBlobId, slot.offset, slot.offset + slot.limit - 1)) + .publishOn(Schedulers.parallel()) + .flatMap(slice -> { + try { + byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), slot.offset); + return Mono.just(new LiveSlotWithContent(slot, decompressed)); + } catch (ObjectStoreIOException e) { + return Mono.error(e); + } + })) + .collectList(); + + return Mono.zip(c1Live, c2Live) + .flatMap(tuple -> { + List allLive = new ArrayList<>(tuple.getT1()); + allLive.addAll(tuple.getT2()); + + List slotContents = allLive.stream() + .map(slot -> BlobSlotContent.of(slot.decompressedContent)) + .toList(); + + ChunkWriteResult writeResult; + try { + writeResult = ChunkFormat.writeChunk(slotContents); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Failed to write merged chunk " + newChunkId.chunkId(), e)); + } + + // 1. Save merged chunk + return Mono.from(rawStore.save(request.bucketName(), newChunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))) + // 2. Update table references for all slots in both chunks + .then(Flux.range(0, allLive.size()) + .concatMap(j -> { + LiveSlotWithContent liveSlot = allLive.get(j); + SlotRange range = writeResult.slotRanges().get(j); + ChunkId newSlotRef = ChunkId.slotRef(newChunkId, range.offset(), range.limit()); + return blobIdUpdater.replaceReferences(liveSlot.meta.slotRef, newSlotRef, liveSlot.meta.messageIds); + }) + .then()) + // 3. Delete both old chunks + .then(Mono.from(rawStore.delete(request.bucketName(), pair.c1.chunk.chunkBlobId))) + .then(Mono.from(rawStore.delete(request.bucketName(), pair.c2.chunk.chunkBlobId))) + .thenReturn(CompactionResult.builder() + .mergedChunks(2) + .freedBytes(Math.max(0, (pair.c1.chunk.totalChunkSize + pair.c2.chunk.totalChunkSize) - writeResult.chunkBytes().length)) + .build()); + }); + } + + private ChunkAnalysis analyzeChunk(ExistingChunk chunk, Map> mapping) { + ChunkId chunkId = ChunkId.parseChunk(chunk.chunkBlobId.asString()); + ChunkFooter footer = chunk.footer; + List starts = footer.slotStarts(); + + List liveSlots = new ArrayList<>(); + int deadCount = 0; + long deadBytes = 0; + long totalSlotBytes = 0; + + for (int i = 0; i < starts.size(); i++) { + long offset = starts.get(i); + long limit = footer.slotLength(i); + totalSlotBytes += limit; + + ChunkId slotRef = ChunkId.slotRef(chunkId, offset, limit); + Set messageIds = findMessageIdsForSlot(mapping, slotRef); + + if (messageIds.isEmpty()) { + deadCount++; + deadBytes += limit; + } else { + liveSlots.add(new LiveSlotMeta(offset, limit, slotRef, messageIds)); + } + } + + return new ChunkAnalysis(chunk, liveSlots, starts.size(), deadCount, deadBytes, totalSlotBytes); + } + + private Set findMessageIdsForSlot(Map> mapping, ChunkId slotRef) { + Set direct = mapping.get(slotRef); + if (direct != null && !direct.isEmpty()) { + return direct; + } + ChunkId zeroLimit = ChunkId.slotRef(slotRef.chunkBlobId().asString(), slotRef.offset(), 0L); + Set zero = mapping.get(zeroLimit); + if (zero != null && !zero.isEmpty()) { + return zero; + } + for (Map.Entry> entry : mapping.entrySet()) { + String keyStr = entry.getKey().asString(); + if (ChunkId.isChunkRef(keyStr)) { + try { + ChunkId parsed = ChunkId.parse(keyStr); + if (parsed.chunkId().equals(slotRef.chunkId()) && parsed.offset() == slotRef.offset()) { + return entry.getValue(); + } + } catch (Exception e) { + // ignore + } + } + } + return Set.of(); + } + + /** + * Loads generation live references from {@link BlobReferenceMappingSource}. + *

+ * Operational ceiling: Materializes the generation's reference multimap into memory. + * Memory consumption is O(liveReferences) * ~200 bytes/reference. + * High-scale deployments with tens of millions of references can introduce partition-paged + * reference lookups in future iterations. + *

+ */ + private Mono>> loadReferenceMapping() { + return Flux.from(mappingSource.listBlobIdMessageIdMappings()) + .collectMultimap(BlobIdMessageIdMapping::blobId, BlobIdMessageIdMapping::messageId) + .map(multimap -> { + Map> result = new HashMap<>(); + multimap.forEach((k, v) -> result.put(k, new HashSet<>(v))); + return result; + }); + } + + static boolean matchesGenerationAndFamily(String blobIdStr, long targetGeneration, Optional targetFamily) { + char sep = '_'; + int firstSep = blobIdStr.indexOf(sep); + if (firstSep == -1) { + sep = '/'; + firstSep = blobIdStr.indexOf(sep); + } + if (firstSep == -1) { + return false; + } + int secondSep = blobIdStr.indexOf(sep, firstSep + 1); + if (secondSep == -1) { + return false; + } + try { + int family = Integer.parseInt(blobIdStr.substring(0, firstSep)); + long generation = Long.parseLong(blobIdStr.substring(firstSep + 1, secondSep)); + if (generation != targetGeneration) { + return false; + } + return targetFamily.map(f -> f == family).orElse(true); + } catch (NumberFormatException e) { + return false; + } + } + + static int extractFamily(String blobIdStr) { + int firstSep = blobIdStr.indexOf('_'); + if (firstSep == -1) { + firstSep = blobIdStr.indexOf('/'); + } + if (firstSep > 0) { + try { + return Integer.parseInt(blobIdStr.substring(0, firstSep)); + } catch (NumberFormatException e) { + // fallback + } + } + return 1; + } + + private record CandidateBlob(BlobId blobId, byte[] payload) {} + + private record ExistingChunk(BlobId chunkBlobId, long totalChunkSize, ChunkFooter footer) {} + + private record LiveSlotMeta(long offset, long limit, ChunkId slotRef, Set messageIds) {} + + private record LiveSlotWithContent(LiveSlotMeta meta, byte[] decompressedContent) {} + + private record ChunkPair(ChunkAnalysis c1, ChunkAnalysis c2) {} + + private record ChunkAnalysis(ExistingChunk chunk, + List liveSlots, + int totalSlotsCount, + int deadSlotsCount, + long deadBytes, + long totalSlotBytes) { + double deadRatio() { + if (totalSlotBytes == 0) { + return 1.0; + } + return (double) deadBytes / totalSlotBytes; + } + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java new file mode 100644 index 00000000000..2f66d5c13fb --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java @@ -0,0 +1,35 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Clock; + +import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule; +import org.apache.james.server.task.json.dto.TaskDTOModule; + +public class BlobCompactionDTOModules { + public static TaskDTOModule taskModule(BlobCompactionAlgorithm algorithm, Clock clock) { + return BlobCompactionTaskDTO.module(algorithm, clock); + } + + public static AdditionalInformationDTOModule additionalInformationModule() { + return BlobCompactionTaskAdditionalInformationDTO.SERIALIZATION_MODULE; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTask.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTask.java new file mode 100644 index 00000000000..7e5d2e1742a --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTask.java @@ -0,0 +1,237 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Clock; +import java.time.Instant; +import java.util.Objects; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicReference; + +import org.apache.james.task.Task; +import org.apache.james.task.TaskExecutionDetails; +import org.apache.james.task.TaskType; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.base.MoreObjects; +import com.google.common.base.Preconditions; + +public class BlobCompactionTask implements Task { + public static final TaskType TASK_TYPE = TaskType.of("BlobCompactionTask"); + private static final Logger LOGGER = LoggerFactory.getLogger(BlobCompactionTask.class); + + public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation { + private final Instant timestamp; + private final String bucketName; + private final long generation; + private final Optional family; + private final long packedBlobs; + private final long packedBytes; + private final long chunksWritten; + private final long deadPurged; + private final long mergedChunks; + private final long freedBytes; + + public AdditionalInformation(Instant timestamp, + String bucketName, + long generation, + Optional family, + long packedBlobs, + long packedBytes, + long chunksWritten, + long deadPurged, + long mergedChunks, + long freedBytes) { + this.timestamp = Preconditions.checkNotNull(timestamp, "'timestamp' must not be null"); + this.bucketName = Preconditions.checkNotNull(bucketName, "'bucketName' must not be null"); + this.generation = generation; + this.family = Preconditions.checkNotNull(family, "'family' must not be null"); + this.packedBlobs = packedBlobs; + this.packedBytes = packedBytes; + this.chunksWritten = chunksWritten; + this.deadPurged = deadPurged; + this.mergedChunks = mergedChunks; + this.freedBytes = freedBytes; + } + + @Override + public Instant timestamp() { + return timestamp; + } + + public Instant getTimestamp() { + return timestamp; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public long getPackedBlobs() { + return packedBlobs; + } + + public long getPackedBytes() { + return packedBytes; + } + + public long getChunksWritten() { + return chunksWritten; + } + + public long getDeadPurged() { + return deadPurged; + } + + public long getMergedChunks() { + return mergedChunks; + } + + public long getFreedBytes() { + return freedBytes; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof AdditionalInformation that) { + return generation == that.generation + && packedBlobs == that.packedBlobs + && packedBytes == that.packedBytes + && chunksWritten == that.chunksWritten + && deadPurged == that.deadPurged + && mergedChunks == that.mergedChunks + && freedBytes == that.freedBytes + && Objects.equals(timestamp, that.timestamp) + && Objects.equals(bucketName, that.bucketName) + && Objects.equals(family, that.family); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(timestamp, bucketName, generation, family, packedBlobs, packedBytes, chunksWritten, deadPurged, mergedChunks, freedBytes); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this) + .add("timestamp", timestamp) + .add("bucketName", bucketName) + .add("generation", generation) + .add("family", family) + .add("packedBlobs", packedBlobs) + .add("packedBytes", packedBytes) + .add("chunksWritten", chunksWritten) + .add("deadPurged", deadPurged) + .add("mergedChunks", mergedChunks) + .add("freedBytes", freedBytes) + .toString(); + } + } + + private final BlobCompactionAlgorithm algorithm; + private final CompactionRequest request; + private final Clock clock; + private final AtomicReference currentResult; + + public BlobCompactionTask(BlobCompactionAlgorithm algorithm, CompactionRequest request, Clock clock) { + this.algorithm = Preconditions.checkNotNull(algorithm, "'algorithm' must not be null"); + this.request = Preconditions.checkNotNull(request, "'request' must not be null"); + this.clock = Preconditions.checkNotNull(clock, "'clock' must not be null"); + this.currentResult = new AtomicReference<>(CompactionResult.NONE); + } + + @Override + public Result run() { + try { + CompactionResult result = algorithm.compact(request).block(); + if (result != null) { + currentResult.set(result); + } + return Result.COMPLETED; + } catch (Exception e) { + LOGGER.error("Error while running BlobCompactionTask for generation {}", request.generation(), e); + return Result.PARTIAL; + } + } + + @Override + public TaskType type() { + return TASK_TYPE; + } + + @Override + public Optional details() { + CompactionResult res = currentResult.get(); + return Optional.of(new AdditionalInformation( + clock.instant(), + request.bucketName().asString(), + request.generation(), + request.family(), + res.packedBlobs(), + res.packedBytes(), + res.chunksWritten(), + res.deadPurged(), + res.mergedChunks(), + res.freedBytes() + )); + } + + public CompactionRequest getRequest() { + return request; + } + + public Clock getClock() { + return clock; + } + + public BlobCompactionAlgorithm getAlgorithm() { + return algorithm; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof BlobCompactionTask that) { + return Objects.equals(request, that.request); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(request); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTO.java new file mode 100644 index 00000000000..7eaf26928c1 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTO.java @@ -0,0 +1,144 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Instant; +import java.util.Optional; + +import org.apache.james.json.DTOModule; +import org.apache.james.server.task.json.dto.AdditionalInformationDTO; +import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class BlobCompactionTaskAdditionalInformationDTO implements AdditionalInformationDTO { + + public static final AdditionalInformationDTOModule SERIALIZATION_MODULE = + DTOModule.forDomainObject(BlobCompactionTask.AdditionalInformation.class) + .convertToDTO(BlobCompactionTaskAdditionalInformationDTO.class) + .toDomainObjectConverter(dto -> + new BlobCompactionTask.AdditionalInformation( + dto.timestamp, + dto.bucketName, + dto.generation, + dto.family, + dto.packedBlobs, + dto.packedBytes, + dto.chunksWritten, + dto.deadPurged, + dto.mergedChunks, + dto.freedBytes)) + .toDTOConverter((domain, type) -> + new BlobCompactionTaskAdditionalInformationDTO( + type, + domain.getTimestamp(), + domain.getBucketName(), + domain.getGeneration(), + domain.getFamily(), + domain.getPackedBlobs(), + domain.getPackedBytes(), + domain.getChunksWritten(), + domain.getDeadPurged(), + domain.getMergedChunks(), + domain.getFreedBytes())) + .typeName(BlobCompactionTask.TASK_TYPE.asString()) + .withFactory(AdditionalInformationDTOModule::new); + + private final String type; + private final Instant timestamp; + private final String bucketName; + private final long generation; + private final Optional family; + private final long packedBlobs; + private final long packedBytes; + private final long chunksWritten; + private final long deadPurged; + private final long mergedChunks; + private final long freedBytes; + + public BlobCompactionTaskAdditionalInformationDTO(@JsonProperty("type") String type, + @JsonProperty("timestamp") Instant timestamp, + @JsonProperty("bucketName") String bucketName, + @JsonProperty("generation") long generation, + @JsonProperty("family") Optional family, + @JsonProperty("packedBlobs") long packedBlobs, + @JsonProperty("packedBytes") long packedBytes, + @JsonProperty("chunksWritten") long chunksWritten, + @JsonProperty("deadPurged") long deadPurged, + @JsonProperty("mergedChunks") long mergedChunks, + @JsonProperty("freedBytes") long freedBytes) { + this.type = type; + this.timestamp = timestamp; + this.bucketName = bucketName; + this.generation = generation; + this.family = family; + this.packedBlobs = packedBlobs; + this.packedBytes = packedBytes; + this.chunksWritten = chunksWritten; + this.deadPurged = deadPurged; + this.mergedChunks = mergedChunks; + this.freedBytes = freedBytes; + } + + @Override + public String getType() { + return type; + } + + public Instant getTimestamp() { + return timestamp; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public long getPackedBlobs() { + return packedBlobs; + } + + public long getPackedBytes() { + return packedBytes; + } + + public long getChunksWritten() { + return chunksWritten; + } + + public long getDeadPurged() { + return deadPurged; + } + + public long getMergedChunks() { + return mergedChunks; + } + + public long getFreedBytes() { + return freedBytes; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java new file mode 100644 index 00000000000..73b66baf31b --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java @@ -0,0 +1,139 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Clock; +import java.util.Optional; + +import org.apache.james.blob.api.BucketName; +import org.apache.james.json.DTOModule; +import org.apache.james.server.task.json.dto.TaskDTO; +import org.apache.james.server.task.json.dto.TaskDTOModule; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class BlobCompactionTaskDTO implements TaskDTO { + + private final String type; + private final String bucketName; + private final long generation; + private final Optional family; + private final Optional chunkTargetSize; + private final Optional maxPackableSize; + private final Optional purgeDeadRatio; + private final Optional mergeDeadRatio; + private final Optional gainThreshold; + + public BlobCompactionTaskDTO(@JsonProperty("type") String type, + @JsonProperty("bucketName") String bucketName, + @JsonProperty("generation") long generation, + @JsonProperty("family") Optional family, + @JsonProperty("chunkTargetSize") Optional chunkTargetSize, + @JsonProperty("maxPackableSize") Optional maxPackableSize, + @JsonProperty("purgeDeadRatio") Optional purgeDeadRatio, + @JsonProperty("mergeDeadRatio") Optional mergeDeadRatio, + @JsonProperty("gainThreshold") Optional gainThreshold) { + this.type = type; + this.bucketName = bucketName; + this.generation = generation; + this.family = family; + this.chunkTargetSize = chunkTargetSize; + this.maxPackableSize = maxPackableSize; + this.purgeDeadRatio = purgeDeadRatio; + this.mergeDeadRatio = mergeDeadRatio; + this.gainThreshold = gainThreshold; + } + + public static TaskDTOModule module(BlobCompactionAlgorithm algorithm, Clock clock) { + return DTOModule.forDomainObject(BlobCompactionTask.class) + .convertToDTO(BlobCompactionTaskDTO.class) + .toDomainObjectConverter(dto -> { + CompactionConfiguration.Builder configBuilder = CompactionConfiguration.builder(); + dto.getChunkTargetSize().ifPresent(configBuilder::chunkTargetSize); + dto.getMaxPackableSize().ifPresent(configBuilder::maxPackableSize); + dto.getPurgeDeadRatio().ifPresent(configBuilder::purgeDeadRatio); + dto.getMergeDeadRatio().ifPresent(configBuilder::mergeDeadRatio); + dto.getGainThreshold().ifPresent(configBuilder::gainThreshold); + + CompactionRequest.Builder requestBuilder = CompactionRequest.builder() + .bucketName(BucketName.of(dto.getBucketName())) + .generation(dto.getGeneration()) + .configuration(configBuilder.build()); + + dto.getFamily().ifPresent(requestBuilder::family); + + return new BlobCompactionTask(algorithm, requestBuilder.build(), clock); + }) + .toDTOConverter((domain, type) -> { + CompactionRequest req = domain.getRequest(); + CompactionConfiguration conf = req.configuration(); + return new BlobCompactionTaskDTO( + type, + req.bucketName().asString(), + req.generation(), + req.family(), + Optional.of(conf.chunkTargetSize()), + Optional.of(conf.maxPackableSize()), + Optional.of(conf.purgeDeadRatio()), + Optional.of(conf.mergeDeadRatio()), + Optional.of(conf.gainThreshold()) + ); + }) + .typeName(BlobCompactionTask.TASK_TYPE.asString()) + .withFactory(TaskDTOModule::new); + } + + @Override + public String getType() { + return type; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public Optional getChunkTargetSize() { + return chunkTargetSize; + } + + public Optional getMaxPackableSize() { + return maxPackableSize; + } + + public Optional getPurgeDeadRatio() { + return purgeDeadRatio; + } + + public Optional getMergeDeadRatio() { + return mergeDeadRatio; + } + + public Optional getGainThreshold() { + return gainThreshold; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdRepairer.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdRepairer.java new file mode 100644 index 00000000000..7dd89751711 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdRepairer.java @@ -0,0 +1,31 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BucketName; + +import reactor.core.publisher.Mono; + +public interface BlobIdRepairer { + BlobIdRepairer NOOP = (bucketName, blobId) -> Mono.empty(); + + Mono repair(BucketName bucketName, BlobId staleBlobId); +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdUpdater.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdUpdater.java new file mode 100644 index 00000000000..0d32e6a8b7c --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobIdUpdater.java @@ -0,0 +1,32 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.util.Collection; + +import org.apache.james.blob.api.BlobId; + +import reactor.core.publisher.Mono; + +public interface BlobIdUpdater { + BlobIdUpdater NOOP = (oldId, newId, messageIds) -> Mono.empty(); + + Mono replaceReferences(BlobId oldId, BlobId newId, Collection messageIds); +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java new file mode 100644 index 00000000000..4088340d03f --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java @@ -0,0 +1,36 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import org.apache.james.blob.api.BlobId; +import org.reactivestreams.Publisher; + +import com.google.common.base.Preconditions; + +public interface BlobReferenceMappingSource { + record BlobIdMessageIdMapping(BlobId blobId, String messageId) { + public BlobIdMessageIdMapping { + Preconditions.checkNotNull(blobId, "'blobId' must not be null"); + Preconditions.checkNotNull(messageId, "'messageId' must not be null"); + } + } + + Publisher listBlobIdMessageIdMappings(); +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java new file mode 100644 index 00000000000..311c108579c --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java @@ -0,0 +1,81 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.util.Arrays; +import java.util.Objects; +import java.util.zip.CRC32C; + +import org.apache.james.blob.api.ObjectStoreIOException; + +import com.github.luben.zstd.Zstd; + +public record BlobSlot(long contentStart, int crc32c, long originalSize, byte[] compressedContent) { + public byte[] decompress() throws ObjectStoreIOException { + if (originalSize < 0 || originalSize > Integer.MAX_VALUE) { + throw new ObjectStoreIOException("Invalid original size in chunk slot: " + originalSize); + } + byte[] decompressed; + if (originalSize == 0) { + decompressed = new byte[0]; + } else { + try { + decompressed = Zstd.decompress(compressedContent, (int) originalSize); + } catch (Exception e) { + throw new ObjectStoreIOException("Failed to decompress slot content at " + contentStart, e); + } + } + + if (decompressed.length != originalSize) { + throw new ObjectStoreIOException("Decompressed size " + decompressed.length + " does not match expected " + originalSize); + } + + CRC32C crc = new CRC32C(); + crc.update(decompressed); + int computedCrc = (int) crc.getValue(); + if (computedCrc != crc32c) { + throw new ObjectStoreIOException(String.format("CRC mismatch for chunk slot at offset %d: expected %d, got %d", + contentStart, crc32c, computedCrc)); + } + + return decompressed; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof BlobSlot other) { + return contentStart == other.contentStart + && crc32c == other.crc32c + && originalSize == other.originalSize + && Arrays.equals(compressedContent, other.compressedContent); + } + return false; + } + + @Override + public int hashCode() { + int result = Objects.hash(contentStart, crc32c, originalSize); + result = 31 * result + Arrays.hashCode(compressedContent); + return result; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFooter.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFooter.java new file mode 100644 index 00000000000..08eca455b6a --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFooter.java @@ -0,0 +1,43 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.util.List; + +import com.google.common.collect.ImmutableList; + +public record ChunkFooter(List slotStarts, long footerPosition) { + public ChunkFooter { + slotStarts = ImmutableList.copyOf(slotStarts); + } + + public int slotCount() { + return slotStarts.size(); + } + + public long slotLength(int slotIndex) { + if (slotIndex < 0 || slotIndex >= slotStarts.size()) { + throw new IndexOutOfBoundsException("Slot index out of bounds: " + slotIndex); + } + long start = slotStarts.get(slotIndex); + long end = (slotIndex == slotStarts.size() - 1) ? footerPosition : slotStarts.get(slotIndex + 1); + return end - start; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java new file mode 100644 index 00000000000..b1ca36e896f --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java @@ -0,0 +1,337 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; +import java.util.zip.CRC32C; + +import org.apache.james.blob.api.ObjectStoreIOException; + +import com.github.luben.zstd.Zstd; +import com.google.common.base.Preconditions; +import com.google.common.base.Splitter; +import com.google.common.io.ByteStreams; + +public class ChunkFormat { + public static final byte FORMAT_BYTE = 0x01; + public static final int FOOTER_METADATA_LENGTH = 12; // 4 bytes footerLength + 8 bytes footerPosition + public static final String METADATA_ENCODING = "content-encoding=zstd\n"; + public static final String METADATA_SIZE_PREFIX = "content-original-size="; + + public record BlobSlotContent(byte[] rawContent, long originalSize) { + public static BlobSlotContent of(byte[] rawContent) { + Preconditions.checkNotNull(rawContent, "'rawContent' must not be null"); + return new BlobSlotContent(rawContent, rawContent.length); + } + } + + public static void write(List slots, OutputStream outputStream) throws IOException { + Preconditions.checkNotNull(slots, "'slots' must not be null"); + Preconditions.checkNotNull(outputStream, "'outputStream' must not be null"); + + DataOutputStream dataOut = new DataOutputStream(outputStream); + dataOut.writeByte(FORMAT_BYTE); + long currentOffset = 1L; + + List slotStarts = new ArrayList<>(slots.size()); + + for (BlobSlotContent slot : slots) { + byte[] raw = slot.rawContent(); + long originalSize = slot.originalSize(); + + CRC32C crc = new CRC32C(); + crc.update(raw); + int crc32c = (int) crc.getValue(); + + byte[] compressed; + if (raw.length == 0) { + compressed = new byte[0]; + } else { + compressed = Zstd.compress(raw); + } + + String metadata = METADATA_ENCODING + METADATA_SIZE_PREFIX + originalSize + "\n"; + byte[] metadataBytes = metadata.getBytes(StandardCharsets.US_ASCII); + + slotStarts.add(currentOffset); + + dataOut.writeLong(currentOffset); + dataOut.writeInt(crc32c); + dataOut.write(metadataBytes); + dataOut.write(compressed); + + currentOffset += 8L + 4L + metadataBytes.length + compressed.length; + } + + long footerPosition = currentOffset; + String footerString = slotStarts.stream() + .map(String::valueOf) + .collect(Collectors.joining(",")); + byte[] footerBytes = footerString.getBytes(StandardCharsets.US_ASCII); + + dataOut.write(footerBytes); + dataOut.writeInt(footerBytes.length); + dataOut.writeLong(footerPosition); + dataOut.flush(); + } + + public static byte[] writeToBytes(List slots) throws IOException { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + write(slots, baos); + return baos.toByteArray(); + } + + public record SlotRange(long offset, long limit) {} + + public record ChunkWriteResult(byte[] chunkBytes, List slotRanges) {} + + public static ChunkWriteResult writeChunk(List slots) throws IOException { + Preconditions.checkNotNull(slots, "'slots' must not be null"); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + DataOutputStream dataOut = new DataOutputStream(baos); + dataOut.writeByte(FORMAT_BYTE); + long currentOffset = 1L; + + List slotStarts = new ArrayList<>(slots.size()); + List slotLengths = new ArrayList<>(slots.size()); + + for (BlobSlotContent slot : slots) { + byte[] raw = slot.rawContent(); + long originalSize = slot.originalSize(); + + CRC32C crc = new CRC32C(); + crc.update(raw); + int crc32c = (int) crc.getValue(); + + byte[] compressed; + if (raw.length == 0) { + compressed = new byte[0]; + } else { + compressed = Zstd.compress(raw); + } + + String metadata = METADATA_ENCODING + METADATA_SIZE_PREFIX + originalSize + "\n"; + byte[] metadataBytes = metadata.getBytes(StandardCharsets.US_ASCII); + + slotStarts.add(currentOffset); + long slotLength = 8L + 4L + metadataBytes.length + compressed.length; + slotLengths.add(slotLength); + + dataOut.writeLong(currentOffset); + dataOut.writeInt(crc32c); + dataOut.write(metadataBytes); + dataOut.write(compressed); + + currentOffset += slotLength; + } + + long footerPosition = currentOffset; + String footerString = slotStarts.stream() + .map(String::valueOf) + .collect(Collectors.joining(",")); + byte[] footerBytes = footerString.getBytes(StandardCharsets.US_ASCII); + + dataOut.write(footerBytes); + dataOut.writeInt(footerBytes.length); + dataOut.writeLong(footerPosition); + dataOut.flush(); + + List ranges = new ArrayList<>(slots.size()); + for (int i = 0; i < slots.size(); i++) { + ranges.add(new SlotRange(slotStarts.get(i), slotLengths.get(i))); + } + + return new ChunkWriteResult(baos.toByteArray(), ranges); + } + + public record ParsedSlot(long offset, long limit, byte[] decompressedContent) {} + + public static List parseAllSlots(byte[] chunkBytes) throws ObjectStoreIOException { + Preconditions.checkNotNull(chunkBytes, "'chunkBytes' must not be null"); + ChunkFooter footer = readFooter(chunkBytes); + List starts = footer.slotStarts(); + long footerPosition = footer.footerPosition(); + List result = new ArrayList<>(starts.size()); + for (int i = 0; i < starts.size(); i++) { + long offset = starts.get(i); + long end = (i + 1 < starts.size()) ? starts.get(i + 1) : footerPosition; + long limit = end - offset; + byte[] slotSlice = Arrays.copyOfRange(chunkBytes, (int) offset, (int) end); + byte[] decompressed = parseSlotBytes(slotSlice, offset); + result.add(new ParsedSlot(offset, limit, decompressed)); + } + return result; + } + + public static ChunkFooter readFooter(byte[] tailBuffer) throws ObjectStoreIOException { + Preconditions.checkNotNull(tailBuffer, "'tailBuffer' must not be null"); + return readFooter(tailBuffer, tailBuffer.length); + } + + public static ChunkFooter readFooter(byte[] tailBuffer, long totalObjectSize) throws ObjectStoreIOException { + Preconditions.checkNotNull(tailBuffer, "'tailBuffer' must not be null"); + if (tailBuffer.length < FOOTER_METADATA_LENGTH) { + throw new ObjectStoreIOException("Tail buffer is too short to contain chunk footer metadata: " + tailBuffer.length); + } + if (totalObjectSize < FOOTER_METADATA_LENGTH + 1) { // 1 byte format + 12 bytes footer + throw new ObjectStoreIOException("Total object size is too small to be a valid chunk: " + totalObjectSize); + } + + int bufferLen = tailBuffer.length; + ByteBuffer buffer = ByteBuffer.wrap(tailBuffer); + int footerLength = buffer.getInt(bufferLen - FOOTER_METADATA_LENGTH); + long footerPosition = buffer.getLong(bufferLen - 8); + + if (footerLength < 0) { + throw new ObjectStoreIOException("Corrupt footer length: " + footerLength); + } + if (footerPosition < 1 || footerPosition > totalObjectSize - FOOTER_METADATA_LENGTH) { + throw new ObjectStoreIOException("Corrupt footer position: " + footerPosition + " for total size " + totalObjectSize); + } + if (footerPosition + footerLength != totalObjectSize - FOOTER_METADATA_LENGTH) { + throw new ObjectStoreIOException(String.format("Footer integrity mismatch: footerPosition(%d) + footerLength(%d) != expected(%d)", + footerPosition, footerLength, totalObjectSize - FOOTER_METADATA_LENGTH)); + } + + long tailBufferStartOffset = totalObjectSize - tailBuffer.length; + if (footerPosition < tailBufferStartOffset) { + throw new ObjectStoreIOException("Footer start offset " + footerPosition + " is before tail buffer start " + tailBufferStartOffset); + } + + int footerOffsetInTail = (int) (footerPosition - tailBufferStartOffset); + if (footerOffsetInTail + footerLength > bufferLen - FOOTER_METADATA_LENGTH) { + throw new ObjectStoreIOException("Footer exceeds tail buffer bounds"); + } + + if (footerLength == 0) { + return new ChunkFooter(List.of(), footerPosition); + } + + String footerString = new String(tailBuffer, footerOffsetInTail, footerLength, StandardCharsets.US_ASCII); + List slotStarts = Splitter.on(',') + .trimResults() + .omitEmptyStrings() + .splitToStream(footerString) + .map(s -> { + try { + return Long.parseLong(s); + } catch (NumberFormatException e) { + throw new IllegalArgumentException("Corrupt contentStart value in footer: " + s, e); + } + }) + .collect(Collectors.toList()); + + return new ChunkFooter(slotStarts, footerPosition); + } + + public static byte[] readSlot(InputStream chunkRangeStream, long contentStart, long nextContentStart) throws ObjectStoreIOException { + Preconditions.checkNotNull(chunkRangeStream, "'chunkRangeStream' must not be null"); + Preconditions.checkArgument(nextContentStart > contentStart, + "nextContentStart (%s) must be strictly greater than contentStart (%s)", nextContentStart, contentStart); + + long slotLength = nextContentStart - contentStart; + if (slotLength > Integer.MAX_VALUE) { + throw new ObjectStoreIOException("Slot length exceeds maximum supported size: " + slotLength); + } + + byte[] slotBytes; + try { + slotBytes = ByteStreams.toByteArray(ByteStreams.limit(chunkRangeStream, slotLength)); + } catch (IOException e) { + throw new ObjectStoreIOException("Failed reading slot bytes from stream at " + contentStart, e); + } + + if (slotBytes.length != slotLength) { + throw new ObjectStoreIOException(String.format("Truncated slot bytes at %d: expected %d bytes, got %d", + contentStart, slotLength, slotBytes.length)); + } + + return parseSlotBytes(slotBytes, contentStart); + } + + public static byte[] parseSlotBytes(byte[] slotBytes, long expectedContentStart) throws ObjectStoreIOException { + if (slotBytes.length < 8 + 4 + 2) { // 8B contentStart, 4B crc, at least 2 bytes metadata + throw new ObjectStoreIOException("Slot byte array too short: " + slotBytes.length); + } + + ByteBuffer buffer = ByteBuffer.wrap(slotBytes); + long actualContentStart = buffer.getLong(0); + if (actualContentStart != expectedContentStart) { + throw new ObjectStoreIOException(String.format("Slot contentStart mismatch: expected %d, got %d", + expectedContentStart, actualContentStart)); + } + + int crc32c = buffer.getInt(8); + + // Find metadata lines (up to second '\n') + int firstNewline = -1; + int secondNewline = -1; + for (int i = 12; i < slotBytes.length; i++) { + if (slotBytes[i] == '\n') { + if (firstNewline == -1) { + firstNewline = i; + } else { + secondNewline = i; + break; + } + } + } + + if (firstNewline == -1 || secondNewline == -1) { + throw new ObjectStoreIOException("Corrupt slot metadata: missing newlines in header at " + expectedContentStart); + } + + String encodingLine = new String(slotBytes, 12, firstNewline - 12, StandardCharsets.US_ASCII); + if (!"content-encoding=zstd".equals(encodingLine.trim())) { + throw new ObjectStoreIOException("Unsupported slot encoding: " + encodingLine); + } + + String sizeLine = new String(slotBytes, firstNewline + 1, secondNewline - (firstNewline + 1), StandardCharsets.US_ASCII).trim(); + if (!sizeLine.startsWith(METADATA_SIZE_PREFIX)) { + throw new ObjectStoreIOException("Missing original size metadata line: " + sizeLine); + } + + long originalSize; + try { + originalSize = Long.parseLong(sizeLine.substring(METADATA_SIZE_PREFIX.length()).trim()); + } catch (NumberFormatException e) { + throw new ObjectStoreIOException("Corrupt original size metadata: " + sizeLine, e); + } + + int contentOffset = secondNewline + 1; + int compressedLength = slotBytes.length - contentOffset; + byte[] compressedContent = new byte[compressedLength]; + System.arraycopy(slotBytes, contentOffset, compressedContent, 0, compressedLength); + + BlobSlot blobSlot = new BlobSlot(expectedContentStart, crc32c, originalSize, compressedContent); + return blobSlot.decompress(); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkId.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkId.java new file mode 100644 index 00000000000..79cda13e6af --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkId.java @@ -0,0 +1,227 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.security.SecureRandom; +import java.util.Objects; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.ChunkMarker; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; + +import com.google.common.base.Preconditions; +import com.google.common.io.BaseEncoding; + +public class ChunkId implements BlobId { + private static final String CHUNK_MARKER = "_chunk"; + private static final SecureRandom RANDOM = new SecureRandom(); + private static final int RANDOM_BYTES_COUNT = 16; + private static final int MIN_RANDOM_LENGTH = 16; + + public static ChunkId ofChunk(int family, long generation) { + Preconditions.checkArgument(family > 0, "'family' must be strictly positive"); + Preconditions.checkArgument(generation >= 0, "'generation' must not be negative"); + + byte[] randomBytes = new byte[RANDOM_BYTES_COUNT]; + RANDOM.nextBytes(randomBytes); + String randomPart = BaseEncoding.base64Url().omitPadding().encode(randomBytes); + + return new ChunkId(family, generation, randomPart, 0L, 0L); + } + + public static ChunkId ofChunk(GenerationAwareBlobId.Configuration configuration, long generation) { + Preconditions.checkNotNull(configuration, "'configuration' must not be null"); + return ofChunk(configuration.getFamily(), generation); + } + + public static ChunkId slotRef(ChunkId chunkId, long offset, long limit) { + Preconditions.checkNotNull(chunkId, "'chunkId' must not be null"); + Preconditions.checkArgument(offset >= 0, "'offset' must not be negative"); + Preconditions.checkArgument(limit >= 0, "'limit' must not be negative"); + + return new ChunkId(chunkId.family(), chunkId.generation(), chunkId.randomPart(), offset, limit); + } + + public static ChunkId slotRef(String chunkIdStr, long offset, long limit) { + Preconditions.checkNotNull(chunkIdStr, "'chunkIdStr' must not be null"); + Preconditions.checkArgument(offset >= 0, "'offset' must not be negative"); + Preconditions.checkArgument(limit >= 0, "'limit' must not be negative"); + + ChunkId base = parseChunkOrSlotRef(chunkIdStr); + return new ChunkId(base.family(), base.generation(), base.randomPart(), offset, limit); + } + + public static boolean isChunkRef(BlobId blobId) { + if (blobId == null) { + return false; + } + return isChunkRef(blobId.asString()); + } + + public static boolean isChunkRef(String id) { + return ChunkMarker.looksLikeChunkId(id); + } + + public static ChunkId parse(String id) { + Preconditions.checkNotNull(id, "'id' must not be null"); + int firstTilde = id.indexOf('~'); + Preconditions.checkArgument(firstTilde != -1, "Missing '~' in chunk-slot ref: " + id); + int secondTilde = id.indexOf('~', firstTilde + 1); + Preconditions.checkArgument(secondTilde != -1, "Missing second '~' in chunk-slot ref: " + id); + + String basePart = id.substring(0, firstTilde); + long offset; + long limit; + try { + offset = Long.parseLong(id.substring(firstTilde + 1, secondTilde)); + limit = Long.parseLong(id.substring(secondTilde + 1)); + } catch (NumberFormatException e) { + throw new IllegalArgumentException("Invalid offset or limit in chunk ref: " + id, e); + } + Preconditions.checkArgument(offset >= 0, "'offset' must not be negative"); + Preconditions.checkArgument(limit >= 0, "'limit' must not be negative"); + + return parseBase(basePart, offset, limit); + } + + public static ChunkId parseChunk(String id) { + Preconditions.checkNotNull(id, "'id' must not be null"); + return parseBase(id, 0L, 0L); + } + + public static ChunkId parseChunkOrSlotRef(String id) { + int firstTilde = id.indexOf('~'); + if (firstTilde == -1) { + return parseBase(id, 0L, 0L); + } + return parse(id); + } + + private static ChunkId parseBase(String basePart, long offset, long limit) { + int firstUnder = basePart.indexOf('_'); + Preconditions.checkArgument(firstUnder > 0, "Missing family delimiter '_' in chunk id: " + basePart); + int chunkIndex = basePart.indexOf(CHUNK_MARKER, firstUnder + 1); + Preconditions.checkArgument(chunkIndex > firstUnder, "Missing '_chunk' marker in chunk id: " + basePart); + + int family; + long generation; + try { + family = Integer.parseInt(basePart.substring(0, firstUnder)); + generation = Long.parseLong(basePart.substring(firstUnder + 1, chunkIndex)); + } catch (NumberFormatException e) { + throw new IllegalArgumentException("Invalid family or generation in chunk id: " + basePart, e); + } + + Preconditions.checkArgument(family > 0, "'family' must be strictly positive"); + Preconditions.checkArgument(generation >= 0, "'generation' must not be negative"); + + String randomPart = basePart.substring(chunkIndex + CHUNK_MARKER.length()); + Preconditions.checkArgument(randomPart.length() >= MIN_RANDOM_LENGTH, + "Random part of chunk id is too short: " + randomPart); + + return new ChunkId(family, generation, randomPart, offset, limit); + } + + private final int family; + private final long generation; + private final String randomPart; + private final long offset; + private final long limit; + + public ChunkId(int family, long generation, String randomPart, long offset, long limit) { + this.family = family; + this.generation = generation; + this.randomPart = randomPart; + this.offset = offset; + this.limit = limit; + } + + public int family() { + return family; + } + + public long generation() { + return generation; + } + + public String randomPart() { + return randomPart; + } + + public long offset() { + return offset; + } + + public long limit() { + return limit; + } + + public boolean isSlotRef() { + return offset > 0 || limit > 0; + } + + public String chunkId() { + return family + "_" + generation + "_chunk" + randomPart; + } + + public BlobId chunkBlobId() { + return new PlainBlobId(chunkId()); + } + + public BlobId asBlobId() { + return this; + } + + @Override + public String asString() { + return chunkId() + "~" + offset + "~" + limit; + } + + @Override + public BlobId withSuffix(String suffix) { + throw new UnsupportedOperationException("ChunkId does not support withSuffix"); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof ChunkId other) { + return family == other.family + && generation == other.generation + && offset == other.offset + && limit == other.limit + && Objects.equals(randomPart, other.randomPart); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(family, generation, randomPart, offset, limit); + } + + @Override + public String toString() { + return asString(); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java new file mode 100644 index 00000000000..d22bccc0c06 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java @@ -0,0 +1,246 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.io.ByteArrayInputStream; +import java.util.Collection; +import java.util.List; +import java.util.Optional; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BlobStoreDAO.Blob; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.ObjectNotFoundException; +import org.apache.james.blob.api.ObjectStoreIOException; +import org.reactivestreams.Publisher; + +import com.google.common.base.Preconditions; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.core.scheduler.Schedulers; + +public class ChunkedBlobStoreDAO implements BlobStoreDAO { + private final BlobStoreDAO plainChain; + private final BlobStoreDAO rawStore; + private final BlobIdRepairer blobIdRepairer; + + public ChunkedBlobStoreDAO(BlobStoreDAO plainChain, BlobStoreDAO rawStore) { + this(plainChain, rawStore, Optional.empty()); + } + + public ChunkedBlobStoreDAO(BlobStoreDAO plainChain, BlobStoreDAO rawStore, Optional blobIdRepairer) { + this.plainChain = Preconditions.checkNotNull(plainChain, "'plainChain' must not be null"); + this.rawStore = Preconditions.checkNotNull(rawStore, "'rawStore' must not be null"); + this.blobIdRepairer = blobIdRepairer.orElse(BlobIdRepairer.NOOP); + } + + @Override + public InputStreamBlob read(BucketName bucketName, BlobId blobId) throws ObjectStoreIOException, ObjectNotFoundException { + if (ChunkId.isChunkRef(blobId)) { + ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); + if (chunkId.isSlotRef()) { + BytesBlob bytes = Mono.from(readChunkSlotWithRepair(bucketName, chunkId, blobId)).block(); + return InputStreamBlob.of(new ByteArrayInputStream(bytes.payload()), bytes.metadata()); + } + return rawStore.read(bucketName, chunkId.chunkBlobId()); + } + return plainChain.read(bucketName, blobId); + } + + @Override + public Publisher readReactive(BucketName bucketName, BlobId blobId) { + if (ChunkId.isChunkRef(blobId)) { + ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); + if (chunkId.isSlotRef()) { + return Mono.from(readChunkSlotWithRepair(bucketName, chunkId, blobId)) + .map(bytesBlob -> InputStreamBlob.of(new ByteArrayInputStream(bytesBlob.payload()), bytesBlob.metadata())); + } + return rawStore.readReactive(bucketName, chunkId.chunkBlobId()); + } + return plainChain.readReactive(bucketName, blobId); + } + + @Override + public Publisher readBytes(BucketName bucketName, BlobId blobId) { + if (ChunkId.isChunkRef(blobId)) { + ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); + if (chunkId.isSlotRef()) { + return readChunkSlotWithRepair(bucketName, chunkId, blobId); + } + return rawStore.readBytes(bucketName, chunkId.chunkBlobId()); + } + return plainChain.readBytes(bucketName, blobId); + } + + private Mono readChunkSlotWithRepair(BucketName bucketName, ChunkId slotRef, BlobId originalBlobId) { + return readChunkSlot(bucketName, slotRef) + .onErrorResume(ObjectNotFoundException.class, notFound -> + blobIdRepairer.repair(bucketName, originalBlobId) + .flatMap(repairedBlobId -> { + if (repairedBlobId.equals(originalBlobId)) { + return Mono.error(notFound); + } + if (ChunkId.isChunkRef(repairedBlobId)) { + ChunkId repairedChunkId = ChunkId.parseChunkOrSlotRef(repairedBlobId.asString()); + if (repairedChunkId.isSlotRef()) { + return readChunkSlot(bucketName, repairedChunkId); + } + return Mono.from(rawStore.readBytes(bucketName, repairedChunkId.chunkBlobId())); + } + return Mono.from(plainChain.readBytes(bucketName, repairedBlobId)); + }) + .switchIfEmpty(Mono.error(notFound))); + } + + private Mono readChunkSlot(BucketName bucketName, ChunkId slotRef) { + if (slotRef.limit() > 0) { + long offset = slotRef.offset(); + long limit = slotRef.limit(); + BlobId chunkObjectBlobId = slotRef.chunkBlobId(); + return rawStore.readRange(bucketName, chunkObjectBlobId, offset, offset + limit - 1) + .publishOn(Schedulers.parallel()) + .flatMap(rangeSlice -> { + if (rangeSlice.data().length == 0 || offset >= rangeSlice.totalObjectSize()) { + return Mono.error(new ObjectNotFoundException("Slot not found at offset " + offset + " in chunk " + chunkObjectBlobId.asString())); + } + try { + byte[] decompressed = ChunkFormat.parseSlotBytes(rangeSlice.data(), offset); + return Mono.just(BytesBlob.of(decompressed)); + } catch (ObjectStoreIOException e) { + return Mono.error(e); + } + }); + } else { + BlobId chunkObjectBlobId = slotRef.chunkBlobId(); + return rawStore.readRange(bucketName, chunkObjectBlobId, -65536, -1) + .flatMap(tailSlice -> { + try { + ChunkFooter footer = ChunkFormat.readFooter(tailSlice.data(), tailSlice.totalObjectSize()); + if (footer.slotCount() == 0) { + return Mono.just(BytesBlob.of(new byte[0])); + } + long start = slotRef.offset() > 0 ? slotRef.offset() : footer.slotStarts().get(0); + int slotIndex = footer.slotStarts().indexOf(start); + if (slotIndex == -1) { + return Mono.error(new ObjectNotFoundException("Slot not found at offset " + start + " in chunk " + chunkObjectBlobId.asString())); + } + long length = footer.slotLength(slotIndex); + return rawStore.readRange(bucketName, chunkObjectBlobId, start, start + length - 1) + .publishOn(Schedulers.parallel()) + .flatMap(slice -> { + try { + byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), start); + return Mono.just(BytesBlob.of(decompressed)); + } catch (ObjectStoreIOException e) { + return Mono.error(e); + } + }); + } catch (ObjectStoreIOException e) { + return Mono.error(e); + } + }); + } + } + + @Override + public Publisher save(BucketName bucketName, BlobId blobId, Blob blob) { + if (ChunkId.isChunkRef(blobId)) { + ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); + if (chunkId.isSlotRef()) { + return Mono.error(new UnsupportedOperationException( + "Individual chunk slot refs cannot be saved directly; chunks are immutable. Ref: " + blobId.asString())); + } + return rawStore.save(bucketName, chunkId.chunkBlobId(), blob); + } + return plainChain.save(bucketName, blobId, blob); + } + + @Override + public Publisher delete(BucketName bucketName, BlobId blobId) { + if (ChunkId.isChunkRef(blobId)) { + ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); + if (chunkId.isSlotRef()) { + return Mono.error(new UnsupportedOperationException( + "Individual chunk slot refs cannot be deleted directly; slots are reclaimed via compaction GC. Ref: " + blobId.asString())); + } + return rawStore.delete(bucketName, chunkId.chunkBlobId()); + } + return plainChain.delete(bucketName, blobId); + } + + @Override + public Publisher delete(BucketName bucketName, Collection blobIds) { + return Mono.defer(() -> { + for (BlobId blobId : blobIds) { + if (ChunkId.isChunkRef(blobId)) { + ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); + if (chunkId.isSlotRef()) { + return Mono.error(new UnsupportedOperationException( + "Individual chunk slot refs cannot be deleted directly; slots are reclaimed via compaction GC. Ref: " + blobId.asString())); + } + } + } + List chunkDeletions = blobIds.stream() + .filter(ChunkId::isChunkRef) + .map(id -> ChunkId.parseChunkOrSlotRef(id.asString()).chunkBlobId()) + .toList(); + List plainDeletions = blobIds.stream() + .filter(id -> !ChunkId.isChunkRef(id)) + .toList(); + + return Flux.mergeDelayError( + 1, + chunkDeletions.isEmpty() ? Mono.empty() : rawStore.delete(bucketName, chunkDeletions), + plainDeletions.isEmpty() ? Mono.empty() : plainChain.delete(bucketName, plainDeletions) + ).then(); + }); + } + + @Override + public Publisher deleteBucket(BucketName bucketName) { + return rawStore.deleteBucket(bucketName); + } + + @Override + public Publisher listBuckets() { + return rawStore.listBuckets(); + } + + @Override + public Publisher listBlobs(BucketName bucketName) { + return rawStore.listBlobs(bucketName); + } + + @Override + public Publisher listBlobs(BucketName bucketName, String prefix) { + return rawStore.listBlobs(bucketName, prefix); + } + + @Override + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + if (ChunkId.isChunkRef(blobId)) { + ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); + return rawStore.readRange(bucketName, chunkId.chunkBlobId(), start, end); + } + return rawStore.readRange(bucketName, blobId, start, end); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java new file mode 100644 index 00000000000..972b7a3f256 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java @@ -0,0 +1,149 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.util.Objects; + +import com.google.common.base.MoreObjects; +import com.google.common.base.Preconditions; + +public class CompactionConfiguration { + public static final long DEFAULT_CHUNK_TARGET_SIZE = 100 * 1024 * 1024L; // 100MB + public static final long DEFAULT_MAX_PACKABLE_SIZE = 1024 * 1024L; // 1MB + public static final double DEFAULT_PURGE_DEAD_RATIO = 0.1; // 10% + public static final double DEFAULT_MERGE_DEAD_RATIO = 0.5; // 50% + public static final double DEFAULT_GAIN_THRESHOLD = 0.1; // 10% + + public static final CompactionConfiguration DEFAULT = builder().build(); + + public static class Builder { + private long chunkTargetSize = DEFAULT_CHUNK_TARGET_SIZE; + private long maxPackableSize = DEFAULT_MAX_PACKABLE_SIZE; + private double purgeDeadRatio = DEFAULT_PURGE_DEAD_RATIO; + private double mergeDeadRatio = DEFAULT_MERGE_DEAD_RATIO; + private double gainThreshold = DEFAULT_GAIN_THRESHOLD; + + public Builder chunkTargetSize(long chunkTargetSize) { + Preconditions.checkArgument(chunkTargetSize > 0, "'chunkTargetSize' must be strictly positive"); + this.chunkTargetSize = chunkTargetSize; + return this; + } + + public Builder maxPackableSize(long maxPackableSize) { + Preconditions.checkArgument(maxPackableSize > 0, "'maxPackableSize' must be strictly positive"); + this.maxPackableSize = maxPackableSize; + return this; + } + + public Builder purgeDeadRatio(double purgeDeadRatio) { + Preconditions.checkArgument(purgeDeadRatio >= 0.0 && purgeDeadRatio <= 1.0, + "'purgeDeadRatio' must be between 0.0 and 1.0"); + this.purgeDeadRatio = purgeDeadRatio; + return this; + } + + public Builder mergeDeadRatio(double mergeDeadRatio) { + Preconditions.checkArgument(mergeDeadRatio >= 0.0 && mergeDeadRatio <= 1.0, + "'mergeDeadRatio' must be between 0.0 and 1.0"); + this.mergeDeadRatio = mergeDeadRatio; + return this; + } + + public Builder gainThreshold(double gainThreshold) { + Preconditions.checkArgument(gainThreshold >= 0.0 && gainThreshold <= 1.0, + "'gainThreshold' must be between 0.0 and 1.0"); + this.gainThreshold = gainThreshold; + return this; + } + + public CompactionConfiguration build() { + return new CompactionConfiguration(chunkTargetSize, maxPackableSize, purgeDeadRatio, mergeDeadRatio, gainThreshold); + } + } + + public static Builder builder() { + return new Builder(); + } + + private final long chunkTargetSize; + private final long maxPackableSize; + private final double purgeDeadRatio; + private final double mergeDeadRatio; + private final double gainThreshold; + + public CompactionConfiguration(long chunkTargetSize, long maxPackableSize, double purgeDeadRatio, double mergeDeadRatio, double gainThreshold) { + this.chunkTargetSize = chunkTargetSize; + this.maxPackableSize = maxPackableSize; + this.purgeDeadRatio = purgeDeadRatio; + this.mergeDeadRatio = mergeDeadRatio; + this.gainThreshold = gainThreshold; + } + + public long chunkTargetSize() { + return chunkTargetSize; + } + + public long maxPackableSize() { + return maxPackableSize; + } + + public double purgeDeadRatio() { + return purgeDeadRatio; + } + + public double mergeDeadRatio() { + return mergeDeadRatio; + } + + public double gainThreshold() { + return gainThreshold; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof CompactionConfiguration that) { + return chunkTargetSize == that.chunkTargetSize + && maxPackableSize == that.maxPackableSize + && Double.compare(that.purgeDeadRatio, purgeDeadRatio) == 0 + && Double.compare(that.mergeDeadRatio, mergeDeadRatio) == 0 + && Double.compare(that.gainThreshold, gainThreshold) == 0; + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(chunkTargetSize, maxPackableSize, purgeDeadRatio, mergeDeadRatio, gainThreshold); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this) + .add("chunkTargetSize", chunkTargetSize) + .add("maxPackableSize", maxPackableSize) + .add("purgeDeadRatio", purgeDeadRatio) + .add("mergeDeadRatio", mergeDeadRatio) + .add("gainThreshold", gainThreshold) + .toString(); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionRequest.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionRequest.java new file mode 100644 index 00000000000..6237b6de421 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionRequest.java @@ -0,0 +1,130 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.util.Objects; +import java.util.Optional; + +import org.apache.james.blob.api.BucketName; + +import com.google.common.base.MoreObjects; +import com.google.common.base.Preconditions; + +public class CompactionRequest { + public static class Builder { + private BucketName bucketName = BucketName.DEFAULT; + private Long generation; + private Optional family = Optional.empty(); + private CompactionConfiguration configuration = CompactionConfiguration.DEFAULT; + + public Builder bucketName(BucketName bucketName) { + this.bucketName = Preconditions.checkNotNull(bucketName, "'bucketName' must not be null"); + return this; + } + + public Builder generation(long generation) { + Preconditions.checkArgument(generation >= 0, "'generation' must not be negative"); + this.generation = generation; + return this; + } + + public Builder family(int family) { + Preconditions.checkArgument(family > 0, "'family' must be strictly positive"); + this.family = Optional.of(family); + return this; + } + + public Builder family(Optional family) { + this.family = Preconditions.checkNotNull(family, "'family' must not be null"); + return this; + } + + public Builder configuration(CompactionConfiguration configuration) { + this.configuration = Preconditions.checkNotNull(configuration, "'configuration' must not be null"); + return this; + } + + public CompactionRequest build() { + Preconditions.checkState(generation != null, "'generation' is required"); + return new CompactionRequest(bucketName, generation, family, configuration); + } + } + + public static Builder builder() { + return new Builder(); + } + + private final BucketName bucketName; + private final long generation; + private final Optional family; + private final CompactionConfiguration configuration; + + public CompactionRequest(BucketName bucketName, long generation, Optional family, CompactionConfiguration configuration) { + this.bucketName = Preconditions.checkNotNull(bucketName, "'bucketName' must not be null"); + this.generation = generation; + this.family = Preconditions.checkNotNull(family, "'family' must not be null"); + this.configuration = Preconditions.checkNotNull(configuration, "'configuration' must not be null"); + } + + public BucketName bucketName() { + return bucketName; + } + + public long generation() { + return generation; + } + + public Optional family() { + return family; + } + + public CompactionConfiguration configuration() { + return configuration; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof CompactionRequest that) { + return generation == that.generation + && Objects.equals(bucketName, that.bucketName) + && Objects.equals(family, that.family) + && Objects.equals(configuration, that.configuration); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(bucketName, generation, family, configuration); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this) + .add("bucketName", bucketName) + .add("generation", generation) + .add("family", family) + .add("configuration", configuration) + .toString(); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionResult.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionResult.java new file mode 100644 index 00000000000..ade1b638e0d --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionResult.java @@ -0,0 +1,161 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.util.Objects; + +import com.google.common.base.MoreObjects; + +public class CompactionResult { + public static final CompactionResult NONE = builder().build(); + + public static class Builder { + private long packedBlobs; + private long packedBytes; + private long chunksWritten; + private long deadPurged; + private long mergedChunks; + private long freedBytes; + + public Builder packedBlobs(long packedBlobs) { + this.packedBlobs = packedBlobs; + return this; + } + + public Builder packedBytes(long packedBytes) { + this.packedBytes = packedBytes; + return this; + } + + public Builder chunksWritten(long chunksWritten) { + this.chunksWritten = chunksWritten; + return this; + } + + public Builder deadPurged(long deadPurged) { + this.deadPurged = deadPurged; + return this; + } + + public Builder mergedChunks(long mergedChunks) { + this.mergedChunks = mergedChunks; + return this; + } + + public Builder freedBytes(long freedBytes) { + this.freedBytes = freedBytes; + return this; + } + + public CompactionResult build() { + return new CompactionResult(packedBlobs, packedBytes, chunksWritten, deadPurged, mergedChunks, freedBytes); + } + } + + public static Builder builder() { + return new Builder(); + } + + private final long packedBlobs; + private final long packedBytes; + private final long chunksWritten; + private final long deadPurged; + private final long mergedChunks; + private final long freedBytes; + + public CompactionResult(long packedBlobs, long packedBytes, long chunksWritten, long deadPurged, long mergedChunks, long freedBytes) { + this.packedBlobs = packedBlobs; + this.packedBytes = packedBytes; + this.chunksWritten = chunksWritten; + this.deadPurged = deadPurged; + this.mergedChunks = mergedChunks; + this.freedBytes = freedBytes; + } + + public long packedBlobs() { + return packedBlobs; + } + + public long packedBytes() { + return packedBytes; + } + + public long chunksWritten() { + return chunksWritten; + } + + public long deadPurged() { + return deadPurged; + } + + public long mergedChunks() { + return mergedChunks; + } + + public long freedBytes() { + return freedBytes; + } + + public CompactionResult combine(CompactionResult other) { + if (other == null) { + return this; + } + return new CompactionResult( + this.packedBlobs + other.packedBlobs, + this.packedBytes + other.packedBytes, + this.chunksWritten + other.chunksWritten, + this.deadPurged + other.deadPurged, + this.mergedChunks + other.mergedChunks, + this.freedBytes + other.freedBytes); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof CompactionResult that) { + return packedBlobs == that.packedBlobs + && packedBytes == that.packedBytes + && chunksWritten == that.chunksWritten + && deadPurged == that.deadPurged + && mergedChunks == that.mergedChunks + && freedBytes == that.freedBytes; + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(packedBlobs, packedBytes, chunksWritten, deadPurged, mergedChunks, freedBytes); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this) + .add("packedBlobs", packedBlobs) + .add("packedBytes", packedBytes) + .add("chunksWritten", chunksWritten) + .add("deadPurged", deadPurged) + .add("mergedChunks", mergedChunks) + .add("freedBytes", freedBytes) + .toString(); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java new file mode 100644 index 00000000000..9dcc57e1ee2 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -0,0 +1,538 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicLong; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.ObjectNotFoundException; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.blob.compaction.BlobReferenceMappingSource.BlobIdMessageIdMapping; +import org.apache.james.blob.compaction.ChunkFormat.BlobSlotContent; +import org.apache.james.blob.compaction.ChunkFormat.ChunkWriteResult; +import org.apache.james.blob.compaction.ChunkFormat.SlotRange; +import org.apache.james.blob.memory.MemoryBlobStoreDAO; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.reactivestreams.Publisher; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +class BlobCompactionAlgorithmTest { + private static final BucketName TEST_BUCKET = BucketName.of("compaction-test-bucket"); + private static final long TARGET_GENERATION = 2L; + private static final int FAMILY = 1; + + static class RecordingBlobIdUpdater implements BlobIdUpdater { + record Replacement(BlobId oldId, BlobId newId, Collection messageIds) {} + + private final List replacements = new ArrayList<>(); + + @Override + public synchronized Mono replaceReferences(BlobId oldId, BlobId newId, Collection messageIds) { + replacements.add(new Replacement(oldId, newId, new ArrayList<>(messageIds))); + return Mono.empty(); + } + + public synchronized List getReplacements() { + return new ArrayList<>(replacements); + } + } + + static class TestMappingSource implements BlobReferenceMappingSource { + private final Map> mappings = new ConcurrentHashMap<>(); + + public void add(BlobId blobId, String messageId) { + mappings.computeIfAbsent(blobId, k -> ConcurrentHashMap.newKeySet()).add(messageId); + } + + public void remove(BlobId blobId) { + mappings.remove(blobId); + } + + @Override + public Publisher listBlobIdMessageIdMappings() { + List list = new ArrayList<>(); + mappings.forEach((blobId, msgIds) -> + msgIds.forEach(msgId -> list.add(new BlobIdMessageIdMapping(blobId, msgId)))); + return Flux.fromIterable(list); + } + } + + private MemoryBlobStoreDAO rawStore; + private ChunkedBlobStoreDAO chunkedBlobStoreDAO; + private TestMappingSource mappingSource; + private RecordingBlobIdUpdater recordingUpdater; + private BlobCompactionAlgorithm testee; + + @BeforeEach + void setUp() { + rawStore = new MemoryBlobStoreDAO(); + chunkedBlobStoreDAO = new ChunkedBlobStoreDAO(rawStore, rawStore); + mappingSource = new TestMappingSource(); + recordingUpdater = new RecordingBlobIdUpdater(); + testee = new BlobCompactionAlgorithm(rawStore, rawStore, mappingSource, recordingUpdater); + } + + @Test + void initialCompactionShouldPackSmallBlobsAndSkipLargeBlobs() { + BlobId small1 = new PlainBlobId("1_2_small1"); + BlobId small2 = new PlainBlobId("1_2_small2"); + BlobId large = new PlainBlobId("1_2_large"); + + byte[] payload1 = "Small payload 1".getBytes(StandardCharsets.UTF_8); + byte[] payload2 = "Small payload 2".getBytes(StandardCharsets.UTF_8); + byte[] largePayload = new byte[2 * 1024 * 1024]; // 2MB + Arrays.fill(largePayload, (byte) 'L'); + + Mono.from(rawStore.save(TEST_BUCKET, small1, BlobStoreDAO.BytesBlob.of(payload1))).block(); + Mono.from(rawStore.save(TEST_BUCKET, small2, BlobStoreDAO.BytesBlob.of(payload2))).block(); + Mono.from(rawStore.save(TEST_BUCKET, large, BlobStoreDAO.BytesBlob.of(largePayload))).block(); + + mappingSource.add(small1, "msg-1"); + mappingSource.add(small2, "msg-2"); + mappingSource.add(large, "msg-3"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .maxPackableSize(1024 * 1024L) // 1MB + .build()) + .build(); + + CompactionResult result = testee.initialCompact(request).block(); + + assertThat(result.packedBlobs()).isEqualTo(2); + assertThat(result.chunksWritten()).isEqualTo(1); + + // Standalone small blobs should be deleted + assertThat(Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block()) + .doesNotContain(small1, small2) + .contains(large); + + // Replacements recorded + assertThat(recordingUpdater.getReplacements()).hasSize(2); + BlobId newSlotRef1 = recordingUpdater.getReplacements().get(0).newId(); + BlobId newSlotRef2 = recordingUpdater.getReplacements().get(1).newId(); + + assertThat(ChunkId.isChunkRef(newSlotRef1)).isTrue(); + assertThat(ChunkId.isChunkRef(newSlotRef2)).isTrue(); + + // Ranged read of new slot refs returns exact content + byte[] read1 = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlotRef1)).block().payload(); + byte[] read2 = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlotRef2)).block().payload(); + assertThat(read1).isEqualTo(payload1); + assertThat(read2).isEqualTo(payload2); + } + + @Test + void initialCompactionPreservesDeduplication() { + BlobId sharedBlobId = new PlainBlobId("1_2_shared"); + byte[] sharedPayload = "Shared email attachment body".getBytes(StandardCharsets.UTF_8); + + Mono.from(rawStore.save(TEST_BUCKET, sharedBlobId, BlobStoreDAO.BytesBlob.of(sharedPayload))).block(); + + // Two messages reference the SAME blob + mappingSource.add(sharedBlobId, "msg-A"); + mappingSource.add(sharedBlobId, "msg-B"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + CompactionResult result = testee.initialCompact(request).block(); + + assertThat(result.packedBlobs()).isEqualTo(1); + assertThat(result.chunksWritten()).isEqualTo(1); + + // One replacement covering both messages + assertThat(recordingUpdater.getReplacements()).hasSize(1); + RecordingBlobIdUpdater.Replacement rep = recordingUpdater.getReplacements().get(0); + assertThat(rep.oldId()).isEqualTo(sharedBlobId); + assertThat(rep.messageIds()).containsExactlyInAnyOrder("msg-A", "msg-B"); + + // Content readable + byte[] read = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, rep.newId())).block().payload(); + assertThat(read).isEqualTo(sharedPayload); + } + + @Test + void gcCompactShouldPurgeDeadSlotsWhenThresholdExceeded() throws Exception { + ChunkId chunkId = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + + List slots = new ArrayList<>(); + for (int i = 0; i < 10; i++) { + slots.add(BlobSlotContent.of(("Slot content " + i).getBytes(StandardCharsets.UTF_8))); + } + + ChunkWriteResult writeResult = ChunkFormat.writeChunk(slots); + Mono.from(rawStore.save(TEST_BUCKET, chunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))).block(); + + // Only 8 slots are live, 2 slots (slots 8 and 9) are dead (20% dead > 10% threshold) + for (int i = 0; i < 8; i++) { + ChunkId slotRef = ChunkId.slotRef(chunkId, writeResult.slotRanges().get(i).offset(), writeResult.slotRanges().get(i).limit()); + mappingSource.add(slotRef, "msg-" + i); + } + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .purgeDeadRatio(0.1) + .gainThreshold(0.1) + .build()) + .build(); + + CompactionResult result = testee.gcCompact(request).block(); + + assertThat(result.deadPurged()).isEqualTo(2); + + // Old chunk deleted + List remainingBlobs = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(remainingBlobs).doesNotContain(chunkId.chunkBlobId()); + assertThat(remainingBlobs).hasSize(1); // the new purged chunk + + // 8 surviving slots updated + assertThat(recordingUpdater.getReplacements()).hasSize(8); + + // Surviving slots readable + for (int i = 0; i < 8; i++) { + BlobId newSlotRef = recordingUpdater.getReplacements().get(i).newId(); + byte[] read = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlotRef)).block().payload(); + assertThat(read).isEqualTo(("Slot content " + i).getBytes(StandardCharsets.UTF_8)); + } + } + + @Test + void gcCompactShouldRespectGainThreshold() throws Exception { + ChunkId chunkId = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + + List slots = new ArrayList<>(); + for (int i = 0; i < 10; i++) { + slots.add(BlobSlotContent.of(("Slot content " + i).getBytes(StandardCharsets.UTF_8))); + } + + ChunkWriteResult writeResult = ChunkFormat.writeChunk(slots); + Mono.from(rawStore.save(TEST_BUCKET, chunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))).block(); + + // 9 slots are live, 1 slot is dead (10% dead). Gain threshold is 20%. + for (int i = 0; i < 9; i++) { + ChunkId slotRef = ChunkId.slotRef(chunkId, writeResult.slotRanges().get(i).offset(), writeResult.slotRanges().get(i).limit()); + mappingSource.add(slotRef, "msg-" + i); + } + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .purgeDeadRatio(0.1) + .gainThreshold(0.20) // 20% > 10% dead + .build()) + .build(); + + CompactionResult result = testee.gcCompact(request).block(); + + assertThat(result.deadPurged()).isEqualTo(0); + // Old chunk is untouched + assertThat(Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block()) + .contains(chunkId.chunkBlobId()); + } + + @Test + void gcCompactShouldMergeTwoSmallChunks() throws Exception { + ChunkId chunk1 = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + ChunkId chunk2 = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + + byte[] payload1 = "Chunk 1 payload".getBytes(StandardCharsets.UTF_8); + byte[] payload2 = "Chunk 2 payload".getBytes(StandardCharsets.UTF_8); + + ChunkWriteResult w1 = ChunkFormat.writeChunk(List.of(BlobSlotContent.of(payload1))); + ChunkWriteResult w2 = ChunkFormat.writeChunk(List.of(BlobSlotContent.of(payload2))); + + Mono.from(rawStore.save(TEST_BUCKET, chunk1.chunkBlobId(), BlobStoreDAO.BytesBlob.of(w1.chunkBytes()))).block(); + Mono.from(rawStore.save(TEST_BUCKET, chunk2.chunkBlobId(), BlobStoreDAO.BytesBlob.of(w2.chunkBytes()))).block(); + + ChunkId slot1 = ChunkId.slotRef(chunk1, w1.slotRanges().get(0).offset(), w1.slotRanges().get(0).limit()); + ChunkId slot2 = ChunkId.slotRef(chunk2, w2.slotRanges().get(0).offset(), w2.slotRanges().get(0).limit()); + + mappingSource.add(slot1, "msg-1"); + mappingSource.add(slot2, "msg-2"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .chunkTargetSize(100_000L) + .mergeDeadRatio(0.5) // chunks < 50,000 bytes qualify for merge + .build()) + .build(); + + CompactionResult result = testee.gcCompact(request).block(); + + assertThat(result.mergedChunks()).isEqualTo(2); + + // Old chunks deleted + List blobs = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(blobs).doesNotContain(chunk1.chunkBlobId(), chunk2.chunkBlobId()); + assertThat(blobs).hasSize(1); // merged chunk + + // Both slots updated and readable + assertThat(recordingUpdater.getReplacements()).hasSize(2); + BlobId newSlot1 = recordingUpdater.getReplacements().get(0).newId(); + BlobId newSlot2 = recordingUpdater.getReplacements().get(1).newId(); + + assertThat(Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlot1)).block().payload()).isEqualTo(payload1); + assertThat(Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlot2)).block().payload()).isEqualTo(payload2); + } + + @Test + void gcCompactShouldKeepSingleSmallChunkAlone() throws Exception { + ChunkId chunk1 = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + byte[] payload1 = "Single chunk payload".getBytes(StandardCharsets.UTF_8); + + ChunkWriteResult w1 = ChunkFormat.writeChunk(List.of(BlobSlotContent.of(payload1))); + Mono.from(rawStore.save(TEST_BUCKET, chunk1.chunkBlobId(), BlobStoreDAO.BytesBlob.of(w1.chunkBytes()))).block(); + + ChunkId slot1 = ChunkId.slotRef(chunk1, w1.slotRanges().get(0).offset(), w1.slotRanges().get(0).limit()); + mappingSource.add(slot1, "msg-1"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .chunkTargetSize(100_000L) + .mergeDeadRatio(0.5) + .build()) + .build(); + + CompactionResult result = testee.gcCompact(request).block(); + + assertThat(result.mergedChunks()).isEqualTo(0); + assertThat(Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block()) + .contains(chunk1.chunkBlobId()); + } + + @Test + void gcCompactShouldDeleteOrphanChunkForCrashRecovery() throws Exception { + ChunkId orphanChunk = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + byte[] payload = "Orphan chunk content with no references".getBytes(StandardCharsets.UTF_8); + + ChunkWriteResult w = ChunkFormat.writeChunk(List.of(BlobSlotContent.of(payload))); + Mono.from(rawStore.save(TEST_BUCKET, orphanChunk.chunkBlobId(), BlobStoreDAO.BytesBlob.of(w.chunkBytes()))).block(); + + // Notice: NO references in mappingSource (100% orphan) + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + CompactionResult result = testee.gcCompact(request).block(); + + assertThat(result.deadPurged()).isEqualTo(1); + assertThat(Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block()) + .doesNotContain(orphanChunk.chunkBlobId()); + } + + @Test + void gcCompactShouldBeIdempotentAndTerminateWithEmptyResult() { + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + CompactionResult result = testee.gcCompact(request).block(); + assertThat(result).isEqualTo(CompactionResult.NONE); + } + + @Test + void initialCompactShouldNotDeleteOriginalBlobWhenReferenceUpdateFails() { + BlobId failingBlob = new PlainBlobId("1_2_failing"); + BlobId succeedingBlob = new PlainBlobId("1_2_succeeding"); + + byte[] payloadFailing = "Payload failing".getBytes(StandardCharsets.UTF_8); + byte[] payloadSucceeding = "Payload succeeding".getBytes(StandardCharsets.UTF_8); + + Mono.from(rawStore.save(TEST_BUCKET, failingBlob, BlobStoreDAO.BytesBlob.of(payloadFailing))).block(); + Mono.from(rawStore.save(TEST_BUCKET, succeedingBlob, BlobStoreDAO.BytesBlob.of(payloadSucceeding))).block(); + + mappingSource.add(failingBlob, "msg-fail"); + mappingSource.add(succeedingBlob, "msg-success"); + + BlobIdUpdater partiallyFailingUpdater = (oldId, newId, messageIds) -> { + if (oldId.equals(failingBlob)) { + return Mono.error(new RuntimeException("Cassandra write timeout simulation")); + } + return Mono.empty(); + }; + + BlobCompactionAlgorithm algorithmWithFailingUpdater = new BlobCompactionAlgorithm( + rawStore, rawStore, mappingSource, partiallyFailingUpdater); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + CompactionResult result = algorithmWithFailingUpdater.initialCompact(request).block(); + + assertThat(result.packedBlobs()).isEqualTo(1); + + List remainingBlobs = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(remainingBlobs).contains(failingBlob); + assertThat(remainingBlobs).doesNotContain(succeedingBlob); + } + + static class RangedReadTrackingMemoryBlobStoreDAO extends MemoryBlobStoreDAO { + private final AtomicLong maxRangedReadBytes = new AtomicLong(0); + private final List wholeChunkReads = new CopyOnWriteArrayList<>(); + private final Map rawBlobs = new ConcurrentHashMap<>(); + + @Override + public Mono save(BucketName bucketName, BlobId blobId, BytesBlob blob) { + rawBlobs.put(blobId, blob.payload()); + return super.save(bucketName, blobId, blob); + } + + @Override + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + byte[] allBytes = rawBlobs.get(blobId); + if (allBytes == null) { + return Mono.error(new ObjectNotFoundException("Blob not found: " + blobId.asString())); + } + long totalSize = allBytes.length; + int from; + int to; + if (start < 0) { + int suffixLength = (int) Math.min(totalSize, -start); + from = (int) (totalSize - suffixLength); + to = (int) totalSize; + } else { + from = (int) Math.min(totalSize, start); + to = (int) Math.min(totalSize, end + 1); + } + byte[] slice = Arrays.copyOfRange(allBytes, from, to); + maxRangedReadBytes.updateAndGet(curr -> Math.max(curr, slice.length)); + return Mono.just(RangeByteSlice.of(slice, totalSize)); + } + + @Override + public Publisher readBytes(BucketName bucketName, BlobId blobId) { + if (ChunkId.isChunkRef(blobId)) { + wholeChunkReads.add(blobId); + } + return super.readBytes(bucketName, blobId); + } + + public long getMaxRangedReadBytes() { + return maxRangedReadBytes.get(); + } + + public List getWholeChunkReads() { + return new ArrayList<>(wholeChunkReads); + } + } + + @Test + void gcCompactShouldBoundHeapUsageByMaxSlotSizeAndNeverReadWholeChunkPayloads() throws Exception { + RangedReadTrackingMemoryBlobStoreDAO trackingStore = new RangedReadTrackingMemoryBlobStoreDAO(); + TestMappingSource localMapping = new TestMappingSource(); + RecordingBlobIdUpdater localUpdater = new RecordingBlobIdUpdater(); + BlobCompactionAlgorithm algorithm = new BlobCompactionAlgorithm( + trackingStore, trackingStore, localMapping, localUpdater); + + ChunkId chunkId = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + + Random random = new Random(42); + byte[] slot1Payload = new byte[50 * 1024]; + random.nextBytes(slot1Payload); + byte[] slot2Payload = new byte[100 * 1024]; + random.nextBytes(slot2Payload); + byte[] slot3Payload = new byte[75 * 1024]; + random.nextBytes(slot3Payload); + + ChunkWriteResult writeResult = ChunkFormat.writeChunk(List.of( + BlobSlotContent.of(slot1Payload), + BlobSlotContent.of(slot2Payload), + BlobSlotContent.of(slot3Payload) + )); + + Mono.from(trackingStore.save(TEST_BUCKET, chunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))).block(); + + SlotRange r1 = writeResult.slotRanges().get(0); + SlotRange r3 = writeResult.slotRanges().get(2); + ChunkId slot1Ref = ChunkId.slotRef(chunkId, r1.offset(), r1.limit()); + ChunkId slot3Ref = ChunkId.slotRef(chunkId, r3.offset(), r3.limit()); + + localMapping.add(slot1Ref, "msg-1"); + localMapping.add(slot3Ref, "msg-3"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .purgeDeadRatio(0.1) + .gainThreshold(0.1) + .build()) + .build(); + + CompactionResult result = algorithm.gcCompact(request).block(); + + assertThat(result.deadPurged()).isEqualTo(1); + assertThat(result.packedBlobs()).isEqualTo(0); + assertThat(localUpdater.getReplacements()).hasSize(2); + + // Verification of memory bounds (R2-2): + // 1. Whole chunk readBytes must NEVER be invoked on chunk objects + assertThat(trackingStore.getWholeChunkReads()).isEmpty(); + + // 2. The largest single ranged read must be strictly bounded by max(64KB footer, maxSlotSize + slot header) + long totalChunkSize = writeResult.chunkBytes().length; + assertThat(trackingStore.getMaxRangedReadBytes()).isLessThan(totalChunkSize); + assertThat(trackingStore.getMaxRangedReadBytes()).isLessThanOrEqualTo(Math.max(65536, 76 * 1024)); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTOTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTOTest.java new file mode 100644 index 00000000000..4056ea4e4e4 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskAdditionalInformationDTOTest.java @@ -0,0 +1,49 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Instant; +import java.util.Optional; + +import org.apache.james.JsonSerializationVerifier; +import org.apache.james.util.ClassLoaderUtils; +import org.junit.jupiter.api.Test; + +class BlobCompactionTaskAdditionalInformationDTOTest { + + @Test + void shouldMatchJsonSerializationContract() throws Exception { + JsonSerializationVerifier.dtoModule(BlobCompactionDTOModules.additionalInformationModule()) + .bean(new BlobCompactionTask.AdditionalInformation( + Instant.parse("2020-01-01T00:00:00Z"), + "default", + 2L, + Optional.of(1), + 10L, + 1024L, + 1L, + 2L, + 2L, + 512L + )) + .json(ClassLoaderUtils.getSystemResourceAsString("json/blobCompaction.additionalInformation.json")) + .verify(); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskSerializationTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskSerializationTest.java new file mode 100644 index 00000000000..47ef54ef9c3 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionTaskSerializationTest.java @@ -0,0 +1,68 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.mockito.Mockito.mock; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; + +import org.apache.james.JsonSerializationVerifier; +import org.apache.james.blob.api.BucketName; +import org.apache.james.util.ClassLoaderUtils; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +class BlobCompactionTaskSerializationTest { + private BlobCompactionAlgorithm algorithm; + private Clock clock; + + @BeforeEach + void setUp() { + algorithm = mock(BlobCompactionAlgorithm.class); + clock = Clock.fixed(Instant.parse("2020-01-01T00:00:00Z"), ZoneOffset.UTC); + } + + @Test + void shouldMatchJsonSerializationContract() throws Exception { + CompactionConfiguration config = CompactionConfiguration.builder() + .chunkTargetSize(100 * 1024 * 1024L) + .maxPackableSize(1024 * 1024L) + .purgeDeadRatio(0.1) + .mergeDeadRatio(0.5) + .gainThreshold(0.1) + .build(); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(BucketName.DEFAULT) + .generation(2L) + .family(1) + .configuration(config) + .build(); + + BlobCompactionTask task = new BlobCompactionTask(algorithm, request, clock); + + JsonSerializationVerifier.dtoModule(BlobCompactionDTOModules.taskModule(algorithm, clock)) + .bean(task) + .json(ClassLoaderUtils.getSystemResourceAsString("json/blobCompaction.task.json")) + .verify(); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java new file mode 100644 index 00000000000..5953e1f5341 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java @@ -0,0 +1,222 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.io.ByteArrayInputStream; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.List; + +import org.apache.james.blob.api.ObjectStoreIOException; +import org.junit.jupiter.api.Test; + +class ChunkFormatTest { + + @Test + void writeShouldStartWithFormatByte() throws Exception { + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of()); + assertThat(chunkBytes[0]).isEqualTo((byte) 0x01); + } + + @Test + void singleSlotRoundTrip() throws Exception { + byte[] raw = "Hello Apache James S3 Compaction!".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw))); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + assertThat(footer.slotCount()).isEqualTo(1); + assertThat(footer.slotStarts()).containsExactly(1L); + + long start = footer.slotStarts().get(0); + long end = footer.footerPosition(); + + byte[] slotData = Arrays.copyOfRange(chunkBytes, (int) start, (int) end); + byte[] decompressed = ChunkFormat.readSlot(new ByteArrayInputStream(slotData), start, end); + + assertThat(decompressed).isEqualTo(raw); + } + + @Test + void multipleSlotsRoundTrip() throws Exception { + byte[] raw1 = "Slot 1: Small text payload".getBytes(StandardCharsets.UTF_8); + byte[] raw2 = "Slot 2: Another chunked email part with somewhat larger content".getBytes(StandardCharsets.UTF_8); + byte[] raw3 = "Slot 3: Final message content in chunk".getBytes(StandardCharsets.UTF_8); + + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of( + ChunkFormat.BlobSlotContent.of(raw1), + ChunkFormat.BlobSlotContent.of(raw2), + ChunkFormat.BlobSlotContent.of(raw3) + )); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + assertThat(footer.slotCount()).isEqualTo(3); + assertThat(footer.slotStarts().get(0)).isEqualTo(1L); + + for (int i = 0; i < 3; i++) { + long start = footer.slotStarts().get(i); + long end = (i == 2) ? footer.footerPosition() : footer.slotStarts().get(i + 1); + + byte[] slotSlice = Arrays.copyOfRange(chunkBytes, (int) start, (int) end); + byte[] decompressed = ChunkFormat.readSlot(new ByteArrayInputStream(slotSlice), start, end); + + byte[] expected = switch (i) { + case 0 -> raw1; + case 1 -> raw2; + case 2 -> raw3; + default -> throw new IllegalStateException(); + }; + assertThat(decompressed).isEqualTo(expected); + } + } + + @Test + void emptyContentSlotRoundTrip() throws Exception { + byte[] raw = new byte[0]; + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw))); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + assertThat(footer.slotCount()).isEqualTo(1); + + long start = footer.slotStarts().get(0); + long end = footer.footerPosition(); + + byte[] slotData = Arrays.copyOfRange(chunkBytes, (int) start, (int) end); + byte[] decompressed = ChunkFormat.readSlot(new ByteArrayInputStream(slotData), start, end); + + assertThat(decompressed).isEmpty(); + } + + @Test + void crcCorruptionShouldThrowObjectStoreIOException() throws Exception { + byte[] raw = "Data that will be corrupted".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw))); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + long start = footer.slotStarts().get(0); + long end = footer.footerPosition(); + + byte[] slotData = Arrays.copyOfRange(chunkBytes, (int) start, (int) end); + + // Corrupt CRC32C field (bytes 8-11 in slot) + slotData[8] ^= 0x55; + + assertThatThrownBy(() -> ChunkFormat.readSlot(new ByteArrayInputStream(slotData), start, end)) + .isInstanceOf(ObjectStoreIOException.class) + .hasMessageContaining("CRC mismatch"); + } + + @Test + void corruptContentByteShouldFailDecompressionOrCRC() throws Exception { + byte[] raw = "Data to corrupt in compressed payload".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw))); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + long start = footer.slotStarts().get(0); + long end = footer.footerPosition(); + + byte[] slotData = Arrays.copyOfRange(chunkBytes, (int) start, (int) end); + + // Corrupt the last byte of the slot (compressed content) + slotData[slotData.length - 1] ^= 0xFF; + + assertThatThrownBy(() -> ChunkFormat.readSlot(new ByteArrayInputStream(slotData), start, end)) + .isInstanceOf(ObjectStoreIOException.class); + } + + @Test + void truncatedFooterShouldThrowObjectStoreIOException() { + byte[] truncated = new byte[6]; // Less than 12 bytes + assertThatThrownBy(() -> ChunkFormat.readFooter(truncated)) + .isInstanceOf(ObjectStoreIOException.class) + .hasMessageContaining("Tail buffer is too short"); + } + + @Test + void footerReadableFromTailBuffer() throws Exception { + byte[] raw = "Some content for tail buffer test".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw))); + + // Pretend chunk is part of a larger file, or tail buffer is exactly 65536 bytes + int tailSize = Math.min(chunkBytes.length, 65536); + byte[] tailBuffer = Arrays.copyOfRange(chunkBytes, chunkBytes.length - tailSize, chunkBytes.length); + + ChunkFooter footer = ChunkFormat.readFooter(tailBuffer, chunkBytes.length); + assertThat(footer.slotCount()).isEqualTo(1); + assertThat(footer.slotStarts()).containsExactly(1L); + } + + @Test + void bigPayloadRoundTrip() throws Exception { + byte[] raw = new byte[512 * 1024]; // 512 KB + for (int i = 0; i < raw.length; i++) { + raw[i] = (byte) (i % 127); + } + + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw))); + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + + long start = footer.slotStarts().get(0); + long end = footer.footerPosition(); + + byte[] slotData = Arrays.copyOfRange(chunkBytes, (int) start, (int) end); + byte[] decompressed = ChunkFormat.readSlot(new ByteArrayInputStream(slotData), start, end); + + assertThat(decompressed).isEqualTo(raw); + } + + @Test + void verifyBinaryLayoutExactStructure() throws Exception { + byte[] raw = "Test".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw))); + + // Byte 0: format byte 0x01 + assertThat(chunkBytes[0]).isEqualTo((byte) 0x01); + + // Bytes 1..8: contentStart = 1L + ByteBuffer bb = ByteBuffer.wrap(chunkBytes); + assertThat(bb.getLong(1)).isEqualTo(1L); + + // Bytes 9..12: CRC32C of "Test" + int crc = bb.getInt(9); + java.util.zip.CRC32C crcCalculator = new java.util.zip.CRC32C(); + crcCalculator.update(raw); + assertThat(crc).isEqualTo((int) crcCalculator.getValue()); + + // Followed by metadata + String metadataExpected = "content-encoding=zstd\ncontent-original-size=4\n"; + byte[] metadataBytes = metadataExpected.getBytes(StandardCharsets.US_ASCII); + byte[] actualMetadata = Arrays.copyOfRange(chunkBytes, 13, 13 + metadataBytes.length); + assertThat(actualMetadata).isEqualTo(metadataBytes); + + // Last 12 bytes + int totalLen = chunkBytes.length; + int footerLength = bb.getInt(totalLen - 12); + long footerPosition = bb.getLong(totalLen - 8); + + assertThat(footerPosition + footerLength).isEqualTo(totalLen - 12); + // footer content should be "1" + byte[] footerBytes = Arrays.copyOfRange(chunkBytes, (int) footerPosition, (int) footerPosition + footerLength); + assertThat(new String(footerBytes, StandardCharsets.US_ASCII)).isEqualTo("1"); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkIdTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkIdTest.java new file mode 100644 index 00000000000..d958bf89f59 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkIdTest.java @@ -0,0 +1,176 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.time.Clock; +import java.time.Duration; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; +import org.junit.jupiter.api.Test; + +class ChunkIdTest { + private static final GenerationAwareBlobId.Configuration CONFIG = + new GenerationAwareBlobId.Configuration(1, Duration.ofDays(30)); + + @Test + void ofChunkShouldCreateValidChunkId() { + ChunkId chunkId = ChunkId.ofChunk(CONFIG, 690); + + assertThat(chunkId.family()).isEqualTo(1); + assertThat(chunkId.generation()).isEqualTo(690); + assertThat(chunkId.offset()).isZero(); + assertThat(chunkId.limit()).isZero(); + assertThat(chunkId.randomPart()).hasSizeGreaterThanOrEqualTo(16); + assertThat(chunkId.asString()).startsWith("1_690_chunk").endsWith("~0~0"); + } + + @Test + void slotRefShouldCreateValidSlotRef() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slot = ChunkId.slotRef(chunk, 12345, 678); + + assertThat(slot.family()).isEqualTo(1); + assertThat(slot.generation()).isEqualTo(690); + assertThat(slot.offset()).isEqualTo(12345); + assertThat(slot.limit()).isEqualTo(678); + assertThat(slot.chunkId()).isEqualTo(chunk.chunkId()); + assertThat(slot.asString()).isEqualTo(chunk.chunkId() + "~12345~678"); + } + + @Test + void roundTripParseShouldPreserveAllFields() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slot = ChunkId.slotRef(chunk, 54321, 999); + + ChunkId parsed = ChunkId.parse(slot.asString()); + + assertThat(parsed).isEqualTo(slot); + assertThat(parsed.family()).isEqualTo(1); + assertThat(parsed.generation()).isEqualTo(690); + assertThat(parsed.randomPart()).isEqualTo(chunk.randomPart()); + assertThat(parsed.offset()).isEqualTo(54321); + assertThat(parsed.limit()).isEqualTo(999); + assertThat(parsed.chunkId()).isEqualTo(chunk.chunkId()); + assertThat(parsed.asString()).isEqualTo(slot.asString()); + } + + @Test + void parseShouldThrowWhenMissingTilde() { + assertThatThrownBy(() -> ChunkId.parse("1_690_chunkAb3def1234567890")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Missing '~'"); + } + + @Test + void parseShouldThrowWhenMissingSecondTilde() { + assertThatThrownBy(() -> ChunkId.parse("1_690_chunkAb3def1234567890~12345")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Missing second '~'"); + } + + @Test + void parseShouldThrowWhenBadGeneration() { + assertThatThrownBy(() -> ChunkId.parse("1_badGen_chunkAb3def1234567890~0~0")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Invalid family or generation"); + } + + @Test + void parseShouldThrowWhenRandomPartTooShort() { + assertThatThrownBy(() -> ChunkId.parse("1_690_chunkShort~0~0")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("too short"); + } + + @Test + void parseShouldThrowWhenNegativeOffset() { + assertThatThrownBy(() -> ChunkId.parse("1_690_chunkAb3def1234567890~-5~10")) + .isInstanceOf(IllegalArgumentException.class); + } + + @Test + void parseShouldThrowWhenNegativeLimit() { + assertThatThrownBy(() -> ChunkId.parse("1_690_chunkAb3def1234567890~5~-10")) + .isInstanceOf(IllegalArgumentException.class); + } + + @Test + void isChunkRefShouldDetectChunkRefString() { + assertThat(ChunkId.isChunkRef("1_690_chunkAb3def1234567890~12345~678")).isTrue(); + assertThat(ChunkId.isChunkRef("1_690_chunkAb3def1234567890")).isTrue(); + assertThat(ChunkId.isChunkRef("1_690_regularHashValue")).isFalse(); + assertThat(ChunkId.isChunkRef((String) null)).isFalse(); + } + + @Test + void isChunkRefShouldDetectChunkRefBlobId() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slot = ChunkId.slotRef(chunk, 100, 200); + + assertThat(ChunkId.isChunkRef(slot)).isTrue(); + assertThat(ChunkId.isChunkRef(new PlainBlobId("regularBlobId"))).isFalse(); + assertThat(ChunkId.isChunkRef((BlobId) null)).isFalse(); + } + + @Test + void shouldRoundTripThroughGenerationAwareBlobIdFactory() { + GenerationAwareBlobId.Factory factory = new GenerationAwareBlobId.Factory( + Clock.systemUTC(), + new PlainBlobId.Factory(), + CONFIG); + + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slot = ChunkId.slotRef(chunk, 12345, 678); + + GenerationAwareBlobId generationAwareBlobId = factory.parse(slot.asString()); + + assertThat(generationAwareBlobId.asString()).isEqualTo(slot.asString()); + } + + @Test + void withSuffixShouldThrowUnsupportedOperationException() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + + assertThatThrownBy(() -> chunk.withSuffix("suffix")) + .isInstanceOf(UnsupportedOperationException.class); + } + + @Test + void isChunkRefShouldNotMatchPlainHashesContainingChunkSubstring() { + assertThat(ChunkId.isChunkRef("1_690_hash_chunk_test")).isFalse(); + assertThat(ChunkId.isChunkRef("some_chunk_in_the_middle")).isFalse(); + assertThat(ChunkId.isChunkRef("abc_chunk1234")).isFalse(); + assertThat(ChunkId.isChunkRef("1_690_chunkShort")).isFalse(); + } + + @Test + void equalsShouldBeSymmetricWithPlainBlobId() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + PlainBlobId plainWithSameString = new PlainBlobId(chunk.asString()); + + assertThat(chunk.equals(plainWithSameString)).isFalse(); + assertThat(plainWithSameString.equals(chunk)).isFalse(); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java new file mode 100644 index 00000000000..d29d68a796d --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java @@ -0,0 +1,352 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.Collection; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.james.blob.aes.AESBlobStoreDAO; +import org.apache.james.blob.aes.CryptoConfig; +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BlobStoreDAO.Blob; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.ObjectNotFoundException; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.blob.memory.MemoryBlobStoreDAO; +import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.reactivestreams.Publisher; + +import reactor.core.publisher.Mono; + +class ChunkedBlobStoreDAOTest { + private static final BucketName TEST_BUCKET = BucketName.of("test-bucket"); + private static final GenerationAwareBlobId.Configuration CONFIG = + new GenerationAwareBlobId.Configuration(1, Duration.ofDays(30)); + private static final String SAMPLE_SALT = "c603a7327ee3dcbc031d8d34b1096c605feca5e1"; + private static final CryptoConfig CRYPTO_CONFIG = CryptoConfig.builder() + .salt(SAMPLE_SALT) + .password("testing-password".toCharArray()) + .build(); + + static class CountingRawStore implements BlobStoreDAO { + private final BlobStoreDAO delegate; + private final AtomicInteger readRangeCallCount = new AtomicInteger(); + + CountingRawStore(BlobStoreDAO delegate) { + this.delegate = delegate; + } + + int rangeCallCount() { + return readRangeCallCount.get(); + } + + void resetCount() { + readRangeCallCount.set(0); + } + + @Override + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + readRangeCallCount.incrementAndGet(); + return delegate.readRange(bucketName, blobId, start, end); + } + + @Override + public InputStreamBlob read(BucketName bucketName, BlobId blobId) { + return delegate.read(bucketName, blobId); + } + + @Override + public Publisher readReactive(BucketName bucketName, BlobId blobId) { + return delegate.readReactive(bucketName, blobId); + } + + @Override + public Publisher readBytes(BucketName bucketName, BlobId blobId) { + return delegate.readBytes(bucketName, blobId); + } + + @Override + public Publisher save(BucketName bucketName, BlobId blobId, Blob blob) { + return delegate.save(bucketName, blobId, blob); + } + + @Override + public Publisher delete(BucketName bucketName, BlobId blobId) { + return delegate.delete(bucketName, blobId); + } + + @Override + public Publisher delete(BucketName bucketName, Collection blobIds) { + return delegate.delete(bucketName, blobIds); + } + + @Override + public Publisher deleteBucket(BucketName bucketName) { + return delegate.deleteBucket(bucketName); + } + + @Override + public Publisher listBuckets() { + return delegate.listBuckets(); + } + + @Override + public Publisher listBlobs(BucketName bucketName) { + return delegate.listBlobs(bucketName); + } + + @Override + public Publisher listBlobs(BucketName bucketName, String prefix) { + return delegate.listBlobs(bucketName, prefix); + } + } + + private MemoryBlobStoreDAO rawMemoryStore; + private CountingRawStore countingRawStore; + private AESBlobStoreDAO plainChain; + private ChunkedBlobStoreDAO testee; + + @BeforeEach + void setUp() { + rawMemoryStore = new MemoryBlobStoreDAO(); + countingRawStore = new CountingRawStore(rawMemoryStore); + + // Plain chain: AES(raw) + plainChain = new AESBlobStoreDAO(countingRawStore, CRYPTO_CONFIG); + + // Outermost: Chunked(plainChain, raw) + testee = new ChunkedBlobStoreDAO(plainChain, countingRawStore); + } + + @Test + void plainIdsShouldPassThroughPlainChainByteIdentical() { + BlobId plainId = new PlainBlobId("normal-blob-id"); + byte[] payload = "Standard uncompressed plaintext email body".getBytes(StandardCharsets.UTF_8); + + Mono.from(testee.save(TEST_BUCKET, plainId, BlobStoreDAO.BytesBlob.of(payload))).block(); + + byte[] readBack = Mono.from(testee.readBytes(TEST_BUCKET, plainId)).block().payload(); + assertThat(readBack).isEqualTo(payload); + + // Confirm raw storage holds encrypted/compressed bytes, not plain text + byte[] rawBytes = Mono.from(rawMemoryStore.readBytes(TEST_BUCKET, plainId)).block().payload(); + assertThat(rawBytes).isNotEqualTo(payload); + } + + @Test + void chunkSlotRefReadReturnsOriginalContentWithExactlyOneRangedRead() throws Exception { + byte[] content1 = "First compacted message content".getBytes(StandardCharsets.UTF_8); + byte[] content2 = "Second compacted message content".getBytes(StandardCharsets.UTF_8); + + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of( + ChunkFormat.BlobSlotContent.of(content1), + ChunkFormat.BlobSlotContent.of(content2) + )); + + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 690); + Mono.from(testee.save(TEST_BUCKET, baseChunk, BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + long offset1 = footer.slotStarts().get(0); + long limit1 = footer.slotLength(0); + long offset2 = footer.slotStarts().get(1); + long limit2 = footer.slotLength(1); + + ChunkId slot1 = ChunkId.slotRef(baseChunk, offset1, limit1); + ChunkId slot2 = ChunkId.slotRef(baseChunk, offset2, limit2); + + countingRawStore.resetCount(); + + byte[] readSlot1 = Mono.from(testee.readBytes(TEST_BUCKET, slot1)).block().payload(); + assertThat(readSlot1).isEqualTo(content1); + assertThat(countingRawStore.rangeCallCount()).isEqualTo(1); // EXACTLY ONE ranged read! + + countingRawStore.resetCount(); + + byte[] readSlot2 = Mono.from(testee.readBytes(TEST_BUCKET, slot2)).block().payload(); + assertThat(readSlot2).isEqualTo(content2); + assertThat(countingRawStore.rangeCallCount()).isEqualTo(1); // EXACTLY ONE ranged read! + } + + @Test + void limitZeroSlotReadPerformsFooterWalk() throws Exception { + byte[] content = "Payload for footer walk".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(content))); + + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 690); + Mono.from(testee.save(TEST_BUCKET, baseChunk, BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + ChunkId walkRef = ChunkId.slotRef(baseChunk, 1, 0); // limit == 0 triggers footer walk + + countingRawStore.resetCount(); + byte[] readResult = Mono.from(testee.readBytes(TEST_BUCKET, walkRef)).block().payload(); + + assertThat(readResult).isEqualTo(content); + // Footer walk does 2 range reads: 1 for tail buffer (footer), 1 for slot data + assertThat(countingRawStore.rangeCallCount()).isEqualTo(2); + } + + @Test + void deleteChunkSlotRefShouldThrowUnsupportedOperationException() { + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slotRef = ChunkId.slotRef(baseChunk, 123, 456); + + assertThatThrownBy(() -> Mono.from(testee.delete(TEST_BUCKET, slotRef)).block()) + .isInstanceOf(UnsupportedOperationException.class) + .hasMessageContaining("Individual chunk slot refs cannot be deleted directly"); + } + + @Test + void deleteWholeChunkIdShouldSucceed() throws Exception { + byte[] content = "Whole chunk delete test".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(content))); + + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 690); + Mono.from(testee.save(TEST_BUCKET, baseChunk, BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + assertThat(Mono.from(rawMemoryStore.readBytes(TEST_BUCKET, baseChunk.chunkBlobId())).block()).isNotNull(); + + // Delete using chunkId (limit == 0) + Mono.from(testee.delete(TEST_BUCKET, baseChunk)).block(); + + assertThatThrownBy(() -> Mono.from(rawMemoryStore.readBytes(TEST_BUCKET, baseChunk.chunkBlobId())).block()) + .isInstanceOf(ObjectNotFoundException.class); + } + + @Test + void deduplicationInvariantBothMessagesReadIdenticalBytesFromSameSlot() throws Exception { + byte[] sharedEmailBody = "Identical deduplicated message body content".getBytes(StandardCharsets.UTF_8); + + // One slot in chunk holds shared content + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(sharedEmailBody))); + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 690); + Mono.from(testee.save(TEST_BUCKET, baseChunk, BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + long offset = footer.slotStarts().get(0); + long limit = footer.slotLength(0); + + ChunkId slotRefForMessageA = ChunkId.slotRef(baseChunk, offset, limit); + ChunkId slotRefForMessageB = ChunkId.slotRef(baseChunk, offset, limit); + + assertThat(slotRefForMessageA).isEqualTo(slotRefForMessageB); + + byte[] readForA = Mono.from(testee.readBytes(TEST_BUCKET, slotRefForMessageA)).block().payload(); + byte[] readForB = Mono.from(testee.readBytes(TEST_BUCKET, slotRefForMessageB)).block().payload(); + + assertThat(readForA).isEqualTo(sharedEmailBody); + assertThat(readForB).isEqualTo(sharedEmailBody); + } + + @Test + void readChunkSlotShouldTriggerRepairWhenCollaboratorProvided() throws Exception { + byte[] content = "Self-healing test data".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(content))); + + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 690); + Mono.from(testee.save(TEST_BUCKET, baseChunk, BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + ChunkId missingChunk = ChunkId.ofChunk(CONFIG, 999); + ChunkId staleSlot = ChunkId.slotRef(missingChunk, 0, 100); + + // Save original plain blob in plainChain + BlobId originalPlainBlob = new PlainBlobId("original-pre-compaction-blob"); + Mono.from(plainChain.save(TEST_BUCKET, originalPlainBlob, BlobStoreDAO.BytesBlob.of(content))).block(); + + // Repairer that repairs staleSlot to originalPlainBlob + BlobIdRepairer repairer = (bucket, staleBlobId) -> { + if (staleBlobId.equals(staleSlot)) { + return Mono.just(originalPlainBlob); + } + return Mono.empty(); + }; + + ChunkedBlobStoreDAO daoWithRepairer = new ChunkedBlobStoreDAO(plainChain, countingRawStore, Optional.of(repairer)); + + byte[] repairedRead = Mono.from(daoWithRepairer.readBytes(TEST_BUCKET, staleSlot)).block().payload(); + assertThat(repairedRead).isEqualTo(content); + } + + @Test + void readChunkSlotShouldTriggerRepairWhenRepairedToAnotherSlotRef() throws Exception { + byte[] content = "Repaired slot data".getBytes(StandardCharsets.UTF_8); + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(content))); + + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 700); + Mono.from(testee.save(TEST_BUCKET, baseChunk, BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + ChunkId validSlot = ChunkId.slotRef(baseChunk, footer.slotStarts().get(0), footer.slotLength(0)); + ChunkId staleSlot = ChunkId.slotRef(baseChunk, 999999L, 100L); + + BlobIdRepairer repairer = (bucket, staleBlobId) -> { + if (staleBlobId.equals(staleSlot)) { + return Mono.just(validSlot); + } + return Mono.empty(); + }; + + ChunkedBlobStoreDAO daoWithRepairer = new ChunkedBlobStoreDAO(plainChain, countingRawStore, Optional.of(repairer)); + + byte[] repairedRead = Mono.from(daoWithRepairer.readBytes(TEST_BUCKET, staleSlot)).block().payload(); + assertThat(repairedRead).isEqualTo(content); + } + + @Test + void saveShouldThrowWhenCalledWithSlotRef() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slot = ChunkId.slotRef(chunk, 100, 200); + + assertThatThrownBy(() -> Mono.from(testee.save(TEST_BUCKET, slot, BlobStoreDAO.BytesBlob.of("data".getBytes()))).block()) + .isInstanceOf(UnsupportedOperationException.class) + .hasMessageContaining("Individual chunk slot refs cannot be saved directly"); + } + + @Test + void deleteShouldThrowWhenCalledWithSlotRef() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slot = ChunkId.slotRef(chunk, 100, 200); + + assertThatThrownBy(() -> Mono.from(testee.delete(TEST_BUCKET, slot)).block()) + .isInstanceOf(UnsupportedOperationException.class) + .hasMessageContaining("Individual chunk slot refs cannot be deleted directly"); + } + + @Test + void batchDeleteShouldThrowWhenCollectionContainsSlotRef() { + ChunkId chunk = ChunkId.ofChunk(CONFIG, 690); + ChunkId slot = ChunkId.slotRef(chunk, 100, 200); + BlobId plainId = new PlainBlobId("plain-id"); + + assertThatThrownBy(() -> Mono.from(testee.delete(TEST_BUCKET, List.of(plainId, slot))).block()) + .isInstanceOf(UnsupportedOperationException.class) + .hasMessageContaining("Individual chunk slot refs cannot be deleted directly"); + } +} diff --git a/server/blob/blob-compaction/src/test/resources/json/blobCompaction.additionalInformation.json b/server/blob/blob-compaction/src/test/resources/json/blobCompaction.additionalInformation.json new file mode 100644 index 00000000000..a1fa43f5164 --- /dev/null +++ b/server/blob/blob-compaction/src/test/resources/json/blobCompaction.additionalInformation.json @@ -0,0 +1,13 @@ +{ + "type": "BlobCompactionTask", + "timestamp": "2020-01-01T00:00:00Z", + "bucketName": "default", + "generation": 2, + "family": 1, + "packedBlobs": 10, + "packedBytes": 1024, + "chunksWritten": 1, + "deadPurged": 2, + "mergedChunks": 2, + "freedBytes": 512 +} diff --git a/server/blob/blob-compaction/src/test/resources/json/blobCompaction.task.json b/server/blob/blob-compaction/src/test/resources/json/blobCompaction.task.json new file mode 100644 index 00000000000..487e00c76ef --- /dev/null +++ b/server/blob/blob-compaction/src/test/resources/json/blobCompaction.task.json @@ -0,0 +1,11 @@ +{ + "type": "BlobCompactionTask", + "bucketName": "default", + "generation": 2, + "family": 1, + "chunkTargetSize": 104857600, + "maxPackableSize": 1048576, + "purgeDeadRatio": 0.1, + "mergeDeadRatio": 0.5, + "gainThreshold": 0.1 +} diff --git a/server/blob/pom.xml b/server/blob/pom.xml index 508296036d2..de3e8f34b82 100644 --- a/server/blob/pom.xml +++ b/server/blob/pom.xml @@ -37,6 +37,7 @@ blob-api blob-cassandra blob-common + blob-compaction blob-export-api blob-export-file blob-file From 71d52345dd5193a916d439fd4c1e5e9988b6e20d Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:20:45 +0530 Subject: [PATCH 06/24] JAMES-4231 Cassandra & WebAdmin: Add metadata updater/repairer and WebAdmin API - Implement CassandraBlobReferenceMappingSource, CassandraBlobIdUpdater, and CassandraBlobIdRepairer for updating and self-healing blob references. - Wire BlobCompactionModule in Guice distributed server configuration. - Expose DELETE /blobs?scope=compaction&generation=&family= in BlobRoutes to schedule BlobCompactionTask via WebAdmin. - Add integration tests for Cassandra repair/update and WebAdmin compaction routes. --- mailbox/cassandra/pom.xml | 15 ++ .../mail/CassandraBlobIdRepairer.java | 215 ++++++++++++++++++ .../mail/CassandraBlobIdUpdater.java | 182 +++++++++++++++ .../CassandraBlobReferenceMappingSource.java | 74 ++++++ ...assandraBlobIdRepairerIntegrationTest.java | 203 +++++++++++++++++ ...CassandraBlobIdUpdaterIntegrationTest.java | 143 ++++++++++++ server/container/guice/cassandra/pom.xml | 4 + .../mailbox/CassandraMailboxModule.java | 10 + server/container/guice/distributed/pom.xml | 4 + .../blobstore/BlobCompactionModule.java | 103 +++++++++ .../blobstore/BlobStoreModulesChooser.java | 26 ++- .../BlobStoreModulesChooserTest.java | 80 +++++++ .../protocols/webadmin/webadmin-data/pom.xml | 4 + .../james/webadmin/routes/BlobRoutes.java | 70 +++++- .../routes/BlobRoutesCompactionTest.java | 204 +++++++++++++++++ 15 files changed, 1332 insertions(+), 5 deletions(-) create mode 100644 mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairer.java create mode 100644 mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdater.java create mode 100644 mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java create mode 100644 mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java create mode 100644 mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdaterIntegrationTest.java create mode 100644 server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java create mode 100644 server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java diff --git a/mailbox/cassandra/pom.xml b/mailbox/cassandra/pom.xml index 2cc11e3a858..0d5ea07a7c3 100644 --- a/mailbox/cassandra/pom.xml +++ b/mailbox/cassandra/pom.xml @@ -86,6 +86,21 @@ blob-cassandra test + + ${james.groupId} + blob-compaction + + + ${james.groupId} + blob-compaction + test-jar + test + + + ${james.groupId} + blob-memory + test + ${james.groupId} event-bus-api diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairer.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairer.java new file mode 100644 index 00000000000..b9a30d835d6 --- /dev/null +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairer.java @@ -0,0 +1,215 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.mailbox.cassandra.mail; + +import static com.datastax.oss.driver.api.querybuilder.QueryBuilder.bindMarker; +import static com.datastax.oss.driver.api.querybuilder.QueryBuilder.selectFrom; +import static com.datastax.oss.driver.api.querybuilder.QueryBuilder.update; +import static com.datastax.oss.driver.api.querybuilder.relation.Relation.column; +import static com.datastax.oss.driver.api.querybuilder.update.Assignment.setColumn; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageIds.IMAP_UID; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageIds.MAILBOX_ID; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageIds.MESSAGE_ID; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.BODY_CONTENT; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.HEADER_CONTENT; + +import java.util.UUID; + +import jakarta.inject.Inject; + +import org.apache.james.backends.cassandra.utils.CassandraAsyncExecutor; +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.compaction.BlobIdRepairer; +import org.apache.james.mailbox.cassandra.table.CassandraMessageIdTable; +import org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table; +import org.apache.james.mailbox.cassandra.table.MessageIdToImapUid; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.datastax.oss.driver.api.core.CqlSession; +import com.datastax.oss.driver.api.core.cql.PreparedStatement; +import com.datastax.oss.driver.api.core.type.codec.TypeCodecs; + +import reactor.core.publisher.Mono; + +public class CassandraBlobIdRepairer implements BlobIdRepairer { + private static final Logger LOGGER = LoggerFactory.getLogger(CassandraBlobIdRepairer.class); + + private final CassandraAsyncExecutor cassandraAsyncExecutor; + private final BlobId.Factory blobIdFactory; + + private final PreparedStatement scanMessageIdTable; + private final PreparedStatement selectImapUidSingle; + private final PreparedStatement updateMessageIdTableHeader; + + private final PreparedStatement scanImapUidTable; + private final PreparedStatement selectMessageIdTableSingle; + private final PreparedStatement updateImapUidTableHeader; + + private final PreparedStatement scanMessageV3; + private final PreparedStatement selectImapUidByMessageId; + private final PreparedStatement updateMessageV3Header; + + @Inject + public CassandraBlobIdRepairer(CqlSession session, BlobId.Factory blobIdFactory) { + this.cassandraAsyncExecutor = new CassandraAsyncExecutor(session); + this.blobIdFactory = blobIdFactory; + + this.scanMessageIdTable = session.prepare(selectFrom(CassandraMessageIdTable.TABLE_NAME) + .columns(MAILBOX_ID, IMAP_UID, MESSAGE_ID, HEADER_CONTENT) + .limit(1000) + .build()); + + this.selectImapUidSingle = session.prepare(selectFrom(MessageIdToImapUid.TABLE_NAME) + .column(HEADER_CONTENT) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID)), + column(MAILBOX_ID).isEqualTo(bindMarker(MAILBOX_ID)), + column(IMAP_UID).isEqualTo(bindMarker(IMAP_UID))) + .build()); + + this.updateMessageIdTableHeader = session.prepare(update(CassandraMessageIdTable.TABLE_NAME) + .set(setColumn(HEADER_CONTENT, bindMarker(HEADER_CONTENT))) + .where(column(MAILBOX_ID).isEqualTo(bindMarker(MAILBOX_ID)), + column(IMAP_UID).isEqualTo(bindMarker(IMAP_UID))) + .build()); + + this.scanImapUidTable = session.prepare(selectFrom(MessageIdToImapUid.TABLE_NAME) + .columns(MESSAGE_ID, MAILBOX_ID, IMAP_UID, HEADER_CONTENT) + .limit(1000) + .build()); + + this.selectMessageIdTableSingle = session.prepare(selectFrom(CassandraMessageIdTable.TABLE_NAME) + .column(HEADER_CONTENT) + .where(column(MAILBOX_ID).isEqualTo(bindMarker(MAILBOX_ID)), + column(IMAP_UID).isEqualTo(bindMarker(IMAP_UID))) + .build()); + + this.updateImapUidTableHeader = session.prepare(update(MessageIdToImapUid.TABLE_NAME) + .set(setColumn(HEADER_CONTENT, bindMarker(HEADER_CONTENT))) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID)), + column(MAILBOX_ID).isEqualTo(bindMarker(MAILBOX_ID)), + column(IMAP_UID).isEqualTo(bindMarker(IMAP_UID))) + .build()); + + this.scanMessageV3 = session.prepare(selectFrom(CassandraMessageV3Table.TABLE_NAME) + .columns(MESSAGE_ID, HEADER_CONTENT, BODY_CONTENT) + .limit(1000) + .build()); + + this.selectImapUidByMessageId = session.prepare(selectFrom(MessageIdToImapUid.TABLE_NAME) + .column(HEADER_CONTENT) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID))) + .build()); + + this.updateMessageV3Header = session.prepare(update(CassandraMessageV3Table.TABLE_NAME) + .set(setColumn(HEADER_CONTENT, bindMarker(HEADER_CONTENT))) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID))) + .build()); + } + + @Override + public Mono repair(BucketName bucketName, BlobId staleBlobId) { + String staleStr = staleBlobId.asString(); + + return repairFromMessageIdTable(staleStr) + .switchIfEmpty(repairFromImapUidTable(staleStr)) + .switchIfEmpty(repairFromMessageV3(staleStr)) + .doOnNext(repairedId -> LOGGER.info("Successfully repaired stale blob ID {} with canonical ID {}", staleStr, repairedId.asString())); + } + + private Mono repairFromMessageIdTable(String staleStr) { + return cassandraAsyncExecutor.executeRows(scanMessageIdTable.bind()) + .filter(row -> staleStr.equals(row.get(HEADER_CONTENT, TypeCodecs.TEXT))) + .concatMap(row -> { + UUID mailboxId = row.get(MAILBOX_ID, TypeCodecs.UUID); + Long imapUid = row.get(IMAP_UID, TypeCodecs.BIGINT); + UUID messageId = row.get(MESSAGE_ID, TypeCodecs.TIMEUUID); + + return cassandraAsyncExecutor.executeSingleRow(selectImapUidSingle.bind() + .set(MESSAGE_ID, messageId, TypeCodecs.TIMEUUID) + .set(MAILBOX_ID, mailboxId, TypeCodecs.UUID) + .set(IMAP_UID, imapUid, TypeCodecs.BIGINT)) + .flatMap(imapRow -> { + String canonical = imapRow.get(HEADER_CONTENT, TypeCodecs.TEXT); + if (canonical != null && !canonical.equals(staleStr)) { + return cassandraAsyncExecutor.executeVoid(updateMessageIdTableHeader.bind() + .set(HEADER_CONTENT, canonical, TypeCodecs.TEXT) + .set(MAILBOX_ID, mailboxId, TypeCodecs.UUID) + .set(IMAP_UID, imapUid, TypeCodecs.BIGINT)) + .thenReturn(blobIdFactory.parse(canonical)); + } + return Mono.empty(); + }); + }) + .next(); + } + + private Mono repairFromImapUidTable(String staleStr) { + return cassandraAsyncExecutor.executeRows(scanImapUidTable.bind()) + .filter(row -> staleStr.equals(row.get(HEADER_CONTENT, TypeCodecs.TEXT))) + .concatMap(row -> { + UUID messageId = row.get(MESSAGE_ID, TypeCodecs.TIMEUUID); + UUID mailboxId = row.get(MAILBOX_ID, TypeCodecs.UUID); + Long imapUid = row.get(IMAP_UID, TypeCodecs.BIGINT); + + return cassandraAsyncExecutor.executeSingleRow(selectMessageIdTableSingle.bind() + .set(MAILBOX_ID, mailboxId, TypeCodecs.UUID) + .set(IMAP_UID, imapUid, TypeCodecs.BIGINT)) + .flatMap(msgRow -> { + String canonical = msgRow.get(HEADER_CONTENT, TypeCodecs.TEXT); + if (canonical != null && !canonical.equals(staleStr)) { + return cassandraAsyncExecutor.executeVoid(updateImapUidTableHeader.bind() + .set(HEADER_CONTENT, canonical, TypeCodecs.TEXT) + .set(MESSAGE_ID, messageId, TypeCodecs.TIMEUUID) + .set(MAILBOX_ID, mailboxId, TypeCodecs.UUID) + .set(IMAP_UID, imapUid, TypeCodecs.BIGINT)) + .thenReturn(blobIdFactory.parse(canonical)); + } + return Mono.empty(); + }); + }) + .next(); + } + + private Mono repairFromMessageV3(String staleStr) { + return cassandraAsyncExecutor.executeRows(scanMessageV3.bind()) + .filter(row -> staleStr.equals(row.get(HEADER_CONTENT, TypeCodecs.TEXT))) + .concatMap(row -> { + UUID messageId = row.get(MESSAGE_ID, TypeCodecs.TIMEUUID); + + return cassandraAsyncExecutor.executeRows(selectImapUidByMessageId.bind() + .set(MESSAGE_ID, messageId, TypeCodecs.TIMEUUID)) + .filter(imapRow -> { + String canonical = imapRow.get(HEADER_CONTENT, TypeCodecs.TEXT); + return canonical != null && !canonical.equals(staleStr); + }) + .next() + .flatMap(imapRow -> { + String canonical = imapRow.get(HEADER_CONTENT, TypeCodecs.TEXT); + return cassandraAsyncExecutor.executeVoid(updateMessageV3Header.bind() + .set(HEADER_CONTENT, canonical, TypeCodecs.TEXT) + .set(MESSAGE_ID, messageId, TypeCodecs.TIMEUUID)) + .thenReturn(blobIdFactory.parse(canonical)); + }); + }) + .next(); + } +} diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdater.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdater.java new file mode 100644 index 00000000000..b230da52638 --- /dev/null +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdater.java @@ -0,0 +1,182 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.mailbox.cassandra.mail; + +import static com.datastax.oss.driver.api.querybuilder.QueryBuilder.bindMarker; +import static com.datastax.oss.driver.api.querybuilder.QueryBuilder.selectFrom; +import static com.datastax.oss.driver.api.querybuilder.QueryBuilder.update; +import static com.datastax.oss.driver.api.querybuilder.relation.Relation.column; +import static com.datastax.oss.driver.api.querybuilder.update.Assignment.setColumn; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageIds.IMAP_UID; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageIds.MAILBOX_ID; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageIds.MESSAGE_ID; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.BODY_CONTENT; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.HEADER_CONTENT; + +import java.util.Collection; +import java.util.UUID; + +import jakarta.inject.Inject; + +import org.apache.james.backends.cassandra.utils.CassandraAsyncExecutor; +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.compaction.BlobIdUpdater; +import org.apache.james.mailbox.cassandra.table.CassandraMessageIdTable; +import org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table; +import org.apache.james.mailbox.cassandra.table.MessageIdToImapUid; + +import com.datastax.oss.driver.api.core.CqlSession; +import com.datastax.oss.driver.api.core.cql.PreparedStatement; +import com.datastax.oss.driver.api.core.type.codec.TypeCodecs; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +/** + * Updates Cassandra message references from old standalone blob IDs to compacted chunk slot IDs. + * + *

Consistency Window and Crash Safety

+ * Updates span multiple Cassandra tables: + *
    + *
  • {@code messageV3} (primary message metadata table, keyed by {@code message_id})
    • + *
  • {@code messageIdToImapUid} (lookup table, keyed by {@code message_id})
    • + *
  • {@code messageIdTable} (mailbox message table, keyed by {@code mailbox_id})
    • + *
+ * + *

Because these tables have different partition keys, updates cannot be executed in an atomic + * Cassandra logged batch without significant cross-node write penalties. Therefore, statements + * are executed asynchronously. If the process crashes mid-update:

+ *
    + *
  • Candidates whose updates did not succeed will NOT have their original blobs deleted + * by compaction algorithms.
    • + *
  • If {@code messageV3} is updated but denormalized index tables ({@code imapUid} / {@code messageIdTable}) + * are not yet updated before a crash, a transient inconsistency window exists.
    • + *
  • This transient window is non-destructive and is strictly healed by running + * {@link CassandraBlobIdRepairer}.
    • + *
+ */ +public class CassandraBlobIdUpdater implements BlobIdUpdater { + private final CassandraAsyncExecutor cassandraAsyncExecutor; + private final PreparedStatement selectMessageV3; + private final PreparedStatement updateMessageV3Header; + private final PreparedStatement updateMessageV3Body; + private final PreparedStatement selectImapUidByMessageId; + private final PreparedStatement updateImapUidHeader; + private final PreparedStatement updateMessageIdTableHeader; + + @Inject + public CassandraBlobIdUpdater(CqlSession session) { + this.cassandraAsyncExecutor = new CassandraAsyncExecutor(session); + + this.selectMessageV3 = session.prepare(selectFrom(CassandraMessageV3Table.TABLE_NAME) + .columns(HEADER_CONTENT, BODY_CONTENT) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID))) + .build()); + + this.updateMessageV3Header = session.prepare(update(CassandraMessageV3Table.TABLE_NAME) + .set(setColumn(HEADER_CONTENT, bindMarker(HEADER_CONTENT))) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID))) + .build()); + + this.updateMessageV3Body = session.prepare(update(CassandraMessageV3Table.TABLE_NAME) + .set(setColumn(BODY_CONTENT, bindMarker(BODY_CONTENT))) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID))) + .build()); + + this.selectImapUidByMessageId = session.prepare(selectFrom(MessageIdToImapUid.TABLE_NAME) + .columns(MAILBOX_ID, IMAP_UID, HEADER_CONTENT) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID))) + .build()); + + this.updateImapUidHeader = session.prepare(update(MessageIdToImapUid.TABLE_NAME) + .set(setColumn(HEADER_CONTENT, bindMarker(HEADER_CONTENT))) + .where(column(MESSAGE_ID).isEqualTo(bindMarker(MESSAGE_ID)), + column(MAILBOX_ID).isEqualTo(bindMarker(MAILBOX_ID)), + column(IMAP_UID).isEqualTo(bindMarker(IMAP_UID))) + .build()); + + this.updateMessageIdTableHeader = session.prepare(update(CassandraMessageIdTable.TABLE_NAME) + .set(setColumn(HEADER_CONTENT, bindMarker(HEADER_CONTENT))) + .where(column(MAILBOX_ID).isEqualTo(bindMarker(MAILBOX_ID)), + column(IMAP_UID).isEqualTo(bindMarker(IMAP_UID))) + .build()); + } + + @Override + public Mono replaceReferences(BlobId oldId, BlobId newId, Collection messageIds) { + String oldIdStr = oldId.asString(); + String newIdStr = newId.asString(); + + return Flux.fromIterable(messageIds) + .flatMap(messageIdStr -> updateMessageReferences(messageIdStr, oldIdStr, newIdStr), 16) + .then(); + } + + private Mono updateMessageReferences(String messageIdStr, String oldIdStr, String newIdStr) { + UUID messageUuid; + try { + messageUuid = UUID.fromString(messageIdStr); + } catch (IllegalArgumentException e) { + return Mono.empty(); + } + + Mono updateV3 = cassandraAsyncExecutor.executeSingleRow( + selectMessageV3.bind().set(MESSAGE_ID, messageUuid, TypeCodecs.TIMEUUID)) + .flatMap(row -> { + String header = row.get(HEADER_CONTENT, TypeCodecs.TEXT); + String body = row.get(BODY_CONTENT, TypeCodecs.TEXT); + Mono headerUpdate = (header != null && header.equals(oldIdStr)) + ? cassandraAsyncExecutor.executeVoid(updateMessageV3Header.bind() + .set(HEADER_CONTENT, newIdStr, TypeCodecs.TEXT) + .set(MESSAGE_ID, messageUuid, TypeCodecs.TIMEUUID)) + : Mono.empty(); + Mono bodyUpdate = (body != null && body.equals(oldIdStr)) + ? cassandraAsyncExecutor.executeVoid(updateMessageV3Body.bind() + .set(BODY_CONTENT, newIdStr, TypeCodecs.TEXT) + .set(MESSAGE_ID, messageUuid, TypeCodecs.TIMEUUID)) + : Mono.empty(); + return Flux.merge(headerUpdate, bodyUpdate).then(); + }); + + Mono updateDenormalized = cassandraAsyncExecutor.executeRows( + selectImapUidByMessageId.bind().set(MESSAGE_ID, messageUuid, TypeCodecs.TIMEUUID)) + .flatMap(row -> { + String header = row.get(HEADER_CONTENT, TypeCodecs.TEXT); + if (header != null && header.equals(oldIdStr)) { + UUID mailboxId = row.get(MAILBOX_ID, TypeCodecs.UUID); + Long imapUid = row.get(IMAP_UID, TypeCodecs.BIGINT); + Mono imapUidUpdate = cassandraAsyncExecutor.executeVoid(updateImapUidHeader.bind() + .set(HEADER_CONTENT, newIdStr, TypeCodecs.TEXT) + .set(MESSAGE_ID, messageUuid, TypeCodecs.TIMEUUID) + .set(MAILBOX_ID, mailboxId, TypeCodecs.UUID) + .set(IMAP_UID, imapUid, TypeCodecs.BIGINT)); + Mono messageIdTableUpdate = cassandraAsyncExecutor.executeVoid(updateMessageIdTableHeader.bind() + .set(HEADER_CONTENT, newIdStr, TypeCodecs.TEXT) + .set(MAILBOX_ID, mailboxId, TypeCodecs.UUID) + .set(IMAP_UID, imapUid, TypeCodecs.BIGINT)); + return Flux.merge(imapUidUpdate, messageIdTableUpdate).then(); + } + return Mono.empty(); + }) + .then(); + + return Flux.merge(updateV3, updateDenormalized).then(); + } +} diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java new file mode 100644 index 00000000000..af536b0bb1a --- /dev/null +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java @@ -0,0 +1,74 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.mailbox.cassandra.mail; + +import static org.apache.james.mailbox.cassandra.table.CassandraMessageIds.MESSAGE_ID; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.BODY_CONTENT; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.HEADER_CONTENT; +import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.TABLE_NAME; + +import java.util.ArrayList; +import java.util.List; + +import jakarta.inject.Inject; + +import org.apache.james.backends.cassandra.utils.CassandraAsyncExecutor; +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.compaction.BlobReferenceMappingSource; +import org.reactivestreams.Publisher; + +import com.datastax.oss.driver.api.core.CqlSession; +import com.datastax.oss.driver.api.core.cql.PreparedStatement; +import com.datastax.oss.driver.api.core.type.codec.TypeCodecs; +import com.datastax.oss.driver.api.querybuilder.QueryBuilder; + +public class CassandraBlobReferenceMappingSource implements BlobReferenceMappingSource { + private final CassandraAsyncExecutor cassandraAsyncExecutor; + private final BlobId.Factory blobIdFactory; + private final PreparedStatement selectAll; + + @Inject + public CassandraBlobReferenceMappingSource(CqlSession session, BlobId.Factory blobIdFactory) { + this.cassandraAsyncExecutor = new CassandraAsyncExecutor(session); + this.blobIdFactory = blobIdFactory; + this.selectAll = session.prepare(QueryBuilder.selectFrom(TABLE_NAME) + .columns(MESSAGE_ID, HEADER_CONTENT, BODY_CONTENT) + .build()); + } + + @Override + public Publisher listBlobIdMessageIdMappings() { + return cassandraAsyncExecutor.executeRows(selectAll.bind()) + .flatMapIterable(row -> { + String messageIdStr = row.get(MESSAGE_ID, TypeCodecs.TIMEUUID).toString(); + String headerContent = row.get(HEADER_CONTENT, TypeCodecs.TEXT); + String bodyContent = row.get(BODY_CONTENT, TypeCodecs.TEXT); + + List mappings = new ArrayList<>(2); + if (headerContent != null && !headerContent.isEmpty()) { + mappings.add(new BlobIdMessageIdMapping(blobIdFactory.parse(headerContent), messageIdStr)); + } + if (bodyContent != null && !bodyContent.isEmpty()) { + mappings.add(new BlobIdMessageIdMapping(blobIdFactory.parse(bodyContent), messageIdStr)); + } + return mappings; + }); + } +} diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java new file mode 100644 index 00000000000..fa397640e23 --- /dev/null +++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java @@ -0,0 +1,203 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.mailbox.cassandra.mail; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.Date; +import java.util.List; +import java.util.Optional; + +import jakarta.mail.Flags; + +import org.apache.james.backends.cassandra.CassandraCluster; +import org.apache.james.backends.cassandra.CassandraClusterExtension; +import org.apache.james.backends.cassandra.components.CassandraDataDefinition; +import org.apache.james.backends.cassandra.init.configuration.CassandraConfiguration; +import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionDataDefinition; +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.blob.cassandra.CassandraBlobDataDefinition; +import org.apache.james.blob.compaction.ChunkFooter; +import org.apache.james.blob.compaction.ChunkFormat; +import org.apache.james.blob.compaction.ChunkId; +import org.apache.james.blob.compaction.ChunkedBlobStoreDAO; +import org.apache.james.blob.memory.MemoryBlobStoreDAO; +import org.apache.james.mailbox.MessageUid; +import org.apache.james.mailbox.ModSeq; +import org.apache.james.mailbox.cassandra.ids.CassandraId; +import org.apache.james.mailbox.cassandra.ids.CassandraMessageId; +import org.apache.james.mailbox.cassandra.modules.CassandraMessageDataDefinition; +import org.apache.james.mailbox.model.ComposedMessageId; +import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData; +import org.apache.james.mailbox.model.ThreadId; +import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +import reactor.core.publisher.Mono; + +class CassandraBlobIdRepairerIntegrationTest { + public static final CassandraDataDefinition MODULES = CassandraDataDefinition.aggregateModules( + CassandraMessageDataDefinition.MODULE, + CassandraBlobDataDefinition.MODULE, + CassandraSchemaVersionDataDefinition.MODULE); + + private static final BucketName TEST_BUCKET = BucketName.of("test-bucket"); + private static final GenerationAwareBlobId.Configuration CONFIG = + new GenerationAwareBlobId.Configuration(1, Duration.ofDays(30)); + + @RegisterExtension + static CassandraClusterExtension cassandraCluster = new CassandraClusterExtension(MODULES); + + private CassandraMessageId.Factory messageIdFactory; + private PlainBlobId.Factory blobIdFactory; + private CassandraMessageIdDAO messageIdDAO; + private CassandraMessageIdToImapUidDAO imapUidDAO; + private MemoryBlobStoreDAO rawStore; + private ChunkedBlobStoreDAO chunkedBlobStoreDAO; + + @BeforeEach + void setUp(CassandraCluster cassandra) { + messageIdFactory = new CassandraMessageId.Factory(); + blobIdFactory = new PlainBlobId.Factory(); + messageIdDAO = new CassandraMessageIdDAO(cassandra.getConf(), blobIdFactory); + imapUidDAO = new CassandraMessageIdToImapUidDAO(cassandra.getConf(), blobIdFactory, CassandraConfiguration.DEFAULT_CONFIGURATION); + rawStore = new MemoryBlobStoreDAO(); + + CassandraBlobIdRepairer repairer = new CassandraBlobIdRepairer(cassandra.getConf(), blobIdFactory); + chunkedBlobStoreDAO = new ChunkedBlobStoreDAO(rawStore, rawStore, Optional.of(repairer)); + } + + @Test + void repairShouldFixCorruptedMessageIdTableFromImapUidTable() throws Exception { + byte[] content = "Subject: Hello\n\nThis is the email body".getBytes(StandardCharsets.UTF_8); + + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(content))); + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 555); + Mono.from(rawStore.save(TEST_BUCKET, baseChunk.chunkBlobId(), BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + long offset = footer.slotStarts().get(0); + long limit = footer.slotLength(0); + ChunkId validSlotRef = ChunkId.slotRef(baseChunk, offset, limit); + ChunkId deadSlotRef = ChunkId.slotRef(baseChunk, 999999L, 100L); + + CassandraMessageId messageId = messageIdFactory.generate(); + CassandraId mailboxId = CassandraId.timeBased(); + MessageUid messageUid = MessageUid.of(42); + + // Put valid slot in imapUidTable + imapUidDAO.insert(CassandraMessageMetadata.builder() + .ids(ComposedMessageIdWithMetaData.builder() + .composedMessageId(new ComposedMessageId(mailboxId, messageId, messageUid)) + .flags(new Flags()) + .modSeq(ModSeq.of(1)) + .threadId(ThreadId.fromBaseMessageId(messageId)) + .build()) + .internalDate(new Date()) + .bodyStartOctet(10L) + .size(100L) + .headerContent(Optional.of(validSlotRef)) + .build()).block(); + + // Corrupt messageIdTable with deadSlotRef + messageIdDAO.insert(CassandraMessageMetadata.builder() + .ids(ComposedMessageIdWithMetaData.builder() + .composedMessageId(new ComposedMessageId(mailboxId, messageId, messageUid)) + .flags(new Flags()) + .modSeq(ModSeq.of(1)) + .threadId(ThreadId.fromBaseMessageId(messageId)) + .build()) + .internalDate(new Date()) + .bodyStartOctet(10L) + .size(100L) + .headerContent(Optional.of(deadSlotRef)) + .build()).block(); + + // Read using deadSlotRef: triggers repair -> fetches canonical from imapUidTable -> fixes messageIdTable + byte[] readBack = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, deadSlotRef)).block().payload(); + assertThat(readBack).isEqualTo(content); + + // Verify messageIdTable is now repaired with the canonical validSlotRef + CassandraMessageMetadata repairedMetadata = messageIdDAO.retrieve(mailboxId, messageUid).block().orElseThrow(); + assertThat(repairedMetadata.getHeaderContent().map(BlobId::asString)).contains(validSlotRef.asString()); + } + + @Test + void repairShouldFixCorruptedImapUidTableFromMessageIdTable() throws Exception { + byte[] content = "Subject: Another Test\n\nSecond body".getBytes(StandardCharsets.UTF_8); + + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(content))); + ChunkId baseChunk = ChunkId.ofChunk(CONFIG, 777); + Mono.from(rawStore.save(TEST_BUCKET, baseChunk.chunkBlobId(), BlobStoreDAO.BytesBlob.of(chunkBytes))).block(); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + long offset = footer.slotStarts().get(0); + long limit = footer.slotLength(0); + ChunkId validSlotRef = ChunkId.slotRef(baseChunk, offset, limit); + ChunkId deadSlotRef = ChunkId.slotRef(baseChunk, 888888L, 200L); + + CassandraMessageId messageId = messageIdFactory.generate(); + CassandraId mailboxId = CassandraId.timeBased(); + MessageUid messageUid = MessageUid.of(99); + + // Put valid slot in messageIdTable + messageIdDAO.insert(CassandraMessageMetadata.builder() + .ids(ComposedMessageIdWithMetaData.builder() + .composedMessageId(new ComposedMessageId(mailboxId, messageId, messageUid)) + .flags(new Flags()) + .modSeq(ModSeq.of(1)) + .threadId(ThreadId.fromBaseMessageId(messageId)) + .build()) + .internalDate(new Date()) + .bodyStartOctet(10L) + .size(100L) + .headerContent(Optional.of(validSlotRef)) + .build()).block(); + + // Corrupt imapUidTable with deadSlotRef + imapUidDAO.insert(CassandraMessageMetadata.builder() + .ids(ComposedMessageIdWithMetaData.builder() + .composedMessageId(new ComposedMessageId(mailboxId, messageId, messageUid)) + .flags(new Flags()) + .modSeq(ModSeq.of(1)) + .threadId(ThreadId.fromBaseMessageId(messageId)) + .build()) + .internalDate(new Date()) + .bodyStartOctet(10L) + .size(100L) + .headerContent(Optional.of(deadSlotRef)) + .build()).block(); + + // Read using deadSlotRef: triggers repair -> fetches canonical from messageIdTable -> fixes imapUidTable + byte[] readBack = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, deadSlotRef)).block().payload(); + assertThat(readBack).isEqualTo(content); + + // Verify imapUidTable is now repaired with the canonical validSlotRef + CassandraMessageMetadata repairedMetadata = imapUidDAO.retrieve(messageId, Optional.of(mailboxId)).blockFirst(); + assertThat(repairedMetadata.getHeaderContent().map(BlobId::asString)).contains(validSlotRef.asString()); + } +} diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdaterIntegrationTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdaterIntegrationTest.java new file mode 100644 index 00000000000..a121cd6f1dd --- /dev/null +++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdUpdaterIntegrationTest.java @@ -0,0 +1,143 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.mailbox.cassandra.mail; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.time.Instant; +import java.util.Date; +import java.util.List; +import java.util.Optional; + +import jakarta.mail.Flags; + +import org.apache.james.backends.cassandra.CassandraCluster; +import org.apache.james.backends.cassandra.CassandraClusterExtension; +import org.apache.james.backends.cassandra.components.CassandraDataDefinition; +import org.apache.james.backends.cassandra.init.configuration.CassandraConfiguration; +import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionDataDefinition; +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.blob.cassandra.CassandraBlobDataDefinition; +import org.apache.james.mailbox.MessageUid; +import org.apache.james.mailbox.ModSeq; +import org.apache.james.mailbox.cassandra.ids.CassandraId; +import org.apache.james.mailbox.cassandra.ids.CassandraMessageId; +import org.apache.james.mailbox.cassandra.modules.CassandraMessageDataDefinition; +import org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table; +import org.apache.james.mailbox.model.ComposedMessageId; +import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData; +import org.apache.james.mailbox.model.ThreadId; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +class CassandraBlobIdUpdaterIntegrationTest { + public static final CassandraDataDefinition MODULES = CassandraDataDefinition.aggregateModules( + CassandraMessageDataDefinition.MODULE, + CassandraBlobDataDefinition.MODULE, + CassandraSchemaVersionDataDefinition.MODULE); + + @RegisterExtension + static CassandraClusterExtension cassandraCluster = new CassandraClusterExtension(MODULES); + + private CassandraMessageId.Factory messageIdFactory; + private PlainBlobId.Factory blobIdFactory; + private CassandraMessageIdDAO messageIdDAO; + private CassandraMessageIdToImapUidDAO imapUidDAO; + private CassandraBlobIdUpdater testee; + + @BeforeEach + void setUp(CassandraCluster cassandra) { + messageIdFactory = new CassandraMessageId.Factory(); + blobIdFactory = new PlainBlobId.Factory(); + messageIdDAO = new CassandraMessageIdDAO(cassandra.getConf(), blobIdFactory); + imapUidDAO = new CassandraMessageIdToImapUidDAO(cassandra.getConf(), blobIdFactory, CassandraConfiguration.DEFAULT_CONFIGURATION); + testee = new CassandraBlobIdUpdater(cassandra.getConf()); + } + + @Test + void replaceReferencesShouldUpdateMessageV3AndDenormalizedTables(CassandraCluster cassandra) { + CassandraMessageId messageId = messageIdFactory.generate(); + CassandraId mailboxId = CassandraId.timeBased(); + MessageUid messageUid = MessageUid.of(100); + + BlobId originalHeaderBlobId = blobIdFactory.of("header-old-blob-id"); + BlobId originalBodyBlobId = blobIdFactory.of("body-old-blob-id"); + BlobId newHeaderSlotRef = blobIdFactory.of("chunk-123_chunk_456~0~100"); + BlobId newBodySlotRef = blobIdFactory.of("chunk-123_chunk_456~100~500"); + + // Insert into messageV3 + cassandra.getConf().execute( + "INSERT INTO " + CassandraMessageV3Table.TABLE_NAME + + " (messageId, internalDate, bodyStartOctet, fullContentOctets, headerContent, bodyContent)" + + " VALUES (?, ?, ?, ?, ?, ?)", + messageId.get(), Instant.now(), 10, 100L, originalHeaderBlobId.asString(), originalBodyBlobId.asString()); + + // Insert into messageIdTable + messageIdDAO.insert(CassandraMessageMetadata.builder() + .ids(ComposedMessageIdWithMetaData.builder() + .composedMessageId(new ComposedMessageId(mailboxId, messageId, messageUid)) + .flags(new Flags()) + .modSeq(ModSeq.of(1)) + .threadId(ThreadId.fromBaseMessageId(messageId)) + .build()) + .internalDate(new Date()) + .bodyStartOctet(10L) + .size(100L) + .headerContent(Optional.of(originalHeaderBlobId)) + .build()).block(); + + // Insert into imapUidTable + imapUidDAO.insert(CassandraMessageMetadata.builder() + .ids(ComposedMessageIdWithMetaData.builder() + .composedMessageId(new ComposedMessageId(mailboxId, messageId, messageUid)) + .flags(new Flags()) + .modSeq(ModSeq.of(1)) + .threadId(ThreadId.fromBaseMessageId(messageId)) + .build()) + .internalDate(new Date()) + .bodyStartOctet(10L) + .size(100L) + .headerContent(Optional.of(originalHeaderBlobId)) + .build()).block(); + + // Replace header reference + testee.replaceReferences(originalHeaderBlobId, newHeaderSlotRef, List.of(messageId.serialize())).block(); + + // Replace body reference + testee.replaceReferences(originalBodyBlobId, newBodySlotRef, List.of(messageId.serialize())).block(); + + // Verify messageV3 has both new slot refs + var row = cassandra.getConf().execute( + "SELECT headerContent, bodyContent FROM " + CassandraMessageV3Table.TABLE_NAME + " WHERE messageId = ?", + messageId.get()).one(); + assertThat(row.getString("headerContent")).isEqualTo(newHeaderSlotRef.asString()); + assertThat(row.getString("bodyContent")).isEqualTo(newBodySlotRef.asString()); + + // Verify messageIdTable has new header slot ref + CassandraMessageMetadata metaFromMessageIdTable = messageIdDAO.retrieve(mailboxId, messageUid).block().orElseThrow(); + assertThat(metaFromMessageIdTable.getHeaderContent()).contains(newHeaderSlotRef); + + // Verify imapUidTable has new header slot ref + CassandraMessageMetadata metaFromImapUidTable = imapUidDAO.retrieve(messageId, Optional.of(mailboxId)).blockFirst(); + assertThat(metaFromImapUidTable.getHeaderContent()).contains(newHeaderSlotRef); + } +} diff --git a/server/container/guice/cassandra/pom.xml b/server/container/guice/cassandra/pom.xml index 4be22287d6c..d356bd76077 100644 --- a/server/container/guice/cassandra/pom.xml +++ b/server/container/guice/cassandra/pom.xml @@ -75,6 +75,10 @@ blob-cassandra ${project.version} + + ${james.groupId} + blob-compaction + ${james.groupId} blob-export-guice diff --git a/server/container/guice/cassandra/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java b/server/container/guice/cassandra/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java index 5dd427a2a61..af62b81038b 100644 --- a/server/container/guice/cassandra/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java +++ b/server/container/guice/cassandra/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java @@ -35,6 +35,9 @@ import org.apache.james.backends.cassandra.init.configuration.CassandraConfiguration; import org.apache.james.blob.api.BlobReferenceSource; import org.apache.james.blob.api.BlobStoreCacheCallback; +import org.apache.james.blob.compaction.BlobIdRepairer; +import org.apache.james.blob.compaction.BlobIdUpdater; +import org.apache.james.blob.compaction.BlobReferenceMappingSource; import org.apache.james.events.EventListener; import org.apache.james.eventsourcing.Event; import org.apache.james.eventsourcing.eventstore.JsonEventSerializer; @@ -76,6 +79,9 @@ import org.apache.james.mailbox.cassandra.mail.CassandraACLMapper; import org.apache.james.mailbox.cassandra.mail.CassandraApplicableFlagDAO; import org.apache.james.mailbox.cassandra.mail.CassandraAttachmentDAOV2; +import org.apache.james.mailbox.cassandra.mail.CassandraBlobIdRepairer; +import org.apache.james.mailbox.cassandra.mail.CassandraBlobIdUpdater; +import org.apache.james.mailbox.cassandra.mail.CassandraBlobReferenceMappingSource; import org.apache.james.mailbox.cassandra.mail.CassandraDeletedMessageDAO; import org.apache.james.mailbox.cassandra.mail.CassandraFirstUnseenDAO; import org.apache.james.mailbox.cassandra.mail.CassandraMailboxCounterDAO; @@ -267,6 +273,10 @@ protected void configure() { Multibinder.newSetBinder(binder(), BlobReferenceSource.class) .addBinding().to(MessageBlobReferenceSource.class); + bind(BlobIdUpdater.class).to(CassandraBlobIdUpdater.class).in(Scopes.SINGLETON); + bind(BlobIdRepairer.class).to(CassandraBlobIdRepairer.class).in(Scopes.SINGLETON); + bind(BlobReferenceMappingSource.class).to(CassandraBlobReferenceMappingSource.class).in(Scopes.SINGLETON); + Multibinder usernameChangeTaskStepMultibinder = Multibinder.newSetBinder(binder(), UsernameChangeTaskStep.class); usernameChangeTaskStepMultibinder.addBinding().to(MailboxUsernameChangeTaskStep.class); usernameChangeTaskStepMultibinder.addBinding().to(ACLUsernameChangeTaskStep.class); diff --git a/server/container/guice/distributed/pom.xml b/server/container/guice/distributed/pom.xml index 5fb89fefa0a..792d7826557 100644 --- a/server/container/guice/distributed/pom.xml +++ b/server/container/guice/distributed/pom.xml @@ -55,6 +55,10 @@ ${james.groupId} blob-api + + ${james.groupId} + blob-compaction + ${james.groupId} blob-deduplication-gc-guice diff --git a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java new file mode 100644 index 00000000000..f4202870fbb --- /dev/null +++ b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java @@ -0,0 +1,103 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.modules.blobstore; + +import java.time.Clock; +import java.util.Optional; + +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.compaction.BlobCompactionAlgorithm; +import org.apache.james.blob.compaction.BlobCompactionDTOModules; +import org.apache.james.blob.compaction.BlobIdUpdater; +import org.apache.james.blob.compaction.BlobReferenceMappingSource; +import org.apache.james.blob.compaction.CompactionConfiguration; +import org.apache.james.server.task.json.dto.AdditionalInformationDTO; +import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule; +import org.apache.james.server.task.json.dto.TaskDTO; +import org.apache.james.server.task.json.dto.TaskDTOModule; +import org.apache.james.task.Task; +import org.apache.james.task.TaskExecutionDetails; +import org.apache.james.webadmin.dto.DTOModuleInjections; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.inject.AbstractModule; +import com.google.inject.Injector; +import com.google.inject.Key; +import com.google.inject.Provides; +import com.google.inject.Singleton; +import com.google.inject.multibindings.ProvidesIntoSet; +import com.google.inject.name.Named; + +public class BlobCompactionModule extends AbstractModule { + + @Provides + @Singleton + public CompactionConfiguration compactionConfiguration() { + return CompactionConfiguration.DEFAULT; + } + + @Provides + @Singleton + public BlobCompactionAlgorithm blobCompactionAlgorithm(BlobStoreDAO blobStoreDAO, + @Named(BlobStoreModulesChooser.RAW) BlobStoreDAO rawStore, + BlobReferenceMappingSource mappingSource, + BlobIdUpdater blobIdUpdater) { + return new BlobCompactionAlgorithm(blobStoreDAO, rawStore, mappingSource, blobIdUpdater); + } + + private static final Logger LOGGER = LoggerFactory.getLogger(BlobCompactionModule.class); + + @Provides + @Singleton + public Optional optionalBlobCompactionAlgorithm(Injector injector) { + if (injector.getExistingBinding(Key.get(BlobStoreConfiguration.class)) != null) { + BlobStoreConfiguration config = injector.getInstance(BlobStoreConfiguration.class); + if (config.getCryptoConfig().isPresent()) { + LOGGER.warn("Blob compaction is disabled because client-side encryption is enabled in BlobStoreConfiguration"); + return Optional.empty(); + } + if (config.getCompressionConfiguration().enabled()) { + LOGGER.warn("Blob compaction is disabled because blob compression is enabled in BlobStoreConfiguration"); + return Optional.empty(); + } + } + if (injector.getExistingBinding(Key.get(BlobCompactionAlgorithm.class)) != null) { + return Optional.of(injector.getInstance(BlobCompactionAlgorithm.class)); + } + return Optional.empty(); + } + + @ProvidesIntoSet + public TaskDTOModule blobCompactionTask(BlobCompactionAlgorithm algorithm, Clock clock) { + return BlobCompactionDTOModules.taskModule(algorithm, clock); + } + + @ProvidesIntoSet + public AdditionalInformationDTOModule blobCompactionAdditionalInformation() { + return BlobCompactionDTOModules.additionalInformationModule(); + } + + @Named(DTOModuleInjections.WEBADMIN_DTO) + @ProvidesIntoSet + public AdditionalInformationDTOModule webAdminBlobCompactionAdditionalInformation() { + return BlobCompactionDTOModules.additionalInformationModule(); + } +} diff --git a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java index a51b3f964b0..fcfed410782 100644 --- a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java +++ b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java @@ -31,6 +31,8 @@ import org.apache.james.blob.api.ObjectStorageHealthCheck; import org.apache.james.blob.cassandra.CassandraBlobStoreDAO; import org.apache.james.blob.cassandra.cache.CachedBlobStore; +import org.apache.james.blob.compaction.BlobIdRepairer; +import org.apache.james.blob.compaction.ChunkedBlobStoreDAO; import org.apache.james.blob.file.FileBlobStoreDAO; import org.apache.james.blob.objectstorage.aws.S3BlobStoreConfiguration; import org.apache.james.blob.objectstorage.aws.S3BlobStoreDAO; @@ -57,6 +59,8 @@ import com.google.common.collect.ImmutableList; import com.google.inject.AbstractModule; +import com.google.inject.Injector; +import com.google.inject.Key; import com.google.inject.Module; import com.google.inject.Provides; import com.google.inject.Scopes; @@ -68,8 +72,9 @@ import modules.BlobPostgresModule; public class BlobStoreModulesChooser { - private static final String RAW = "raw"; - private static final String ENCRYPTION = "encryption"; + public static final String RAW = "raw"; + public static final String ENCRYPTION = "encryption"; + public static final String PLAIN_CHAIN = "plain-chain"; static class CassandraBlobStoreDAODeclarationModule extends AbstractModule { @Override @@ -133,6 +138,7 @@ protected void configure() { static class NoCompressionModule extends AbstractModule { @Provides @Singleton + @Named(PLAIN_CHAIN) BlobStoreDAO blobStoreDAO(@Named(ENCRYPTION) BlobStoreDAO encryption) { return encryption; } @@ -147,6 +153,7 @@ static class CompressionModule extends AbstractModule { @Provides @Singleton + @Named(PLAIN_CHAIN) BlobStoreDAO blobStoreDAO(@Named(ENCRYPTION) BlobStoreDAO encryption, MetricFactory metricFactory) { return new ZstdBlobStoreDAO(encryption, compressionConfiguration, metricFactory); } @@ -157,6 +164,19 @@ CompressionConfiguration compressionConfiguration() { } } + static class ChunkedBlobStoreModule extends AbstractModule { + @Provides + @Singleton + BlobStoreDAO blobStoreDAO(@Named(PLAIN_CHAIN) BlobStoreDAO plainChain, + @Named(RAW) BlobStoreDAO raw, + Injector injector) { + Optional blobIdRepairer = Optional.ofNullable( + injector.getExistingBinding(Key.get(BlobIdRepairer.class))) + .map(binding -> binding.getProvider().get()); + return new ChunkedBlobStoreDAO(plainChain, raw, blobIdRepairer); + } + } + static class NoEncryptionModule extends AbstractModule { @Provides @Singleton @@ -191,6 +211,8 @@ public static List chooseModules(BlobStoreConfiguration choosingConfigur .add(chooseBlobStoreDAOModule(choosingConfiguration.getImplementation())) .add(chooseEncryptionModule(choosingConfiguration.getCryptoConfig())) .add(chooseCompressionModule(choosingConfiguration.getCompressionConfiguration())) + .add(new ChunkedBlobStoreModule()) + .add(new BlobCompactionModule()) .addAll(chooseStoragePolicyModule(choosingConfiguration.storageStrategy())) .add(new StoragePolicyConfigurationSanityEnforcementModule()) .add(binder -> binder.bind(BlobStoreConfiguration.class).toInstance(choosingConfiguration)) diff --git a/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java b/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java index 53040161e26..60072dea9d9 100644 --- a/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java +++ b/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java @@ -21,10 +21,23 @@ import static org.assertj.core.api.Assertions.assertThat; +import java.time.Clock; +import java.util.Optional; + import org.apache.james.blob.aes.CryptoConfig; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.compaction.BlobCompactionAlgorithm; +import org.apache.james.blob.compaction.BlobIdUpdater; +import org.apache.james.blob.compaction.BlobReferenceMappingSource; import org.apache.james.blob.zstd.CompressionConfiguration; import org.junit.jupiter.api.Test; +import com.google.inject.Guice; +import com.google.inject.Injector; +import com.google.inject.Key; +import com.google.inject.TypeLiteral; +import com.google.inject.name.Names; + class BlobStoreModulesChooserTest { @Test @@ -99,4 +112,71 @@ void provideBlobStoreShouldReturnCompressionWhenConfigured() { .filteredOn(module -> module instanceof BlobStoreModulesChooser.CompressionModule) .hasSize(1); } + + @Test + void provideBlobStoreShouldReturnBlobCompactionModule() { + assertThat(BlobStoreModulesChooser.chooseModules(BlobStoreConfiguration.builder() + .s3() + .disableCache() + .deduplication() + .noCryptoConfig())) + .filteredOn(module -> module instanceof BlobCompactionModule) + .hasSize(1); + } + + @Test + void optionalBlobCompactionAlgorithmShouldReturnEmptyWhenCryptoConfigured() { + BlobStoreConfiguration config = BlobStoreConfiguration.builder() + .cassandra() + .disableCache() + .passthrough() + .cryptoConfig(CryptoConfig.builder() + .password("myPass".toCharArray()) + .salt("73616c7479") + .build()); + + Injector injector = Guice.createInjector( + binder -> { + binder.bind(BlobStoreConfiguration.class).toInstance(config); + binder.bind(Clock.class).toInstance(Clock.systemUTC()); + binder.bind(BlobStoreDAO.class).toProvider(() -> null); + binder.bind(BlobStoreDAO.class).annotatedWith(Names.named(BlobStoreModulesChooser.RAW)).toProvider(() -> null); + binder.bind(BlobIdUpdater.class).toProvider(() -> null); + binder.bind(BlobReferenceMappingSource.class).toProvider(() -> null); + }, + new BlobCompactionModule() + ); + + Optional algorithm = injector.getInstance( + Key.get(new TypeLiteral>() {})); + assertThat(algorithm).isEmpty(); + } + + @Test + void optionalBlobCompactionAlgorithmShouldReturnEmptyWhenCompressionConfigured() { + BlobStoreConfiguration config = BlobStoreConfiguration.builder() + .cassandra() + .disableCache() + .passthrough() + .noCryptoConfig() + .compressionConfig(CompressionConfiguration.builder() + .enabled(true) + .build()); + + Injector injector = Guice.createInjector( + binder -> { + binder.bind(BlobStoreConfiguration.class).toInstance(config); + binder.bind(Clock.class).toInstance(Clock.systemUTC()); + binder.bind(BlobStoreDAO.class).toProvider(() -> null); + binder.bind(BlobStoreDAO.class).annotatedWith(Names.named(BlobStoreModulesChooser.RAW)).toProvider(() -> null); + binder.bind(BlobIdUpdater.class).toProvider(() -> null); + binder.bind(BlobReferenceMappingSource.class).toProvider(() -> null); + }, + new BlobCompactionModule() + ); + + Optional algorithm = injector.getInstance( + Key.get(new TypeLiteral>() {})); + assertThat(algorithm).isEmpty(); + } } \ No newline at end of file diff --git a/server/protocols/webadmin/webadmin-data/pom.xml b/server/protocols/webadmin/webadmin-data/pom.xml index eef654b4a43..1eb35db5fef 100644 --- a/server/protocols/webadmin/webadmin-data/pom.xml +++ b/server/protocols/webadmin/webadmin-data/pom.xml @@ -32,6 +32,10 @@ Apache James :: Server :: Web Admin :: data + + ${james.groupId} + blob-compaction + ${james.groupId} blob-memory diff --git a/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java b/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java index ff37a2aa715..97f53ab5cfc 100644 --- a/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java +++ b/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java @@ -31,6 +31,9 @@ import org.apache.james.blob.api.BlobStore; import org.apache.james.blob.api.BlobStoreDAO; import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.compaction.BlobCompactionAlgorithm; +import org.apache.james.blob.compaction.BlobCompactionTask; +import org.apache.james.blob.compaction.CompactionRequest; import org.apache.james.server.blob.deduplication.BlobGCTask; import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; import org.apache.james.task.Task; @@ -58,8 +61,8 @@ public class BlobRoutes implements Routes { private final Set blobReferenceSources; private final GenerationAwareBlobId.Configuration generationAwareBlobIdConfiguration; private final BlobId.Factory generationAwareBlobIdFactory; + private final Optional blobCompactionAlgorithm; - @Inject public BlobRoutes(TaskManager taskManager, JsonTransformer jsonTransformer, Clock clock, @@ -68,6 +71,20 @@ public BlobRoutes(TaskManager taskManager, Set blobReferenceSources, GenerationAwareBlobId.Configuration generationAwareBlobIdConfiguration, BlobId.Factory generationAwareBlobIdFactory) { + this(taskManager, jsonTransformer, clock, blobStoreDAO, defaultBucketName, blobReferenceSources, + generationAwareBlobIdConfiguration, generationAwareBlobIdFactory, Optional.empty()); + } + + @Inject + public BlobRoutes(TaskManager taskManager, + JsonTransformer jsonTransformer, + Clock clock, + BlobStoreDAO blobStoreDAO, + @Named(BlobStore.DEFAULT_BUCKET_NAME_QUALIFIER) BucketName defaultBucketName, + Set blobReferenceSources, + GenerationAwareBlobId.Configuration generationAwareBlobIdConfiguration, + BlobId.Factory generationAwareBlobIdFactory, + Optional blobCompactionAlgorithm) { this.taskManager = taskManager; this.jsonTransformer = jsonTransformer; this.clock = clock; @@ -76,6 +93,7 @@ public BlobRoutes(TaskManager taskManager, this.blobReferenceSources = blobReferenceSources; this.generationAwareBlobIdConfiguration = generationAwareBlobIdConfiguration; this.generationAwareBlobIdFactory = generationAwareBlobIdFactory; + this.blobCompactionAlgorithm = blobCompactionAlgorithm; } @Override @@ -85,8 +103,54 @@ public String getBasePath() { @Override public void define(Service service) { - TaskFromRequest gcUnreferencedTaskRequest = this::gcUnreferenced; - service.delete(BASE_PATH, gcUnreferencedTaskRequest.asRoute(taskManager), jsonTransformer); + TaskFromRequest deleteTaskRequest = this::delete; + service.delete(BASE_PATH, deleteTaskRequest.asRoute(taskManager), jsonTransformer); + } + + public Task delete(Request request) { + String scope = request.queryParams("scope"); + if ("unreferenced".equals(scope)) { + return gcUnreferenced(request); + } + if ("compaction".equals(scope)) { + return compact(request); + } + throw new IllegalArgumentException("'scope' is missing or must be 'unreferenced'"); + } + + public Task compact(Request request) { + String generationParam = request.queryParams("generation"); + Preconditions.checkArgument(generationParam != null && !generationParam.isBlank(), + "'generation' is compulsory"); + long generation; + try { + generation = Long.parseLong(generationParam); + } catch (NumberFormatException e) { + throw new IllegalArgumentException("Invalid 'generation': " + generationParam, e); + } + Preconditions.checkArgument(generation >= 0, "'generation' must not be negative"); + + Optional family = Optional.ofNullable(request.queryParams("family")) + .map(val -> { + try { + int parsed = Integer.parseInt(val); + Preconditions.checkArgument(parsed > 0, "'family' must be strictly positive"); + return parsed; + } catch (NumberFormatException e) { + throw new IllegalArgumentException("Invalid 'family': " + val, e); + } + }); + + BlobCompactionAlgorithm algorithm = blobCompactionAlgorithm + .orElseThrow(() -> new IllegalStateException("Blob compaction is not configured on this server")); + + CompactionRequest compactionRequest = CompactionRequest.builder() + .bucketName(bucketName) + .generation(generation) + .family(family) + .build(); + + return new BlobCompactionTask(algorithm, compactionRequest, clock); } public Task gcUnreferenced(Request request) { diff --git a/server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java b/server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java new file mode 100644 index 00000000000..097fac5528e --- /dev/null +++ b/server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java @@ -0,0 +1,204 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.webadmin.routes; + +import static io.restassured.RestAssured.given; +import static io.restassured.http.ContentType.JSON; +import static org.eclipse.jetty.http.HttpStatus.BAD_REQUEST_400; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.notNullValue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import java.time.ZonedDateTime; +import java.util.Optional; + +import org.apache.james.blob.api.BlobReferenceSource; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.blob.compaction.BlobCompactionAlgorithm; +import org.apache.james.blob.compaction.BlobCompactionDTOModules; +import org.apache.james.blob.compaction.CompactionResult; +import org.apache.james.blob.memory.MemoryBlobStoreDAO; +import org.apache.james.json.DTOConverter; +import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; +import org.apache.james.task.Hostname; +import org.apache.james.task.MemoryTaskManager; +import org.apache.james.utils.UpdatableTickingClock; +import org.apache.james.webadmin.WebAdminServer; +import org.apache.james.webadmin.WebAdminUtils; +import org.apache.james.webadmin.utils.JsonTransformer; +import org.eclipse.jetty.http.HttpStatus; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import com.google.common.collect.ImmutableSet; + +import io.restassured.RestAssured; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +class BlobRoutesCompactionTest { + private static final String BASE_PATH = "/blobs"; + private static final PlainBlobId.Factory BLOB_ID_FACTORY = new PlainBlobId.Factory(); + private static final ZonedDateTime TIMESTAMP = ZonedDateTime.parse("2024-01-01T00:00:00Z"); + private static final BucketName DEFAULT_BUCKET = BucketName.of("default"); + private static final GenerationAwareBlobId.Configuration GENERATION_AWARE_BLOB_ID_CONFIGURATION = GenerationAwareBlobId.Configuration.DEFAULT; + + private WebAdminServer webAdminServer; + private MemoryTaskManager taskManager; + private BlobCompactionAlgorithm compactionAlgorithm; + + @BeforeEach + void setUp() { + taskManager = new MemoryTaskManager(new Hostname("foo")); + UpdatableTickingClock clock = new UpdatableTickingClock(TIMESTAMP.toInstant()); + BlobReferenceSource blobReferenceSource = mock(BlobReferenceSource.class); + when(blobReferenceSource.listReferencedBlobs()).thenReturn(Flux.empty()); + + GenerationAwareBlobId.Factory generationAwareBlobIdFactory = new GenerationAwareBlobId.Factory(clock, BLOB_ID_FACTORY, GENERATION_AWARE_BLOB_ID_CONFIGURATION); + BlobStoreDAO blobStoreDAO = new MemoryBlobStoreDAO(); + JsonTransformer jsonTransformer = new JsonTransformer(); + TasksRoutes tasksRoutes = new TasksRoutes(taskManager, jsonTransformer, DTOConverter.of(BlobCompactionDTOModules.additionalInformationModule())); + + compactionAlgorithm = mock(BlobCompactionAlgorithm.class); + when(compactionAlgorithm.compact(any())).thenReturn(Mono.just(CompactionResult.NONE)); + + BlobRoutes blobRoutes = new BlobRoutes( + taskManager, + jsonTransformer, + clock, + blobStoreDAO, + DEFAULT_BUCKET, + ImmutableSet.of(blobReferenceSource), + GENERATION_AWARE_BLOB_ID_CONFIGURATION, + generationAwareBlobIdFactory, + Optional.of(compactionAlgorithm)); + + webAdminServer = WebAdminUtils.createWebAdminServer(blobRoutes, tasksRoutes).start(); + + RestAssured.requestSpecification = WebAdminUtils.buildRequestSpecification(webAdminServer) + .setBasePath(BASE_PATH) + .build(); + } + + @AfterEach + void tearDown() { + webAdminServer.destroy(); + taskManager.stop(); + } + + @Test + void deleteCompactionShouldReturnTaskIdWhenValidParameters() { + given() + .queryParam("scope", "compaction") + .queryParam("generation", "2") + .queryParam("family", "1") + .delete() + .then() + .statusCode(HttpStatus.CREATED_201) + .body("taskId", notNullValue()); + } + + @Test + void deleteCompactionShouldWorkWithoutFamily() { + given() + .queryParam("scope", "compaction") + .queryParam("generation", "0") + .delete() + .then() + .statusCode(HttpStatus.CREATED_201) + .body("taskId", notNullValue()); + } + + @Test + void deleteCompactionShouldReturnErrorWhenMissingGeneration() { + given() + .queryParam("scope", "compaction") + .delete() + .then() + .statusCode(BAD_REQUEST_400) + .contentType(JSON) + .body("statusCode", is(BAD_REQUEST_400)) + .body("type", is("InvalidArgument")) + .body("details", is("'generation' is compulsory")); + } + + @Test + void deleteCompactionShouldReturnErrorWhenInvalidGeneration() { + given() + .queryParam("scope", "compaction") + .queryParam("generation", "abc") + .delete() + .then() + .statusCode(BAD_REQUEST_400) + .contentType(JSON) + .body("statusCode", is(BAD_REQUEST_400)) + .body("type", is("InvalidArgument")) + .body("details", is("Invalid 'generation': abc")); + } + + @Test + void deleteCompactionShouldReturnErrorWhenNegativeGeneration() { + given() + .queryParam("scope", "compaction") + .queryParam("generation", "-1") + .delete() + .then() + .statusCode(BAD_REQUEST_400) + .contentType(JSON) + .body("statusCode", is(BAD_REQUEST_400)) + .body("type", is("InvalidArgument")) + .body("details", is("'generation' must not be negative")); + } + + @Test + void deleteCompactionShouldReturnErrorWhenInvalidFamily() { + given() + .queryParam("scope", "compaction") + .queryParam("generation", "1") + .queryParam("family", "abc") + .delete() + .then() + .statusCode(BAD_REQUEST_400) + .contentType(JSON) + .body("statusCode", is(BAD_REQUEST_400)) + .body("type", is("InvalidArgument")) + .body("details", is("Invalid 'family': abc")); + } + + @Test + void deleteCompactionShouldReturnErrorWhenNonPositiveFamily() { + given() + .queryParam("scope", "compaction") + .queryParam("generation", "1") + .queryParam("family", "0") + .delete() + .then() + .statusCode(BAD_REQUEST_400) + .contentType(JSON) + .body("statusCode", is(BAD_REQUEST_400)) + .body("type", is("InvalidArgument")) + .body("details", is("'family' must be strictly positive")); + } +} From 3898500eab579eef0bcd63de4fdc00190181f0aa Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 10:49:46 +0530 Subject: [PATCH 07/24] JAMES-4231 Blob API: Return Mono in readRange - Update BlobStoreDAO.readRange to return Mono instead of RangeByteSlice. - Store totalObjectSize in BlobMetadata (TOTAL_OBJECT_SIZE) and provide BlobStoreDAO.totalObjectSize(Blob) utility method. - Update S3BlobStoreDAO and ChunkedBlobStoreDAO to return Mono with metadata. - Update contract and unit test suites to use Blob. --- .../apache/james/blob/api/BlobStoreDAO.java | 53 +++++++++---------- .../api/ReadSaveBlobStoreDAOContract.java | 48 ++++++++--------- .../compaction/BlobCompactionAlgorithm.java | 37 +++++++------ .../blob/compaction/ChunkedBlobStoreDAO.java | 38 +++++++------ .../BlobCompactionAlgorithmTest.java | 8 +-- .../compaction/ChunkedBlobStoreDAOTest.java | 2 +- .../objectstorage/aws/S3BlobStoreDAO.java | 17 ++++-- .../aws/S3BlobStoreDAORangeReadTest.java | 39 +++++++------- 8 files changed, 131 insertions(+), 111 deletions(-) diff --git a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java index be5d6d2899e..f41a053e31e 100644 --- a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java +++ b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStoreDAO.java @@ -305,51 +305,48 @@ default Publisher listBlobs(BucketName bucketName, String prefix) { .filter(blobId -> blobId.asString().startsWith(prefix)); } - record RangeByteSlice(byte[] data, long totalObjectSize) { - public static RangeByteSlice of(byte[] data, long totalObjectSize) { - return new RangeByteSlice(data, totalObjectSize); - } - - @Override - public boolean equals(Object o) { - if (this == o) { - return true; - } - if (o == null || getClass() != o.getClass()) { - return false; - } - RangeByteSlice that = (RangeByteSlice) o; - return totalObjectSize == that.totalObjectSize && Arrays.equals(data, that.data); - } - - @Override - public int hashCode() { - int result = Objects.hash(totalObjectSize); - result = 31 * result + Arrays.hashCode(data); - return result; - } + BlobMetadataName TOTAL_OBJECT_SIZE = new BlobMetadataName("total-object-size"); + + static long totalObjectSize(Blob blob) { + return blob.metadata().get(TOTAL_OBJECT_SIZE) + .map(val -> { + try { + return Long.parseLong(val.value()); + } catch (NumberFormatException e) { + return 0L; + } + }) + .orElseGet(() -> { + try { + return (long) blob.asBytes().payload().length; + } catch (IOException e) { + return 0L; + } + }); } - default Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + default Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { return Mono.from(readBytes(bucketName, blobId)) .map(bytesBlob -> { byte[] allBytes = bytesBlob.payload(); long totalSize = allBytes.length; + BlobMetadata metadata = bytesBlob.metadata() + .withMetadata(TOTAL_OBJECT_SIZE, new BlobMetadataValue(String.valueOf(totalSize))); if (start < 0) { int suffixLength = (int) Math.min(totalSize, -start); int from = (int) (totalSize - suffixLength); byte[] slice = Arrays.copyOfRange(allBytes, from, (int) totalSize); - return RangeByteSlice.of(slice, totalSize); + return (Blob) BytesBlob.of(slice, metadata); } if (start >= totalSize) { - return RangeByteSlice.of(new byte[0], totalSize); + return (Blob) BytesBlob.of(new byte[0], metadata); } long boundedEnd = Math.min(end, totalSize - 1); if (boundedEnd < start) { - return RangeByteSlice.of(new byte[0], totalSize); + return (Blob) BytesBlob.of(new byte[0], metadata); } byte[] slice = Arrays.copyOfRange(allBytes, (int) start, (int) boundedEnd + 1); - return RangeByteSlice.of(slice, totalSize); + return (Blob) BytesBlob.of(slice, metadata); }); } } diff --git a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java index 9d5340baee2..20c40120ade 100644 --- a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java +++ b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/ReadSaveBlobStoreDAOContract.java @@ -407,68 +407,68 @@ public int read(byte[] b, int off, int len) throws IOException { } @Test - default void readRangeShouldReturnMiddleSlice() { + default void readRangeShouldReturnMiddleSlice() throws IOException { byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); - BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 3, 6).block(); + BlobStoreDAO.Blob slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 3, 6).block(); - assertThat(slice.totalObjectSize()).isEqualTo(10); - assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("3456"); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(10); + assertThat(new String(slice.asBytes().payload(), StandardCharsets.UTF_8)).isEqualTo("3456"); } @Test - default void readRangeShouldReturnPrefix() { + default void readRangeShouldReturnPrefix() throws IOException { byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); - BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 0, 2).block(); + BlobStoreDAO.Blob slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 0, 2).block(); - assertThat(slice.totalObjectSize()).isEqualTo(10); - assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("012"); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(10); + assertThat(new String(slice.asBytes().payload(), StandardCharsets.UTF_8)).isEqualTo("012"); } @Test - default void readRangeShouldReturnSuffixWhenStartNegative() { + default void readRangeShouldReturnSuffixWhenStartNegative() throws IOException { byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); - BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, -4, -1).block(); + BlobStoreDAO.Blob slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, -4, -1).block(); - assertThat(slice.totalObjectSize()).isEqualTo(10); - assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("6789"); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(10); + assertThat(new String(slice.asBytes().payload(), StandardCharsets.UTF_8)).isEqualTo("6789"); } @Test - default void readRangeShouldReturnFullContent() { + default void readRangeShouldReturnFullContent() throws IOException { byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); - BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 0, 9).block(); + BlobStoreDAO.Blob slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 0, 9).block(); - assertThat(slice.totalObjectSize()).isEqualTo(10); - assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("0123456789"); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(10); + assertThat(new String(slice.asBytes().payload(), StandardCharsets.UTF_8)).isEqualTo("0123456789"); } @Test - default void readRangeShouldCapEndAtObjectSize() { + default void readRangeShouldCapEndAtObjectSize() throws IOException { byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); - BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 5, 50).block(); + BlobStoreDAO.Blob slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 5, 50).block(); - assertThat(slice.totalObjectSize()).isEqualTo(10); - assertThat(new String(slice.data(), StandardCharsets.UTF_8)).isEqualTo("56789"); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(10); + assertThat(new String(slice.asBytes().payload(), StandardCharsets.UTF_8)).isEqualTo("56789"); } @Test - default void readRangeShouldReturnEmptyWhenStartBeyondSize() { + default void readRangeShouldReturnEmptyWhenStartBeyondSize() throws IOException { byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8); Mono.from(testee().save(TEST_BUCKET_NAME, TEST_BLOB_ID, BlobStoreDAO.BytesBlob.of(payload))).block(); - BlobStoreDAO.RangeByteSlice slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 20, 30).block(); + BlobStoreDAO.Blob slice = testee().readRange(TEST_BUCKET_NAME, TEST_BLOB_ID, 20, 30).block(); - assertThat(slice.totalObjectSize()).isEqualTo(10); - assertThat(slice.data()).isEmpty(); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(10); + assertThat(slice.asBytes().payload()).isEmpty(); } } diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index 7bd1da8e244..7c747deb2ff 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -184,11 +184,13 @@ public Mono gcCompact(CompactionRequest request) { .filter(blobId -> ChunkId.isChunkRef(blobId) && blobId.asString().indexOf('~') == -1) .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) .flatMap(chunkBlobId -> Mono.from(rawStore.readRange(request.bucketName(), chunkBlobId, -65536, -1)) - .map(tailSlice -> { + .map(tailBlob -> { try { - ChunkFooter footer = ChunkFormat.readFooter(tailSlice.data(), tailSlice.totalObjectSize()); - return new ExistingChunk(chunkBlobId, tailSlice.totalObjectSize(), footer); - } catch (ObjectStoreIOException e) { + byte[] tailData = tailBlob.asBytes().payload(); + long totalSize = BlobStoreDAO.totalObjectSize(tailBlob); + ChunkFooter footer = ChunkFormat.readFooter(tailData, totalSize); + return new ExistingChunk(chunkBlobId, totalSize, footer); + } catch (IOException e) { LOGGER.warn("Failed reading footer for chunk object {}", chunkBlobId.asString(), e); return null; } @@ -343,12 +345,13 @@ private Mono purgeDeadSlots(CompactionRequest request, return Flux.fromIterable(analysis.liveSlots) .concatMap(liveSlot -> Mono.from(rawStore.readRange(request.bucketName(), analysis.chunk.chunkBlobId, liveSlot.offset, liveSlot.offset + liveSlot.limit - 1)) .publishOn(Schedulers.parallel()) - .flatMap(slice -> { + .flatMap(sliceBlob -> { try { - byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), liveSlot.offset); + byte[] sliceData = sliceBlob.asBytes().payload(); + byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, liveSlot.offset); return Mono.just(new LiveSlotWithContent(liveSlot, decompressed)); - } catch (ObjectStoreIOException e) { - return Mono.error(e); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Failed parsing slot", e)); } })) .collectList() @@ -416,12 +419,13 @@ private Mono mergePair(CompactionRequest request, ChunkPair pa Mono> c1Live = Flux.fromIterable(pair.c1.liveSlots) .concatMap(slot -> Mono.from(rawStore.readRange(request.bucketName(), pair.c1.chunk.chunkBlobId, slot.offset, slot.offset + slot.limit - 1)) .publishOn(Schedulers.parallel()) - .flatMap(slice -> { + .flatMap(sliceBlob -> { try { - byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), slot.offset); + byte[] sliceData = sliceBlob.asBytes().payload(); + byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, slot.offset); return Mono.just(new LiveSlotWithContent(slot, decompressed)); - } catch (ObjectStoreIOException e) { - return Mono.error(e); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Failed parsing slot", e)); } })) .collectList(); @@ -429,12 +433,13 @@ private Mono mergePair(CompactionRequest request, ChunkPair pa Mono> c2Live = Flux.fromIterable(pair.c2.liveSlots) .concatMap(slot -> Mono.from(rawStore.readRange(request.bucketName(), pair.c2.chunk.chunkBlobId, slot.offset, slot.offset + slot.limit - 1)) .publishOn(Schedulers.parallel()) - .flatMap(slice -> { + .flatMap(sliceBlob -> { try { - byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), slot.offset); + byte[] sliceData = sliceBlob.asBytes().payload(); + byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, slot.offset); return Mono.just(new LiveSlotWithContent(slot, decompressed)); - } catch (ObjectStoreIOException e) { - return Mono.error(e); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Failed parsing slot", e)); } })) .collectList(); diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java index d22bccc0c06..f5e33df57fb 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java @@ -20,6 +20,7 @@ package org.apache.james.blob.compaction; import java.io.ByteArrayInputStream; +import java.io.IOException; import java.util.Collection; import java.util.List; import java.util.Optional; @@ -118,23 +119,27 @@ private Mono readChunkSlot(BucketName bucketName, ChunkId slotRef) { BlobId chunkObjectBlobId = slotRef.chunkBlobId(); return rawStore.readRange(bucketName, chunkObjectBlobId, offset, offset + limit - 1) .publishOn(Schedulers.parallel()) - .flatMap(rangeSlice -> { - if (rangeSlice.data().length == 0 || offset >= rangeSlice.totalObjectSize()) { - return Mono.error(new ObjectNotFoundException("Slot not found at offset " + offset + " in chunk " + chunkObjectBlobId.asString())); - } + .flatMap(rangeBlob -> { try { - byte[] decompressed = ChunkFormat.parseSlotBytes(rangeSlice.data(), offset); + byte[] rangeData = rangeBlob.asBytes().payload(); + long totalSize = BlobStoreDAO.totalObjectSize(rangeBlob); + if (rangeData.length == 0 || offset >= totalSize) { + return Mono.error(new ObjectNotFoundException("Slot not found at offset " + offset + " in chunk " + chunkObjectBlobId.asString())); + } + byte[] decompressed = ChunkFormat.parseSlotBytes(rangeData, offset); return Mono.just(BytesBlob.of(decompressed)); - } catch (ObjectStoreIOException e) { - return Mono.error(e); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Error reading slot for chunk " + chunkObjectBlobId.asString(), e)); } }); } else { BlobId chunkObjectBlobId = slotRef.chunkBlobId(); return rawStore.readRange(bucketName, chunkObjectBlobId, -65536, -1) - .flatMap(tailSlice -> { + .flatMap(tailBlob -> { try { - ChunkFooter footer = ChunkFormat.readFooter(tailSlice.data(), tailSlice.totalObjectSize()); + byte[] tailData = tailBlob.asBytes().payload(); + long totalSize = BlobStoreDAO.totalObjectSize(tailBlob); + ChunkFooter footer = ChunkFormat.readFooter(tailData, totalSize); if (footer.slotCount() == 0) { return Mono.just(BytesBlob.of(new byte[0])); } @@ -146,16 +151,17 @@ private Mono readChunkSlot(BucketName bucketName, ChunkId slotRef) { long length = footer.slotLength(slotIndex); return rawStore.readRange(bucketName, chunkObjectBlobId, start, start + length - 1) .publishOn(Schedulers.parallel()) - .flatMap(slice -> { + .flatMap(sliceBlob -> { try { - byte[] decompressed = ChunkFormat.parseSlotBytes(slice.data(), start); + byte[] sliceData = sliceBlob.asBytes().payload(); + byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, start); return Mono.just(BytesBlob.of(decompressed)); - } catch (ObjectStoreIOException e) { - return Mono.error(e); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Error reading slot at start " + start, e)); } }); - } catch (ObjectStoreIOException e) { - return Mono.error(e); + } catch (IOException e) { + return Mono.error(new ObjectStoreIOException("Error reading chunk footer for " + chunkObjectBlobId.asString(), e)); } }); } @@ -236,7 +242,7 @@ public Publisher listBlobs(BucketName bucketName, String prefix) { } @Override - public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { if (ChunkId.isChunkRef(blobId)) { ChunkId chunkId = ChunkId.parseChunkOrSlotRef(blobId.asString()); return rawStore.readRange(bucketName, chunkId.chunkBlobId(), start, end); diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index 9dcc57e1ee2..922614b5a4d 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -438,7 +438,7 @@ public Mono save(BucketName bucketName, BlobId blobId, BytesBlob blob) { } @Override - public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { byte[] allBytes = rawBlobs.get(blobId); if (allBytes == null) { return Mono.error(new ObjectNotFoundException("Blob not found: " + blobId.asString())); @@ -448,7 +448,7 @@ public Mono readRange(BucketName bucketName, BlobId blobId, long int to; if (start < 0) { int suffixLength = (int) Math.min(totalSize, -start); - from = (int) (totalSize - suffixLength); + from = (int) Math.max(0, totalSize - suffixLength); to = (int) totalSize; } else { from = (int) Math.min(totalSize, start); @@ -456,7 +456,9 @@ public Mono readRange(BucketName bucketName, BlobId blobId, long } byte[] slice = Arrays.copyOfRange(allBytes, from, to); maxRangedReadBytes.updateAndGet(curr -> Math.max(curr, slice.length)); - return Mono.just(RangeByteSlice.of(slice, totalSize)); + BlobMetadata metadata = BlobMetadata.empty() + .withMetadata(TOTAL_OBJECT_SIZE, new BlobMetadataValue(String.valueOf(totalSize))); + return Mono.just(BytesBlob.of(slice, metadata)); } @Override diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java index d29d68a796d..07252827e97 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java @@ -72,7 +72,7 @@ void resetCount() { } @Override - public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { readRangeCallCount.incrementAndGet(); return delegate.readRange(bucketName, blobId, start, end); } diff --git a/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java b/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java index a0eede2a84d..30e774702a8 100644 --- a/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java +++ b/server/blob/blob-s3/src/main/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAO.java @@ -38,7 +38,9 @@ import org.apache.commons.io.IOUtils; import org.apache.james.blob.api.BlobId; import org.apache.james.blob.api.BlobStoreDAO; -import org.apache.james.blob.api.BlobStoreDAO.RangeByteSlice; +import org.apache.james.blob.api.BlobStoreDAO.Blob; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadata; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadataValue; import org.apache.james.blob.api.BucketName; import org.apache.james.blob.api.ObjectNotFoundException; import org.apache.james.blob.api.ObjectStoreIOException; @@ -249,7 +251,7 @@ public Publisher readBytes(BucketName bucketName, BlobId blobId) { } @Override - public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { + public Mono readRange(BucketName bucketName, BlobId blobId, long start, long end) { BucketName resolvedBucketName = bucketNameResolver.resolve(bucketName); String rangeHeader = formatRangeHeader(start, end); @@ -259,11 +261,18 @@ public Mono readRange(BucketName bucketName, BlobId blobId, long .publishOn(Schedulers.parallel()) .map(responseBytes -> { long totalObjectSize = extractTotalObjectSize(responseBytes.response()); - return RangeByteSlice.of(responseBytes.asByteArrayUnsafe(), totalObjectSize); + BlobMetadata metadata = asBlobMetadata(responseBytes.response().metadata()) + .withMetadata(TOTAL_OBJECT_SIZE, new BlobMetadataValue(String.valueOf(totalObjectSize))); + return (Blob) BytesBlob.of(responseBytes.asByteArrayUnsafe(), metadata); }) .onErrorResume(this::isRangeNotSatisfiable, e -> headObject(resolvedBucketName, blobId) - .map(head -> RangeByteSlice.of(new byte[0], head.contentLength() != null ? head.contentLength() : 0L)) + .map(head -> { + long totalSize = head.contentLength() != null ? head.contentLength() : 0L; + BlobMetadata metadata = asBlobMetadata(head.metadata()) + .withMetadata(TOTAL_OBJECT_SIZE, new BlobMetadataValue(String.valueOf(totalSize))); + return (Blob) BytesBlob.of(new byte[0], metadata); + }) .onErrorMap(NoSuchBucketException.class, ex -> new ObjectNotFoundException("Bucket not found " + resolvedBucketName.asString(), ex)) .onErrorMap(NoSuchKeyException.class, ex -> new ObjectNotFoundException("Blob not found " + blobId.asString() + " in bucket " + resolvedBucketName.asString(), ex))) .onErrorMap(e -> e.getCause() instanceof OutOfMemoryError, Throwable::getCause); diff --git a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java index 2503b63653b..42ac6cde8de 100644 --- a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java +++ b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAORangeReadTest.java @@ -24,6 +24,7 @@ import static org.apache.james.blob.objectstorage.aws.S3BlobStoreConfiguration.UPLOAD_RETRY_EXCEPTION_PREDICATE; import static org.assertj.core.api.Assertions.assertThat; +import java.io.IOException; import java.nio.charset.StandardCharsets; import java.time.Duration; import java.util.Arrays; @@ -85,42 +86,42 @@ void init() { } @Test - void readRangeFirstBytesShouldReturnFirstBytes() { - BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 0, 99).block(); + void readRangeFirstBytesShouldReturnFirstBytes() throws IOException { + BlobStoreDAO.Blob slice = testee.readRange(TEST_BUCKET_NAME, blobId, 0, 99).block(); - assertThat(slice.totalObjectSize()).isEqualTo(1000); - assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(content, 0, 100)); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(1000); + assertThat(slice.asBytes().payload()).isEqualTo(Arrays.copyOfRange(content, 0, 100)); } @Test - void readRangeMidObjectShouldReturnMidBytes() { - BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 200, 399).block(); + void readRangeMidObjectShouldReturnMidBytes() throws IOException { + BlobStoreDAO.Blob slice = testee.readRange(TEST_BUCKET_NAME, blobId, 200, 399).block(); - assertThat(slice.totalObjectSize()).isEqualTo(1000); - assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(content, 200, 400)); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(1000); + assertThat(slice.asBytes().payload()).isEqualTo(Arrays.copyOfRange(content, 200, 400)); } @Test - void readRangeLastBytesNegativeShouldReturnSuffix() { - BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, -100, -1).block(); + void readRangeLastBytesNegativeShouldReturnSuffix() throws IOException { + BlobStoreDAO.Blob slice = testee.readRange(TEST_BUCKET_NAME, blobId, -100, -1).block(); - assertThat(slice.totalObjectSize()).isEqualTo(1000); - assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(content, 900, 1000)); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(1000); + assertThat(slice.asBytes().payload()).isEqualTo(Arrays.copyOfRange(content, 900, 1000)); } @Test - void readRangeFullObjectShouldReturnAllBytes() { - BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 0, 999).block(); + void readRangeFullObjectShouldReturnAllBytes() throws IOException { + BlobStoreDAO.Blob slice = testee.readRange(TEST_BUCKET_NAME, blobId, 0, 999).block(); - assertThat(slice.totalObjectSize()).isEqualTo(1000); - assertThat(slice.data()).isEqualTo(content); + assertThat(BlobStoreDAO.totalObjectSize(slice)).isEqualTo(1000); + assertThat(slice.asBytes().payload()).isEqualTo(content); } @Test - void crossCheckAgainstFullReadShouldMatch() { - BlobStoreDAO.RangeByteSlice slice = testee.readRange(TEST_BUCKET_NAME, blobId, 150, 450).block(); + void crossCheckAgainstFullReadShouldMatch() throws IOException { + BlobStoreDAO.Blob slice = testee.readRange(TEST_BUCKET_NAME, blobId, 150, 450).block(); byte[] fullRead = Mono.from(testee.readBytes(TEST_BUCKET_NAME, blobId)).block().payload(); - assertThat(slice.data()).isEqualTo(Arrays.copyOfRange(fullRead, 150, 451)); + assertThat(slice.asBytes().payload()).isEqualTo(Arrays.copyOfRange(fullRead, 150, 451)); } } From 475171d86dda53b059e840c57e0525078f88c52a Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 11:08:43 +0530 Subject: [PATCH 08/24] JAMES-4231 Compaction: Delegate slot decompression to ZstdBlobStoreDAO --- .../james/blob/compaction/BlobSlot.java | 45 +++++++++---------- .../james/blob/compaction/ChunkFormat.java | 33 +++++++++----- .../blob/compaction/ChunkedBlobStoreDAO.java | 8 ++-- .../BlobCompactionAlgorithmTest.java | 31 ++++++++----- .../blob/compaction/ChunkFormatTest.java | 5 ++- .../compaction/ChunkedBlobStoreDAOTest.java | 45 +++++++++++++------ .../blobstore/BlobStoreModulesChooser.java | 19 ++++---- 7 files changed, 111 insertions(+), 75 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java index 311c108579c..6d3410a6abd 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobSlot.java @@ -23,39 +23,35 @@ import java.util.Objects; import java.util.zip.CRC32C; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadata; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadataName; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadataValue; +import org.apache.james.blob.api.BlobStoreDAO.BytesBlob; +import org.apache.james.blob.api.BlobStoreDAO.ContentEncoding; import org.apache.james.blob.api.ObjectStoreIOException; -import com.github.luben.zstd.Zstd; +public record BlobSlot(long contentStart, int crc32c, long originalSize, byte[] compressedContent, BlobMetadata metadata) { + public static final BlobMetadataName CONTENT_ORIGINAL_SIZE = new BlobMetadataName("content-original-size"); -public record BlobSlot(long contentStart, int crc32c, long originalSize, byte[] compressedContent) { - public byte[] decompress() throws ObjectStoreIOException { - if (originalSize < 0 || originalSize > Integer.MAX_VALUE) { - throw new ObjectStoreIOException("Invalid original size in chunk slot: " + originalSize); - } - byte[] decompressed; - if (originalSize == 0) { - decompressed = new byte[0]; - } else { - try { - decompressed = Zstd.decompress(compressedContent, (int) originalSize); - } catch (Exception e) { - throw new ObjectStoreIOException("Failed to decompress slot content at " + contentStart, e); - } - } - - if (decompressed.length != originalSize) { - throw new ObjectStoreIOException("Decompressed size " + decompressed.length + " does not match expected " + originalSize); - } + public BlobSlot(long contentStart, int crc32c, long originalSize, byte[] compressedContent) { + this(contentStart, crc32c, originalSize, compressedContent, BlobMetadata.empty()); + } + public void verifyCrc() throws ObjectStoreIOException { CRC32C crc = new CRC32C(); - crc.update(decompressed); + crc.update(compressedContent); int computedCrc = (int) crc.getValue(); if (computedCrc != crc32c) { throw new ObjectStoreIOException(String.format("CRC mismatch for chunk slot at offset %d: expected %d, got %d", contentStart, crc32c, computedCrc)); } + } - return decompressed; + public BytesBlob toBlob() { + BlobMetadata enriched = metadata + .withContentEncoding(ContentEncoding.ZSTD) + .withMetadata(CONTENT_ORIGINAL_SIZE, new BlobMetadataValue(String.valueOf(originalSize))); + return BytesBlob.of(compressedContent, enriched); } @Override @@ -67,14 +63,15 @@ public boolean equals(Object o) { return contentStart == other.contentStart && crc32c == other.crc32c && originalSize == other.originalSize - && Arrays.equals(compressedContent, other.compressedContent); + && Arrays.equals(compressedContent, other.compressedContent) + && Objects.equals(metadata, other.metadata); } return false; } @Override public int hashCode() { - int result = Objects.hash(contentStart, crc32c, originalSize); + int result = Objects.hash(contentStart, crc32c, originalSize, metadata); result = 31 * result + Arrays.hashCode(compressedContent); return result; } diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java index b1ca36e896f..bab814fe4f4 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java @@ -66,10 +66,6 @@ public static void write(List slots, OutputStream outputStream) byte[] raw = slot.rawContent(); long originalSize = slot.originalSize(); - CRC32C crc = new CRC32C(); - crc.update(raw); - int crc32c = (int) crc.getValue(); - byte[] compressed; if (raw.length == 0) { compressed = new byte[0]; @@ -77,6 +73,10 @@ public static void write(List slots, OutputStream outputStream) compressed = Zstd.compress(raw); } + CRC32C crc = new CRC32C(); + crc.update(compressed); + int crc32c = (int) crc.getValue(); + String metadata = METADATA_ENCODING + METADATA_SIZE_PREFIX + originalSize + "\n"; byte[] metadataBytes = metadata.getBytes(StandardCharsets.US_ASCII); @@ -127,10 +127,6 @@ public static ChunkWriteResult writeChunk(List slots) throws IO byte[] raw = slot.rawContent(); long originalSize = slot.originalSize(); - CRC32C crc = new CRC32C(); - crc.update(raw); - int crc32c = (int) crc.getValue(); - byte[] compressed; if (raw.length == 0) { compressed = new byte[0]; @@ -138,6 +134,10 @@ public static ChunkWriteResult writeChunk(List slots) throws IO compressed = Zstd.compress(raw); } + CRC32C crc = new CRC32C(); + crc.update(compressed); + int crc32c = (int) crc.getValue(); + String metadata = METADATA_ENCODING + METADATA_SIZE_PREFIX + originalSize + "\n"; byte[] metadataBytes = metadata.getBytes(StandardCharsets.US_ASCII); @@ -277,7 +277,7 @@ public static byte[] readSlot(InputStream chunkRangeStream, long contentStart, l return parseSlotBytes(slotBytes, contentStart); } - public static byte[] parseSlotBytes(byte[] slotBytes, long expectedContentStart) throws ObjectStoreIOException { + public static BlobSlot parseSlot(byte[] slotBytes, long expectedContentStart) throws ObjectStoreIOException { if (slotBytes.length < 8 + 4 + 2) { // 8B contentStart, 4B crc, at least 2 bytes metadata throw new ObjectStoreIOException("Slot byte array too short: " + slotBytes.length); } @@ -332,6 +332,19 @@ public static byte[] parseSlotBytes(byte[] slotBytes, long expectedContentStart) System.arraycopy(slotBytes, contentOffset, compressedContent, 0, compressedLength); BlobSlot blobSlot = new BlobSlot(expectedContentStart, crc32c, originalSize, compressedContent); - return blobSlot.decompress(); + blobSlot.verifyCrc(); + return blobSlot; + } + + public static byte[] parseSlotBytes(byte[] slotBytes, long expectedContentStart) throws ObjectStoreIOException { + BlobSlot blobSlot = parseSlot(slotBytes, expectedContentStart); + if (blobSlot.originalSize() == 0) { + return new byte[0]; + } + try { + return Zstd.decompress(blobSlot.compressedContent(), (int) blobSlot.originalSize()); + } catch (Exception e) { + throw new ObjectStoreIOException("Failed to decompress slot content at " + expectedContentStart, e); + } } } diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java index f5e33df57fb..dcdc9753c05 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAO.java @@ -126,8 +126,8 @@ private Mono readChunkSlot(BucketName bucketName, ChunkId slotRef) { if (rangeData.length == 0 || offset >= totalSize) { return Mono.error(new ObjectNotFoundException("Slot not found at offset " + offset + " in chunk " + chunkObjectBlobId.asString())); } - byte[] decompressed = ChunkFormat.parseSlotBytes(rangeData, offset); - return Mono.just(BytesBlob.of(decompressed)); + BlobSlot slot = ChunkFormat.parseSlot(rangeData, offset); + return Mono.just(slot.toBlob()); } catch (IOException e) { return Mono.error(new ObjectStoreIOException("Error reading slot for chunk " + chunkObjectBlobId.asString(), e)); } @@ -154,8 +154,8 @@ private Mono readChunkSlot(BucketName bucketName, ChunkId slotRef) { .flatMap(sliceBlob -> { try { byte[] sliceData = sliceBlob.asBytes().payload(); - byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, start); - return Mono.just(BytesBlob.of(decompressed)); + BlobSlot slot = ChunkFormat.parseSlot(sliceData, start); + return Mono.just(slot.toBlob()); } catch (IOException e) { return Mono.error(new ObjectStoreIOException("Error reading slot at start " + start, e)); } diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index 922614b5a4d..97139438475 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -47,6 +47,8 @@ import org.junit.jupiter.api.Test; import org.reactivestreams.Publisher; +import com.github.luben.zstd.Zstd; + import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; @@ -106,6 +108,17 @@ void setUp() { testee = new BlobCompactionAlgorithm(rawStore, rawStore, mappingSource, recordingUpdater); } + private byte[] readDecompressed(BlobId blobId) { + BlobStoreDAO.BytesBlob blob = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, blobId)).block(); + if (blob.metadata().contentEncoding().filter(BlobStoreDAO.ContentEncoding.ZSTD::equals).isPresent()) { + long origSize = blob.metadata().get(BlobSlot.CONTENT_ORIGINAL_SIZE) + .map(v -> Long.parseLong(v.value())) + .orElse((long) blob.payload().length); + return Zstd.decompress(blob.payload(), (int) origSize); + } + return blob.payload(); + } + @Test void initialCompactionShouldPackSmallBlobsAndSkipLargeBlobs() { BlobId small1 = new PlainBlobId("1_2_small1"); @@ -152,11 +165,8 @@ void initialCompactionShouldPackSmallBlobsAndSkipLargeBlobs() { assertThat(ChunkId.isChunkRef(newSlotRef1)).isTrue(); assertThat(ChunkId.isChunkRef(newSlotRef2)).isTrue(); - // Ranged read of new slot refs returns exact content - byte[] read1 = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlotRef1)).block().payload(); - byte[] read2 = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlotRef2)).block().payload(); - assertThat(read1).isEqualTo(payload1); - assertThat(read2).isEqualTo(payload2); + assertThat(readDecompressed(newSlotRef1)).isEqualTo(payload1); + assertThat(readDecompressed(newSlotRef2)).isEqualTo(payload2); } @Test @@ -187,9 +197,7 @@ void initialCompactionPreservesDeduplication() { assertThat(rep.oldId()).isEqualTo(sharedBlobId); assertThat(rep.messageIds()).containsExactlyInAnyOrder("msg-A", "msg-B"); - // Content readable - byte[] read = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, rep.newId())).block().payload(); - assertThat(read).isEqualTo(sharedPayload); + assertThat(readDecompressed(rep.newId())).isEqualTo(sharedPayload); } @Test @@ -235,8 +243,7 @@ void gcCompactShouldPurgeDeadSlotsWhenThresholdExceeded() throws Exception { // Surviving slots readable for (int i = 0; i < 8; i++) { BlobId newSlotRef = recordingUpdater.getReplacements().get(i).newId(); - byte[] read = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlotRef)).block().payload(); - assertThat(read).isEqualTo(("Slot content " + i).getBytes(StandardCharsets.UTF_8)); + assertThat(readDecompressed(newSlotRef)).isEqualTo(("Slot content " + i).getBytes(StandardCharsets.UTF_8)); } } @@ -320,8 +327,8 @@ void gcCompactShouldMergeTwoSmallChunks() throws Exception { BlobId newSlot1 = recordingUpdater.getReplacements().get(0).newId(); BlobId newSlot2 = recordingUpdater.getReplacements().get(1).newId(); - assertThat(Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlot1)).block().payload()).isEqualTo(payload1); - assertThat(Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newSlot2)).block().payload()).isEqualTo(payload2); + assertThat(readDecompressed(newSlot1)).isEqualTo(payload1); + assertThat(readDecompressed(newSlot2)).isEqualTo(payload2); } @Test diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java index 5953e1f5341..91759d6719e 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java @@ -197,10 +197,11 @@ void verifyBinaryLayoutExactStructure() throws Exception { ByteBuffer bb = ByteBuffer.wrap(chunkBytes); assertThat(bb.getLong(1)).isEqualTo(1L); - // Bytes 9..12: CRC32C of "Test" + // Bytes 9..12: CRC32C of compressed content int crc = bb.getInt(9); + byte[] compressed = com.github.luben.zstd.Zstd.compress(raw); java.util.zip.CRC32C crcCalculator = new java.util.zip.CRC32C(); - crcCalculator.update(raw); + crcCalculator.update(compressed); assertThat(crc).isEqualTo((int) crcCalculator.getValue()); // Followed by metadata diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java index 07252827e97..c2b8459d6c2 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkedBlobStoreDAOTest.java @@ -43,6 +43,8 @@ import org.junit.jupiter.api.Test; import org.reactivestreams.Publisher; +import com.github.luben.zstd.Zstd; + import reactor.core.publisher.Mono; class ChunkedBlobStoreDAOTest { @@ -184,14 +186,21 @@ void chunkSlotRefReadReturnsOriginalContentWithExactlyOneRangedRead() throws Exc countingRawStore.resetCount(); - byte[] readSlot1 = Mono.from(testee.readBytes(TEST_BUCKET, slot1)).block().payload(); - assertThat(readSlot1).isEqualTo(content1); + // ChunkedBlobStoreDAO returns slot tagged with ContentEncoding.ZSTD + BlobStoreDAO.BytesBlob rawSlot1 = Mono.from(testee.readBytes(TEST_BUCKET, slot1)).block(); + assertThat(rawSlot1.metadata().contentEncoding()).contains(BlobStoreDAO.ContentEncoding.ZSTD); + assertThat(rawSlot1.metadata().get(BlobSlot.CONTENT_ORIGINAL_SIZE)) + .contains(new BlobStoreDAO.BlobMetadataValue(String.valueOf(content1.length))); + byte[] decompressed1 = Zstd.decompress(rawSlot1.payload(), content1.length); + assertThat(decompressed1).isEqualTo(content1); assertThat(countingRawStore.rangeCallCount()).isEqualTo(1); // EXACTLY ONE ranged read! countingRawStore.resetCount(); - byte[] readSlot2 = Mono.from(testee.readBytes(TEST_BUCKET, slot2)).block().payload(); - assertThat(readSlot2).isEqualTo(content2); + BlobStoreDAO.BytesBlob rawSlot2 = Mono.from(testee.readBytes(TEST_BUCKET, slot2)).block(); + assertThat(rawSlot2.metadata().contentEncoding()).contains(BlobStoreDAO.ContentEncoding.ZSTD); + byte[] decompressed2 = Zstd.decompress(rawSlot2.payload(), content2.length); + assertThat(decompressed2).isEqualTo(content2); assertThat(countingRawStore.rangeCallCount()).isEqualTo(1); // EXACTLY ONE ranged read! } @@ -206,9 +215,11 @@ void limitZeroSlotReadPerformsFooterWalk() throws Exception { ChunkId walkRef = ChunkId.slotRef(baseChunk, 1, 0); // limit == 0 triggers footer walk countingRawStore.resetCount(); - byte[] readResult = Mono.from(testee.readBytes(TEST_BUCKET, walkRef)).block().payload(); + BlobStoreDAO.BytesBlob readResult = Mono.from(testee.readBytes(TEST_BUCKET, walkRef)).block(); - assertThat(readResult).isEqualTo(content); + assertThat(readResult.metadata().contentEncoding()).contains(BlobStoreDAO.ContentEncoding.ZSTD); + byte[] decompressed = Zstd.decompress(readResult.payload(), content.length); + assertThat(decompressed).isEqualTo(content); // Footer walk does 2 range reads: 1 for tail buffer (footer), 1 for slot data assertThat(countingRawStore.rangeCallCount()).isEqualTo(2); } @@ -258,11 +269,18 @@ void deduplicationInvariantBothMessagesReadIdenticalBytesFromSameSlot() throws E assertThat(slotRefForMessageA).isEqualTo(slotRefForMessageB); - byte[] readForA = Mono.from(testee.readBytes(TEST_BUCKET, slotRefForMessageA)).block().payload(); - byte[] readForB = Mono.from(testee.readBytes(TEST_BUCKET, slotRefForMessageB)).block().payload(); + BlobStoreDAO.BytesBlob readBlobA = Mono.from(testee.readBytes(TEST_BUCKET, slotRefForMessageA)).block(); + BlobStoreDAO.BytesBlob readBlobB = Mono.from(testee.readBytes(TEST_BUCKET, slotRefForMessageB)).block(); + + assertThat(readBlobA.metadata().contentEncoding()).contains(BlobStoreDAO.ContentEncoding.ZSTD); + assertThat(readBlobB.metadata().contentEncoding()).contains(BlobStoreDAO.ContentEncoding.ZSTD); + assertThat(readBlobA.payload()).isEqualTo(readBlobB.payload()); + + byte[] decompressedA = Zstd.decompress(readBlobA.payload(), sharedEmailBody.length); + byte[] decompressedB = Zstd.decompress(readBlobB.payload(), sharedEmailBody.length); - assertThat(readForA).isEqualTo(sharedEmailBody); - assertThat(readForB).isEqualTo(sharedEmailBody); + assertThat(decompressedA).isEqualTo(sharedEmailBody); + assertThat(decompressedB).isEqualTo(sharedEmailBody); } @Test @@ -314,9 +332,10 @@ void readChunkSlotShouldTriggerRepairWhenRepairedToAnotherSlotRef() throws Excep }; ChunkedBlobStoreDAO daoWithRepairer = new ChunkedBlobStoreDAO(plainChain, countingRawStore, Optional.of(repairer)); - - byte[] repairedRead = Mono.from(daoWithRepairer.readBytes(TEST_BUCKET, staleSlot)).block().payload(); - assertThat(repairedRead).isEqualTo(content); + BlobStoreDAO.BytesBlob repairedSlot = Mono.from(daoWithRepairer.readBytes(TEST_BUCKET, staleSlot)).block(); + assertThat(repairedSlot.metadata().contentEncoding()).contains(BlobStoreDAO.ContentEncoding.ZSTD); + byte[] decompressed = Zstd.decompress(repairedSlot.payload(), content.length); + assertThat(decompressed).isEqualTo(content); } @Test diff --git a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java index fcfed410782..dabdecc14ef 100644 --- a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java +++ b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java @@ -74,7 +74,7 @@ public class BlobStoreModulesChooser { public static final String RAW = "raw"; public static final String ENCRYPTION = "encryption"; - public static final String PLAIN_CHAIN = "plain-chain"; + public static final String CHUNKED = "chunked"; static class CassandraBlobStoreDAODeclarationModule extends AbstractModule { @Override @@ -138,9 +138,8 @@ protected void configure() { static class NoCompressionModule extends AbstractModule { @Provides @Singleton - @Named(PLAIN_CHAIN) - BlobStoreDAO blobStoreDAO(@Named(ENCRYPTION) BlobStoreDAO encryption) { - return encryption; + BlobStoreDAO blobStoreDAO(@Named(CHUNKED) BlobStoreDAO chunked, MetricFactory metricFactory) { + return new ZstdBlobStoreDAO(chunked, CompressionConfiguration.builder().enabled(false).minRatio(0).build(), metricFactory); } } @@ -153,9 +152,8 @@ static class CompressionModule extends AbstractModule { @Provides @Singleton - @Named(PLAIN_CHAIN) - BlobStoreDAO blobStoreDAO(@Named(ENCRYPTION) BlobStoreDAO encryption, MetricFactory metricFactory) { - return new ZstdBlobStoreDAO(encryption, compressionConfiguration, metricFactory); + BlobStoreDAO blobStoreDAO(@Named(CHUNKED) BlobStoreDAO chunked, MetricFactory metricFactory) { + return new ZstdBlobStoreDAO(chunked, compressionConfiguration, metricFactory); } @Provides @@ -167,13 +165,14 @@ CompressionConfiguration compressionConfiguration() { static class ChunkedBlobStoreModule extends AbstractModule { @Provides @Singleton - BlobStoreDAO blobStoreDAO(@Named(PLAIN_CHAIN) BlobStoreDAO plainChain, + @Named(CHUNKED) + BlobStoreDAO blobStoreDAO(@Named(ENCRYPTION) BlobStoreDAO encryption, @Named(RAW) BlobStoreDAO raw, Injector injector) { Optional blobIdRepairer = Optional.ofNullable( injector.getExistingBinding(Key.get(BlobIdRepairer.class))) .map(binding -> binding.getProvider().get()); - return new ChunkedBlobStoreDAO(plainChain, raw, blobIdRepairer); + return new ChunkedBlobStoreDAO(encryption, raw, blobIdRepairer); } } @@ -210,8 +209,8 @@ public static List chooseModules(BlobStoreConfiguration choosingConfigur return ImmutableList.builder() .add(chooseBlobStoreDAOModule(choosingConfiguration.getImplementation())) .add(chooseEncryptionModule(choosingConfiguration.getCryptoConfig())) - .add(chooseCompressionModule(choosingConfiguration.getCompressionConfiguration())) .add(new ChunkedBlobStoreModule()) + .add(chooseCompressionModule(choosingConfiguration.getCompressionConfiguration())) .add(new BlobCompactionModule()) .addAll(chooseStoragePolicyModule(choosingConfiguration.storageStrategy())) .add(new StoragePolicyConfigurationSanityEnforcementModule()) From c8d81c3b7304b49783346be5456ed3e7f3e01685 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 11:59:38 +0530 Subject: [PATCH 09/24] JAMES-4231 Compaction: Preserve and restore blob metadata in chunk slots --- .../compaction/BlobCompactionAlgorithm.java | 28 ++-- .../james/blob/compaction/ChunkFormat.java | 126 +++++++++++++----- .../BlobCompactionAlgorithmTest.java | 43 ++++++ .../blob/compaction/ChunkFormatTest.java | 60 ++++++++- 4 files changed, 212 insertions(+), 45 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index 7c747deb2ff..750e5665625 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -33,6 +33,7 @@ import org.apache.james.blob.api.BlobId; import org.apache.james.blob.api.BlobReferenceSource; import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadata; import org.apache.james.blob.api.ObjectStoreIOException; import org.apache.james.blob.compaction.BlobReferenceMappingSource.BlobIdMessageIdMapping; import org.apache.james.blob.compaction.ChunkFormat.BlobSlotContent; @@ -155,7 +156,7 @@ public Mono initialCompact(CompactionRequest request) { .window(DEFAULT_CANDIDATE_BATCH_SIZE) .concatMap(windowFlux -> windowFlux .flatMap(blobId -> Mono.from(rawStore.readBytes(request.bucketName(), blobId)) - .map(bytesBlob -> new CandidateBlob(blobId, bytesBlob.payload())) + .map(bytesBlob -> new CandidateBlob(blobId, bytesBlob.payload(), bytesBlob.metadata())) .onErrorResume(error -> { LOGGER.warn("Failed reading candidate blob {}", blobId.asString(), error); return Mono.empty(); @@ -240,7 +241,7 @@ private Mono persistChunkBatch(CompactionRequest request, ChunkId chunkId = ChunkId.ofChunk(family, request.generation()); List slots = batch.stream() - .map(candidate -> BlobSlotContent.of(candidate.payload)) + .map(candidate -> BlobSlotContent.of(candidate.payload, candidate.metadata)) .toList(); ChunkWriteResult writeResult; @@ -348,8 +349,9 @@ private Mono purgeDeadSlots(CompactionRequest request, .flatMap(sliceBlob -> { try { byte[] sliceData = sliceBlob.asBytes().payload(); - byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, liveSlot.offset); - return Mono.just(new LiveSlotWithContent(liveSlot, decompressed)); + BlobSlot slot = ChunkFormat.parseSlot(sliceData, liveSlot.offset); + byte[] decompressed = slot.originalSize() == 0 ? new byte[0] : com.github.luben.zstd.Zstd.decompress(slot.compressedContent(), (int) slot.originalSize()); + return Mono.just(new LiveSlotWithContent(liveSlot, decompressed, slot.metadata())); } catch (IOException e) { return Mono.error(new ObjectStoreIOException("Failed parsing slot", e)); } @@ -357,7 +359,7 @@ private Mono purgeDeadSlots(CompactionRequest request, .collectList() .flatMap(liveSlotsWithContent -> { List liveSlotContents = liveSlotsWithContent.stream() - .map(slot -> BlobSlotContent.of(slot.decompressedContent)) + .map(slot -> BlobSlotContent.of(slot.decompressedContent, slot.metadata)) .toList(); ChunkWriteResult writeResult; @@ -422,8 +424,9 @@ private Mono mergePair(CompactionRequest request, ChunkPair pa .flatMap(sliceBlob -> { try { byte[] sliceData = sliceBlob.asBytes().payload(); - byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, slot.offset); - return Mono.just(new LiveSlotWithContent(slot, decompressed)); + BlobSlot parsed = ChunkFormat.parseSlot(sliceData, slot.offset); + byte[] decompressed = parsed.originalSize() == 0 ? new byte[0] : com.github.luben.zstd.Zstd.decompress(parsed.compressedContent(), (int) parsed.originalSize()); + return Mono.just(new LiveSlotWithContent(slot, decompressed, parsed.metadata())); } catch (IOException e) { return Mono.error(new ObjectStoreIOException("Failed parsing slot", e)); } @@ -436,8 +439,9 @@ private Mono mergePair(CompactionRequest request, ChunkPair pa .flatMap(sliceBlob -> { try { byte[] sliceData = sliceBlob.asBytes().payload(); - byte[] decompressed = ChunkFormat.parseSlotBytes(sliceData, slot.offset); - return Mono.just(new LiveSlotWithContent(slot, decompressed)); + BlobSlot parsed = ChunkFormat.parseSlot(sliceData, slot.offset); + byte[] decompressed = parsed.originalSize() == 0 ? new byte[0] : com.github.luben.zstd.Zstd.decompress(parsed.compressedContent(), (int) parsed.originalSize()); + return Mono.just(new LiveSlotWithContent(slot, decompressed, parsed.metadata())); } catch (IOException e) { return Mono.error(new ObjectStoreIOException("Failed parsing slot", e)); } @@ -450,7 +454,7 @@ private Mono mergePair(CompactionRequest request, ChunkPair pa allLive.addAll(tuple.getT2()); List slotContents = allLive.stream() - .map(slot -> BlobSlotContent.of(slot.decompressedContent)) + .map(slot -> BlobSlotContent.of(slot.decompressedContent, slot.metadata)) .toList(); ChunkWriteResult writeResult; @@ -596,13 +600,13 @@ static int extractFamily(String blobIdStr) { return 1; } - private record CandidateBlob(BlobId blobId, byte[] payload) {} + private record CandidateBlob(BlobId blobId, byte[] payload, BlobMetadata metadata) {} private record ExistingChunk(BlobId chunkBlobId, long totalChunkSize, ChunkFooter footer) {} private record LiveSlotMeta(long offset, long limit, ChunkId slotRef, Set messageIds) {} - private record LiveSlotWithContent(LiveSlotMeta meta, byte[] decompressedContent) {} + private record LiveSlotWithContent(LiveSlotMeta meta, byte[] decompressedContent, BlobMetadata metadata) {} private record ChunkPair(ChunkAnalysis c1, ChunkAnalysis c2) {} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java index bab814fe4f4..d7a4fe3d7f2 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/ChunkFormat.java @@ -29,9 +29,13 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.List; +import java.util.Map; import java.util.stream.Collectors; import java.util.zip.CRC32C; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadata; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadataName; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadataValue; import org.apache.james.blob.api.ObjectStoreIOException; import com.github.luben.zstd.Zstd; @@ -45,10 +49,15 @@ public class ChunkFormat { public static final String METADATA_ENCODING = "content-encoding=zstd\n"; public static final String METADATA_SIZE_PREFIX = "content-original-size="; - public record BlobSlotContent(byte[] rawContent, long originalSize) { + public record BlobSlotContent(byte[] rawContent, long originalSize, BlobMetadata metadata) { public static BlobSlotContent of(byte[] rawContent) { + return of(rawContent, BlobMetadata.empty()); + } + + public static BlobSlotContent of(byte[] rawContent, BlobMetadata metadata) { Preconditions.checkNotNull(rawContent, "'rawContent' must not be null"); - return new BlobSlotContent(rawContent, rawContent.length); + Preconditions.checkNotNull(metadata, "'metadata' must not be null"); + return new BlobSlotContent(rawContent, rawContent.length, metadata); } } @@ -77,8 +86,19 @@ public static void write(List slots, OutputStream outputStream) crc.update(compressed); int crc32c = (int) crc.getValue(); - String metadata = METADATA_ENCODING + METADATA_SIZE_PREFIX + originalSize + "\n"; - byte[] metadataBytes = metadata.getBytes(StandardCharsets.US_ASCII); + StringBuilder metadataBuilder = new StringBuilder(); + metadataBuilder.append(METADATA_ENCODING); + metadataBuilder.append(METADATA_SIZE_PREFIX).append(originalSize).append('\n'); + if (slot.metadata() != null) { + for (Map.Entry entry : slot.metadata().underlyingMap().entrySet()) { + String key = entry.getKey().name(); + if (!key.equalsIgnoreCase("content-encoding") && !key.equalsIgnoreCase("content-original-size")) { + metadataBuilder.append(key).append('=').append(entry.getValue().value()).append('\n'); + } + } + } + metadataBuilder.append('\n'); + byte[] metadataBytes = metadataBuilder.toString().getBytes(StandardCharsets.US_ASCII); slotStarts.add(currentOffset); @@ -138,8 +158,19 @@ public static ChunkWriteResult writeChunk(List slots) throws IO crc.update(compressed); int crc32c = (int) crc.getValue(); - String metadata = METADATA_ENCODING + METADATA_SIZE_PREFIX + originalSize + "\n"; - byte[] metadataBytes = metadata.getBytes(StandardCharsets.US_ASCII); + StringBuilder metadataBuilder = new StringBuilder(); + metadataBuilder.append(METADATA_ENCODING); + metadataBuilder.append(METADATA_SIZE_PREFIX).append(originalSize).append('\n'); + if (slot.metadata() != null) { + for (Map.Entry entry : slot.metadata().underlyingMap().entrySet()) { + String key = entry.getKey().name(); + if (!key.equalsIgnoreCase("content-encoding") && !key.equalsIgnoreCase("content-original-size")) { + metadataBuilder.append(key).append('=').append(entry.getValue().value()).append('\n'); + } + } + } + metadataBuilder.append('\n'); + byte[] metadataBytes = metadataBuilder.toString().getBytes(StandardCharsets.US_ASCII); slotStarts.add(currentOffset); long slotLength = 8L + 4L + metadataBytes.length + compressed.length; @@ -291,47 +322,78 @@ public static BlobSlot parseSlot(byte[] slotBytes, long expectedContentStart) th int crc32c = buffer.getInt(8); - // Find metadata lines (up to second '\n') - int firstNewline = -1; - int secondNewline = -1; - for (int i = 12; i < slotBytes.length; i++) { - if (slotBytes[i] == '\n') { - if (firstNewline == -1) { - firstNewline = i; - } else { - secondNewline = i; + int lineStart = 12; + int lineNum = 0; + long originalSize = -1; + BlobMetadata customMetadata = BlobMetadata.empty(); + int contentOffset = -1; + + while (lineStart < slotBytes.length) { + int newlineIndex = -1; + for (int i = lineStart; i < slotBytes.length; i++) { + if (slotBytes[i] == '\n') { + newlineIndex = i; break; } } - } + if (newlineIndex == -1) { + break; + } - if (firstNewline == -1 || secondNewline == -1) { - throw new ObjectStoreIOException("Corrupt slot metadata: missing newlines in header at " + expectedContentStart); - } + int lineLen = newlineIndex - lineStart; + if (lineLen == 0) { + contentOffset = newlineIndex + 1; + break; + } - String encodingLine = new String(slotBytes, 12, firstNewline - 12, StandardCharsets.US_ASCII); - if (!"content-encoding=zstd".equals(encodingLine.trim())) { - throw new ObjectStoreIOException("Unsupported slot encoding: " + encodingLine); + String line = new String(slotBytes, lineStart, lineLen, StandardCharsets.US_ASCII).trim(); + if (lineNum == 0) { + if (!"content-encoding=zstd".equalsIgnoreCase(line)) { + throw new ObjectStoreIOException("Unsupported slot encoding: " + line); + } + } else if (lineNum == 1) { + if (!line.startsWith(METADATA_SIZE_PREFIX)) { + throw new ObjectStoreIOException("Missing original size metadata line: " + line); + } + try { + originalSize = Long.parseLong(line.substring(METADATA_SIZE_PREFIX.length()).trim()); + } catch (NumberFormatException e) { + throw new ObjectStoreIOException("Corrupt original size metadata: " + line, e); + } + } else { + int eqIndex = line.indexOf('='); + if (eqIndex > 0) { + String key = line.substring(0, eqIndex).trim(); + String val = line.substring(eqIndex + 1).trim(); + try { + customMetadata = customMetadata.withMetadata(new BlobMetadataName(key), new BlobMetadataValue(val)); + } catch (IllegalArgumentException e) { + contentOffset = lineStart; + break; + } + } else { + contentOffset = lineStart; + break; + } + } + + lineNum++; + lineStart = newlineIndex + 1; } - String sizeLine = new String(slotBytes, firstNewline + 1, secondNewline - (firstNewline + 1), StandardCharsets.US_ASCII).trim(); - if (!sizeLine.startsWith(METADATA_SIZE_PREFIX)) { - throw new ObjectStoreIOException("Missing original size metadata line: " + sizeLine); + if (originalSize == -1) { + throw new ObjectStoreIOException("Corrupt slot metadata: missing original size at " + expectedContentStart); } - long originalSize; - try { - originalSize = Long.parseLong(sizeLine.substring(METADATA_SIZE_PREFIX.length()).trim()); - } catch (NumberFormatException e) { - throw new ObjectStoreIOException("Corrupt original size metadata: " + sizeLine, e); + if (contentOffset == -1) { + contentOffset = lineStart; } - int contentOffset = secondNewline + 1; int compressedLength = slotBytes.length - contentOffset; byte[] compressedContent = new byte[compressedLength]; System.arraycopy(slotBytes, contentOffset, compressedContent, 0, compressedLength); - BlobSlot blobSlot = new BlobSlot(expectedContentStart, crc32c, originalSize, compressedContent); + BlobSlot blobSlot = new BlobSlot(expectedContentStart, crc32c, originalSize, compressedContent, customMetadata); blobSlot.verifyCrc(); return blobSlot; } diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index 97139438475..e08ba1f6b5e 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -544,4 +544,47 @@ void gcCompactShouldBoundHeapUsageByMaxSlotSizeAndNeverReadWholeChunkPayloads() assertThat(trackingStore.getMaxRangedReadBytes()).isLessThan(totalChunkSize); assertThat(trackingStore.getMaxRangedReadBytes()).isLessThanOrEqualTo(Math.max(65536, 76 * 1024)); } + + @Test + void initialCompactShouldPreserveCustomBlobMetadata() { + BlobId b1 = new PlainBlobId("1_2_blob1"); + BlobId b2 = new PlainBlobId("1_2_blob2"); + + BlobStoreDAO.BlobMetadata meta1 = BlobStoreDAO.BlobMetadata.empty() + .withMetadata(new BlobStoreDAO.BlobMetadataName("custom-header"), new BlobStoreDAO.BlobMetadataValue("meta-value-1")); + BlobStoreDAO.BlobMetadata meta2 = BlobStoreDAO.BlobMetadata.empty() + .withMetadata(new BlobStoreDAO.BlobMetadataName("custom-header"), new BlobStoreDAO.BlobMetadataValue("meta-value-2")); + + byte[] payload1 = "Candidate 1 with metadata".getBytes(StandardCharsets.UTF_8); + byte[] payload2 = "Candidate 2 with metadata".getBytes(StandardCharsets.UTF_8); + + Mono.from(rawStore.save(TEST_BUCKET, b1, BlobStoreDAO.BytesBlob.of(payload1, meta1))).block(); + Mono.from(rawStore.save(TEST_BUCKET, b2, BlobStoreDAO.BytesBlob.of(payload2, meta2))).block(); + + mappingSource.add(b1, "msg1"); + mappingSource.add(b2, "msg2"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder().chunkTargetSize(100_000).build()) + .build(); + + CompactionResult result = testee.initialCompact(request).block(); + assertThat(result.packedBlobs()).isEqualTo(2); + + List replacements = recordingUpdater.getReplacements(); + assertThat(replacements).hasSize(2); + + BlobId newRef1 = replacements.get(0).newId(); + BlobStoreDAO.BytesBlob readBlob1 = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newRef1)).block(); + assertThat(readBlob1.metadata().get(new BlobStoreDAO.BlobMetadataName("custom-header"))) + .contains(new BlobStoreDAO.BlobMetadataValue("meta-value-1")); + + BlobId newRef2 = replacements.get(1).newId(); + BlobStoreDAO.BytesBlob readBlob2 = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, newRef2)).block(); + assertThat(readBlob2.metadata().get(new BlobStoreDAO.BlobMetadataName("custom-header"))) + .contains(new BlobStoreDAO.BlobMetadataValue("meta-value-2")); + } } diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java index 91759d6719e..f397fa4d996 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/ChunkFormatTest.java @@ -28,6 +28,11 @@ import java.util.Arrays; import java.util.List; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadata; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadataName; +import org.apache.james.blob.api.BlobStoreDAO.BlobMetadataValue; +import org.apache.james.blob.api.BlobStoreDAO.BytesBlob; +import org.apache.james.blob.api.BlobStoreDAO.ContentEncoding; import org.apache.james.blob.api.ObjectStoreIOException; import org.junit.jupiter.api.Test; @@ -205,7 +210,7 @@ void verifyBinaryLayoutExactStructure() throws Exception { assertThat(crc).isEqualTo((int) crcCalculator.getValue()); // Followed by metadata - String metadataExpected = "content-encoding=zstd\ncontent-original-size=4\n"; + String metadataExpected = "content-encoding=zstd\ncontent-original-size=4\n\n"; byte[] metadataBytes = metadataExpected.getBytes(StandardCharsets.US_ASCII); byte[] actualMetadata = Arrays.copyOfRange(chunkBytes, 13, 13 + metadataBytes.length); assertThat(actualMetadata).isEqualTo(metadataBytes); @@ -220,4 +225,57 @@ void verifyBinaryLayoutExactStructure() throws Exception { byte[] footerBytes = Arrays.copyOfRange(chunkBytes, (int) footerPosition, (int) footerPosition + footerLength); assertThat(new String(footerBytes, StandardCharsets.US_ASCII)).isEqualTo("1"); } + + @Test + void slotWithCustomMetadataRoundTrip() throws Exception { + byte[] raw = "Payload with custom metadata".getBytes(StandardCharsets.UTF_8); + BlobMetadata customMeta = BlobMetadata.empty() + .withMetadata(new BlobMetadataName("custom-header"), new BlobMetadataValue("my-value")) + .withMetadata(new BlobMetadataName("x-source-app"), new BlobMetadataValue("james-mail")); + + byte[] chunkBytes = ChunkFormat.writeToBytes(List.of(ChunkFormat.BlobSlotContent.of(raw, customMeta))); + + ChunkFooter footer = ChunkFormat.readFooter(chunkBytes); + long start = footer.slotStarts().get(0); + long end = footer.footerPosition(); + + byte[] slotData = Arrays.copyOfRange(chunkBytes, (int) start, (int) end); + BlobSlot slot = ChunkFormat.parseSlot(slotData, start); + + assertThat(slot.metadata().get(new BlobMetadataName("custom-header"))) + .contains(new BlobMetadataValue("my-value")); + assertThat(slot.metadata().get(new BlobMetadataName("x-source-app"))) + .contains(new BlobMetadataValue("james-mail")); + + BytesBlob bytesBlob = slot.toBlob(); + assertThat(bytesBlob.metadata().get(new BlobMetadataName("custom-header"))) + .contains(new BlobMetadataValue("my-value")); + assertThat(bytesBlob.metadata().contentEncoding()).contains(ContentEncoding.ZSTD); + } + + @Test + void backwardsCompatibilityWithTwoLineHeaderWithoutEmptyLine() throws Exception { + byte[] raw = "Test content for backwards compatibility".getBytes(StandardCharsets.UTF_8); + byte[] compressed = com.github.luben.zstd.Zstd.compress(raw); + java.util.zip.CRC32C crc = new java.util.zip.CRC32C(); + crc.update(compressed); + int crc32c = (int) crc.getValue(); + + // 2-line header without terminating empty line + String header = "content-encoding=zstd\ncontent-original-size=" + raw.length + "\n"; + byte[] headerBytes = header.getBytes(StandardCharsets.US_ASCII); + + ByteBuffer bb = ByteBuffer.allocate(8 + 4 + headerBytes.length + compressed.length); + bb.putLong(1L); + bb.putInt(crc32c); + bb.put(headerBytes); + bb.put(compressed); + + byte[] slotBytes = bb.array(); + BlobSlot parsed = ChunkFormat.parseSlot(slotBytes, 1L); + + assertThat(parsed.originalSize()).isEqualTo(raw.length); + assertThat(parsed.compressedContent()).isEqualTo(compressed); + assertThat(ChunkFormat.parseSlotBytes(slotBytes, 1L)).isEqualTo(raw); + } } From d4b75ab46cdf2a7684250d92f0949f85dc9a269d Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:01:20 +0530 Subject: [PATCH 10/24] JAMES-4231 Compaction: Push down generation prefix during candidate listing --- .../compaction/BlobCompactionAlgorithm.java | 13 +++++++-- .../BlobCompactionAlgorithmTest.java | 29 +++++++++++++++++++ 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index 750e5665625..d3e6f876474 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -39,6 +39,7 @@ import org.apache.james.blob.compaction.ChunkFormat.BlobSlotContent; import org.apache.james.blob.compaction.ChunkFormat.ChunkWriteResult; import org.apache.james.blob.compaction.ChunkFormat.SlotRange; +import org.reactivestreams.Publisher; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -149,7 +150,11 @@ public Mono initialCompact(CompactionRequest request) { return Mono.just(CompactionResult.NONE); } - return Flux.from(rawStore.listBlobs(request.bucketName())) + Publisher candidatesListing = request.family() + .map(family -> rawStore.listBlobs(request.bucketName(), family + "_" + request.generation() + "_")) + .orElseGet(() -> rawStore.listBlobs(request.bucketName())); + + return Flux.from(candidatesListing) .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) .filter(blobId -> !ChunkId.isChunkRef(blobId)) .filter(mapping::containsKey) @@ -180,8 +185,12 @@ public Mono initialCompact(CompactionRequest request) { public Mono gcCompact(CompactionRequest request) { Preconditions.checkNotNull(request, "'request' must not be null"); + Publisher chunkListing = request.family() + .map(family -> rawStore.listBlobs(request.bucketName(), family + "_" + request.generation() + "_chunk")) + .orElseGet(() -> rawStore.listBlobs(request.bucketName())); + return loadReferenceMapping() - .flatMap(mapping -> Flux.from(rawStore.listBlobs(request.bucketName())) + .flatMap(mapping -> Flux.from(chunkListing) .filter(blobId -> ChunkId.isChunkRef(blobId) && blobId.asString().indexOf('~') == -1) .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) .flatMap(chunkBlobId -> Mono.from(rawStore.readRange(request.bucketName(), chunkBlobId, -65536, -1)) diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index e08ba1f6b5e..964057d8fcf 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -587,4 +587,33 @@ void initialCompactShouldPreserveCustomBlobMetadata() { assertThat(readBlob2.metadata().get(new BlobStoreDAO.BlobMetadataName("custom-header"))) .contains(new BlobStoreDAO.BlobMetadataValue("meta-value-2")); } + + @Test + void initialCompactShouldUsePrefixPushdownToAvoidListingOtherFamiliesOrGenerations() { + BlobId targetBlob = new PlainBlobId("1_2_target"); + BlobId otherGenBlob = new PlainBlobId("1_3_otherGen"); + BlobId otherFamilyBlob = new PlainBlobId("2_2_otherFamily"); + + Mono.from(rawStore.save(TEST_BUCKET, targetBlob, BlobStoreDAO.BytesBlob.of("target"))).block(); + Mono.from(rawStore.save(TEST_BUCKET, otherGenBlob, BlobStoreDAO.BytesBlob.of("otherGen"))).block(); + Mono.from(rawStore.save(TEST_BUCKET, otherFamilyBlob, BlobStoreDAO.BytesBlob.of("otherFamily"))).block(); + + mappingSource.add(targetBlob, "msg-target"); + mappingSource.add(otherGenBlob, "msg-otherGen"); + mappingSource.add(otherFamilyBlob, "msg-otherFamily"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder().chunkTargetSize(100_000).build()) + .build(); + + CompactionResult result = testee.initialCompact(request).block(); + assertThat(result.packedBlobs()).isEqualTo(1); + + List replacements = recordingUpdater.getReplacements(); + assertThat(replacements).hasSize(1); + assertThat(replacements.get(0).oldId()).isEqualTo(targetBlob); + } } From 3f56a150ac599dd47c46ad770a7e24bcebce4af6 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:13:18 +0530 Subject: [PATCH 11/24] JAMES-4231 Compaction: Filter oversized candidates prior to windowing --- .../blob/compaction/BlobCompactionAlgorithm.java | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index d3e6f876474..a6f6f9af630 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -158,15 +158,15 @@ public Mono initialCompact(CompactionRequest request) { .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) .filter(blobId -> !ChunkId.isChunkRef(blobId)) .filter(mapping::containsKey) + .flatMap(blobId -> Mono.from(rawStore.readBytes(request.bucketName(), blobId)) + .map(bytesBlob -> new CandidateBlob(blobId, bytesBlob.payload(), bytesBlob.metadata())) + .onErrorResume(error -> { + LOGGER.warn("Failed reading candidate blob {}", blobId.asString(), error); + return Mono.empty(); + }), 16) + .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()) .window(DEFAULT_CANDIDATE_BATCH_SIZE) .concatMap(windowFlux -> windowFlux - .flatMap(blobId -> Mono.from(rawStore.readBytes(request.bucketName(), blobId)) - .map(bytesBlob -> new CandidateBlob(blobId, bytesBlob.payload(), bytesBlob.metadata())) - .onErrorResume(error -> { - LOGGER.warn("Failed reading candidate blob {}", blobId.asString(), error); - return Mono.empty(); - })) - .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()) .collectList() .flatMap(candidates -> packAndPersistChunks(request, candidates, mapping))) .reduce(CompactionResult.NONE, CompactionResult::combine); From 8be5cb0c28aa0271d11f230043155ec29595a0a5 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:24:46 +0530 Subject: [PATCH 12/24] JAMES-4231 Compaction: Window candidates by cumulative byte size --- .../compaction/BlobCompactionAlgorithm.java | 51 +++++++++---------- .../BlobCompactionAlgorithmTest.java | 28 ++++++++++ 2 files changed, 51 insertions(+), 28 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index a6f6f9af630..3a60e47a659 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -29,6 +29,7 @@ import java.util.Objects; import java.util.Optional; import java.util.Set; +import java.util.concurrent.atomic.AtomicLong; import org.apache.james.blob.api.BlobId; import org.apache.james.blob.api.BlobReferenceSource; @@ -154,7 +155,7 @@ public Mono initialCompact(CompactionRequest request) { .map(family -> rawStore.listBlobs(request.bucketName(), family + "_" + request.generation() + "_")) .orElseGet(() -> rawStore.listBlobs(request.bucketName())); - return Flux.from(candidatesListing) + Flux packableCandidates = Flux.from(candidatesListing) .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) .filter(blobId -> !ChunkId.isChunkRef(blobId)) .filter(mapping::containsKey) @@ -164,15 +165,29 @@ public Mono initialCompact(CompactionRequest request) { LOGGER.warn("Failed reading candidate blob {}", blobId.asString(), error); return Mono.empty(); }), 16) - .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()) - .window(DEFAULT_CANDIDATE_BATCH_SIZE) - .concatMap(windowFlux -> windowFlux - .collectList() - .flatMap(candidates -> packAndPersistChunks(request, candidates, mapping))) + .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()); + + return windowByCumulativeSize(packableCandidates, request.configuration().chunkTargetSize()) + .concatMap(batch -> persistChunkBatch(request, batch, mapping)) .reduce(CompactionResult.NONE, CompactionResult::combine); }); } + static Flux> windowByCumulativeSize(Flux candidates, long targetSize) { + return Flux.defer(() -> { + AtomicLong currentSize = new AtomicLong(0L); + return candidates.bufferUntil(candidate -> { + long payloadSize = candidate.payload().length; + if (currentSize.get() > 0 && currentSize.get() + payloadSize > targetSize) { + currentSize.set(payloadSize); + return true; + } + currentSize.addAndGet(payloadSize); + return false; + }, true); + }); + } + /** * Executes GC compaction (purge dead slots, merge small chunks, delete orphan chunks). *

@@ -220,27 +235,7 @@ private Mono packAndPersistChunks(CompactionRequest request, if (candidates.isEmpty()) { return Mono.just(CompactionResult.NONE); } - - List> batches = new ArrayList<>(); - List currentBatch = new ArrayList<>(); - long currentBatchSize = 0; - - for (CandidateBlob candidate : candidates) { - if (!currentBatch.isEmpty() && currentBatchSize + candidate.payload.length > request.configuration().chunkTargetSize()) { - batches.add(currentBatch); - currentBatch = new ArrayList<>(); - currentBatchSize = 0; - } - currentBatch.add(candidate); - currentBatchSize += candidate.payload.length; - } - if (!currentBatch.isEmpty()) { - batches.add(currentBatch); - } - - return Flux.fromIterable(batches) - .concatMap(batch -> persistChunkBatch(request, batch, mapping)) - .reduce(CompactionResult.NONE, CompactionResult::combine); + return persistChunkBatch(request, candidates, mapping); } private Mono persistChunkBatch(CompactionRequest request, @@ -609,7 +604,7 @@ static int extractFamily(String blobIdStr) { return 1; } - private record CandidateBlob(BlobId blobId, byte[] payload, BlobMetadata metadata) {} + record CandidateBlob(BlobId blobId, byte[] payload, BlobMetadata metadata) {} private record ExistingChunk(BlobId chunkBlobId, long totalChunkSize, ChunkFooter footer) {} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index 964057d8fcf..3beef46f144 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -616,4 +616,32 @@ void initialCompactShouldUsePrefixPushdownToAvoidListingOtherFamiliesOrGeneratio assertThat(replacements).hasSize(1); assertThat(replacements.get(0).oldId()).isEqualTo(targetBlob); } + + @Test + void windowByCumulativeSizeShouldPartitionCandidatesWhenSizeExceedsTarget() { + BlobStoreDAO.BlobMetadata empty = BlobStoreDAO.BlobMetadata.empty(); + List candidates = List.of( + new BlobCompactionAlgorithm.CandidateBlob(new PlainBlobId("1_2_b1"), new byte[30], empty), + new BlobCompactionAlgorithm.CandidateBlob(new PlainBlobId("1_2_b2"), new byte[40], empty), + new BlobCompactionAlgorithm.CandidateBlob(new PlainBlobId("1_2_b3"), new byte[50], empty), + new BlobCompactionAlgorithm.CandidateBlob(new PlainBlobId("1_2_b4"), new byte[40], empty), + new BlobCompactionAlgorithm.CandidateBlob(new PlainBlobId("1_2_b5"), new byte[20], empty) + ); + + List> batches = BlobCompactionAlgorithm + .windowByCumulativeSize(Flux.fromIterable(candidates), 80L) + .collectList() + .block(); + + assertThat(batches).hasSize(3); + // Batch 1: 30 + 40 = 70 bytes <= 80 + assertThat(batches.get(0)).extracting(c -> c.blobId().asString()) + .containsExactly("1_2_b1", "1_2_b2"); + // Batch 2: 50 bytes <= 80 + assertThat(batches.get(1)).extracting(c -> c.blobId().asString()) + .containsExactly("1_2_b3"); + // Batch 3: 40 + 20 = 60 bytes <= 80 + assertThat(batches.get(2)).extracting(c -> c.blobId().asString()) + .containsExactly("1_2_b4", "1_2_b5"); + } } From a8b25180e7cd9661a572c0879dd1e3e3ce464551 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:28:26 +0530 Subject: [PATCH 13/24] JAMES-4231 Compaction: Skip packing when candidate count is one --- .../compaction/BlobCompactionAlgorithm.java | 7 ++- .../BlobCompactionAlgorithmTest.java | 61 +++++++++++++++---- 2 files changed, 54 insertions(+), 14 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index 3a60e47a659..fcb7346ec0b 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -168,6 +168,7 @@ public Mono initialCompact(CompactionRequest request) { .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()); return windowByCumulativeSize(packableCandidates, request.configuration().chunkTargetSize()) + .filter(batch -> batch.size() > 1) .concatMap(batch -> persistChunkBatch(request, batch, mapping)) .reduce(CompactionResult.NONE, CompactionResult::combine); }); @@ -232,7 +233,7 @@ public Mono gcCompact(CompactionRequest request) { private Mono packAndPersistChunks(CompactionRequest request, List candidates, Map> mapping) { - if (candidates.isEmpty()) { + if (candidates.size() <= 1) { return Mono.just(CompactionResult.NONE); } return persistChunkBatch(request, candidates, mapping); @@ -241,6 +242,10 @@ private Mono packAndPersistChunks(CompactionRequest request, private Mono persistChunkBatch(CompactionRequest request, List batch, Map> mapping) { + if (batch.size() <= 1) { + LOGGER.debug("Skipping compaction for single candidate batch"); + return Mono.just(CompactionResult.NONE); + } int family = request.family().orElseGet(() -> extractFamily(batch.get(0).blobId.asString())); ChunkId chunkId = ChunkId.ofChunk(family, request.generation()); diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index 3beef46f144..e2b4541b746 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -172,13 +172,17 @@ void initialCompactionShouldPackSmallBlobsAndSkipLargeBlobs() { @Test void initialCompactionPreservesDeduplication() { BlobId sharedBlobId = new PlainBlobId("1_2_shared"); + BlobId otherBlobId = new PlainBlobId("1_2_other"); byte[] sharedPayload = "Shared email attachment body".getBytes(StandardCharsets.UTF_8); + byte[] otherPayload = "Other email attachment body".getBytes(StandardCharsets.UTF_8); Mono.from(rawStore.save(TEST_BUCKET, sharedBlobId, BlobStoreDAO.BytesBlob.of(sharedPayload))).block(); + Mono.from(rawStore.save(TEST_BUCKET, otherBlobId, BlobStoreDAO.BytesBlob.of(otherPayload))).block(); // Two messages reference the SAME blob mappingSource.add(sharedBlobId, "msg-A"); mappingSource.add(sharedBlobId, "msg-B"); + mappingSource.add(otherBlobId, "msg-C"); CompactionRequest request = CompactionRequest.builder() .bucketName(TEST_BUCKET) @@ -188,16 +192,19 @@ void initialCompactionPreservesDeduplication() { CompactionResult result = testee.initialCompact(request).block(); - assertThat(result.packedBlobs()).isEqualTo(1); + assertThat(result.packedBlobs()).isEqualTo(2); assertThat(result.chunksWritten()).isEqualTo(1); - // One replacement covering both messages - assertThat(recordingUpdater.getReplacements()).hasSize(1); - RecordingBlobIdUpdater.Replacement rep = recordingUpdater.getReplacements().get(0); - assertThat(rep.oldId()).isEqualTo(sharedBlobId); - assertThat(rep.messageIds()).containsExactlyInAnyOrder("msg-A", "msg-B"); + // Replacements include one covering both messages for sharedBlobId + List reps = recordingUpdater.getReplacements(); + assertThat(reps).hasSize(2); + RecordingBlobIdUpdater.Replacement sharedRep = reps.stream() + .filter(r -> r.oldId().equals(sharedBlobId)) + .findFirst() + .orElseThrow(); + assertThat(sharedRep.messageIds()).containsExactlyInAnyOrder("msg-A", "msg-B"); - assertThat(readDecompressed(rep.newId())).isEqualTo(sharedPayload); + assertThat(readDecompressed(sharedRep.newId())).isEqualTo(sharedPayload); } @Test @@ -590,15 +597,18 @@ void initialCompactShouldPreserveCustomBlobMetadata() { @Test void initialCompactShouldUsePrefixPushdownToAvoidListingOtherFamiliesOrGenerations() { - BlobId targetBlob = new PlainBlobId("1_2_target"); + BlobId targetBlob1 = new PlainBlobId("1_2_target1"); + BlobId targetBlob2 = new PlainBlobId("1_2_target2"); BlobId otherGenBlob = new PlainBlobId("1_3_otherGen"); BlobId otherFamilyBlob = new PlainBlobId("2_2_otherFamily"); - Mono.from(rawStore.save(TEST_BUCKET, targetBlob, BlobStoreDAO.BytesBlob.of("target"))).block(); + Mono.from(rawStore.save(TEST_BUCKET, targetBlob1, BlobStoreDAO.BytesBlob.of("target1"))).block(); + Mono.from(rawStore.save(TEST_BUCKET, targetBlob2, BlobStoreDAO.BytesBlob.of("target2"))).block(); Mono.from(rawStore.save(TEST_BUCKET, otherGenBlob, BlobStoreDAO.BytesBlob.of("otherGen"))).block(); Mono.from(rawStore.save(TEST_BUCKET, otherFamilyBlob, BlobStoreDAO.BytesBlob.of("otherFamily"))).block(); - mappingSource.add(targetBlob, "msg-target"); + mappingSource.add(targetBlob1, "msg-target1"); + mappingSource.add(targetBlob2, "msg-target2"); mappingSource.add(otherGenBlob, "msg-otherGen"); mappingSource.add(otherFamilyBlob, "msg-otherFamily"); @@ -610,11 +620,36 @@ void initialCompactShouldUsePrefixPushdownToAvoidListingOtherFamiliesOrGeneratio .build(); CompactionResult result = testee.initialCompact(request).block(); - assertThat(result.packedBlobs()).isEqualTo(1); + assertThat(result.packedBlobs()).isEqualTo(2); List replacements = recordingUpdater.getReplacements(); - assertThat(replacements).hasSize(1); - assertThat(replacements.get(0).oldId()).isEqualTo(targetBlob); + assertThat(replacements).hasSize(2); + assertThat(replacements).extracting(r -> r.oldId().asString()) + .containsExactlyInAnyOrder("1_2_target1", "1_2_target2"); + } + + @Test + void initialCompactShouldSkipPackingWhenCandidateCountIsOne() { + BlobId singleBlob = new PlainBlobId("1_2_single"); + Mono.from(rawStore.save(TEST_BUCKET, singleBlob, BlobStoreDAO.BytesBlob.of("single-blob-payload"))).block(); + mappingSource.add(singleBlob, "msg-single"); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder().chunkTargetSize(100_000).build()) + .build(); + + CompactionResult result = testee.initialCompact(request).block(); + + assertThat(result.packedBlobs()).isEqualTo(0); + assertThat(result.chunksWritten()).isEqualTo(0); + assertThat(recordingUpdater.getReplacements()).isEmpty(); + + // Standalone blob remains untouched in raw storage + assertThat(Mono.from(rawStore.readBytes(TEST_BUCKET, singleBlob)).block().payload()) + .isEqualTo("single-blob-payload".getBytes(StandardCharsets.UTF_8)); } @Test From 1031d6ece6fee90cbb19d23cb3b50aba2454598f Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:34:27 +0530 Subject: [PATCH 14/24] JAMES-4231 Compaction: Extract reference update and deletion into dedicated method --- .../compaction/BlobCompactionAlgorithm.java | 61 +++++++++++-------- 1 file changed, 35 insertions(+), 26 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index fcb7346ec0b..ae7024997b4 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -35,6 +35,7 @@ import org.apache.james.blob.api.BlobReferenceSource; import org.apache.james.blob.api.BlobStoreDAO; import org.apache.james.blob.api.BlobStoreDAO.BlobMetadata; +import org.apache.james.blob.api.BucketName; import org.apache.james.blob.api.ObjectStoreIOException; import org.apache.james.blob.compaction.BlobReferenceMappingSource.BlobIdMessageIdMapping; import org.apache.james.blob.compaction.ChunkFormat.BlobSlotContent; @@ -262,32 +263,40 @@ private Mono persistChunkBatch(CompactionRequest request, // 1. Save chunk to raw storage return Mono.from(rawStore.save(request.bucketName(), chunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))) - // 2. Update source-of-truth table references - .then(Flux.range(0, batch.size()) - .concatMap(i -> { - CandidateBlob candidate = batch.get(i); - SlotRange range = writeResult.slotRanges().get(i); - ChunkId slotRef = ChunkId.slotRef(chunkId, range.offset(), range.limit()); - Collection messageIds = mapping.getOrDefault(candidate.blobId, Set.of()); - return blobIdUpdater.replaceReferences(candidate.blobId, slotRef, messageIds) - .thenReturn(Optional.of(candidate)) - .onErrorResume(e -> { - LOGGER.error("Failed to update references for candidate blob {}, skipping deletion of original blob", candidate.blobId.asString(), e); - return Mono.just(Optional.empty()); - }); - }) - .flatMap(opt -> opt.map(Flux::just).orElseGet(Flux::empty)) - .collectList() - .flatMap(successfulCandidates -> Flux.fromIterable(successfulCandidates) - // 3. Delete original standalone blobs ONLY for successfully updated candidates - .concatMap(candidate -> Mono.from(rawStore.delete(request.bucketName(), candidate.blobId))) - .then() - .thenReturn(CompactionResult.builder() - .packedBlobs(successfulCandidates.size()) - .packedBytes(successfulCandidates.stream().mapToLong(c -> c.payload.length).sum()) - .chunksWritten(1) - .freedBytes(successfulCandidates.stream().mapToLong(c -> c.payload.length).sum()) - .build()))); + // 2. Update source-of-truth table references and delete original blobs + .then(updateReferencesAndDeleteOriginalBlobs(request.bucketName(), chunkId, batch, writeResult.slotRanges(), mapping)); + } + + private Mono updateReferencesAndDeleteOriginalBlobs(BucketName bucketName, + ChunkId chunkId, + List batch, + List slotRanges, + Map> mapping) { + return Flux.range(0, batch.size()) + .concatMap(i -> { + CandidateBlob candidate = batch.get(i); + SlotRange range = slotRanges.get(i); + ChunkId slotRef = ChunkId.slotRef(chunkId, range.offset(), range.limit()); + Collection messageIds = mapping.getOrDefault(candidate.blobId, Set.of()); + return blobIdUpdater.replaceReferences(candidate.blobId, slotRef, messageIds) + .thenReturn(Optional.of(candidate)) + .onErrorResume(e -> { + LOGGER.error("Failed to update references for candidate blob {}, skipping deletion of original blob", candidate.blobId.asString(), e); + return Mono.just(Optional.empty()); + }); + }) + .flatMap(opt -> opt.map(Flux::just).orElseGet(Flux::empty)) + .collectList() + .flatMap(successfulCandidates -> Flux.fromIterable(successfulCandidates) + // Delete original standalone blobs ONLY for successfully updated candidates + .concatMap(candidate -> Mono.from(rawStore.delete(bucketName, candidate.blobId))) + .then() + .thenReturn(CompactionResult.builder() + .packedBlobs(successfulCandidates.size()) + .packedBytes(successfulCandidates.stream().mapToLong(c -> c.payload.length).sum()) + .chunksWritten(1) + .freedBytes(successfulCandidates.stream().mapToLong(c -> c.payload.length).sum()) + .build())); } private Mono processExistingChunks(CompactionRequest request, From fe3abd0b278fd927990c0c6ca3258631de6182f2 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 13:46:56 +0530 Subject: [PATCH 15/24] JAMES-4231 Compaction: Query reference mappings on-demand to prevent heap exhaustion --- .../CassandraBlobReferenceMappingSource.java | 14 ++ .../compaction/BlobCompactionAlgorithm.java | 147 ++++++++++-------- .../BlobReferenceMappingSource.java | 16 +- .../compaction/CompactionConfiguration.java | 26 +++- .../BlobCompactionAlgorithmTest.java | 12 ++ 5 files changed, 145 insertions(+), 70 deletions(-) diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java index af536b0bb1a..eb99ec4ad42 100644 --- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobReferenceMappingSource.java @@ -25,7 +25,9 @@ import static org.apache.james.mailbox.cassandra.table.CassandraMessageV3Table.TABLE_NAME; import java.util.ArrayList; +import java.util.Collection; import java.util.List; +import java.util.Set; import jakarta.inject.Inject; @@ -39,6 +41,8 @@ import com.datastax.oss.driver.api.core.type.codec.TypeCodecs; import com.datastax.oss.driver.api.querybuilder.QueryBuilder; +import reactor.core.publisher.Flux; + public class CassandraBlobReferenceMappingSource implements BlobReferenceMappingSource { private final CassandraAsyncExecutor cassandraAsyncExecutor; private final BlobId.Factory blobIdFactory; @@ -71,4 +75,14 @@ public Publisher listBlobIdMessageIdMappings() { return mappings; }); } + + @Override + public Publisher loadReferencesFor(Collection blobIds) { + if (blobIds.isEmpty()) { + return Flux.empty(); + } + Set targetBlobIds = Set.copyOf(blobIds); + return Flux.from(listBlobIdMessageIdMappings()) + .filter(mapping -> targetBlobIds.contains(mapping.blobId())); + } } diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java index ae7024997b4..16591f0a06e 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionAlgorithm.java @@ -93,13 +93,10 @@ * and partitions them into windows of {@value #DEFAULT_CANDIDATE_BATCH_SIZE} blobs. Candidate payloads are fetched * and packed chunk-by-chunk. Payloads are persisted and freed window-by-window, ensuring that candidate byte arrays * are never held in heap for the entire generation simultaneously. Candidate payload heap usage is bounded by - * {@code O(min(candidateBatchSize * avgBlobSize, chunkTargetSize))}. - *

  • Reference Mapping Memory Ceiling: {@code loadReferenceMapping()} materializes all live blob-to-messageId - * mappings for the generation into an in-memory multimap. Memory consumption is {@code O(liveGenerationReferences)} - * at approximately ~200 bytes per reference (~200MB heap for 1 million live references; ~2GB heap for 10 million). - * Because {@link BlobReferenceMappingSource} currently exposes a full stream without partition-paged query capabilities, - * this table is loaded per compaction pass. High-scale deployments exceeding tens of millions of live references per - * generation can introduce partition-paged reference lookups in future iterations.
    • + *
  • Reference Mapping Memory Bounds: Reference lookups are queried on-demand via + * {@link BlobReferenceMappingSource#loadReferencesFor(Collection)} in windowed batches bounded by + * {@link CompactionConfiguration#windowBatchSize()}, avoiding full generation materialization and preventing + * heap exhaustion even in high-scale deployments with tens of millions of active references.
    • *
  • GC Compaction Memory Bounds: {@link #gcCompact(CompactionRequest)} discovers chunks by reading only trailing * 64KB footers via HTTP ranged reads (metadata-only). Orphan chunks (100% dead slots) are deleted with 0 payload bytes read. * During chunk purge or merge, surviving live slots are streamed individually via HTTP ranged reads, strictly bounding @@ -145,34 +142,33 @@ public Mono compact(CompactionRequest request) { public Mono initialCompact(CompactionRequest request) { Preconditions.checkNotNull(request, "'request' must not be null"); - return loadReferenceMapping() - .flatMap(mapping -> { - if (mapping.isEmpty()) { - LOGGER.info("No blob references found in mapping source; skipping initial compaction for generation {}", request.generation()); - return Mono.just(CompactionResult.NONE); - } + Publisher candidatesListing = request.family() + .map(family -> rawStore.listBlobs(request.bucketName(), family + "_" + request.generation() + "_")) + .orElseGet(() -> rawStore.listBlobs(request.bucketName())); - Publisher candidatesListing = request.family() - .map(family -> rawStore.listBlobs(request.bucketName(), family + "_" + request.generation() + "_")) - .orElseGet(() -> rawStore.listBlobs(request.bucketName())); - - Flux packableCandidates = Flux.from(candidatesListing) - .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) - .filter(blobId -> !ChunkId.isChunkRef(blobId)) - .filter(mapping::containsKey) - .flatMap(blobId -> Mono.from(rawStore.readBytes(request.bucketName(), blobId)) - .map(bytesBlob -> new CandidateBlob(blobId, bytesBlob.payload(), bytesBlob.metadata())) - .onErrorResume(error -> { - LOGGER.warn("Failed reading candidate blob {}", blobId.asString(), error); - return Mono.empty(); - }), 16) - .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()); - - return windowByCumulativeSize(packableCandidates, request.configuration().chunkTargetSize()) - .filter(batch -> batch.size() > 1) - .concatMap(batch -> persistChunkBatch(request, batch, mapping)) - .reduce(CompactionResult.NONE, CompactionResult::combine); - }); + return Flux.from(candidatesListing) + .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) + .filter(blobId -> !ChunkId.isChunkRef(blobId)) + .buffer(request.configuration().windowBatchSize()) + .concatMap(batchBlobIds -> loadReferenceMappingFor(batchBlobIds) + .flatMap(mapping -> { + Flux packableCandidates = Flux.fromIterable(batchBlobIds) + .filter(mapping::containsKey) + .flatMap(blobId -> Mono.from(rawStore.readBytes(request.bucketName(), blobId)) + .map(bytesBlob -> new CandidateBlob(blobId, bytesBlob.payload(), bytesBlob.metadata())) + .onErrorResume(error -> { + LOGGER.warn("Failed reading candidate blob {}", blobId.asString(), error); + return Mono.empty(); + }), 16) + .filter(candidate -> candidate.payload.length < request.configuration().maxPackableSize()); + + return windowByCumulativeSize(packableCandidates, request.configuration().chunkTargetSize()) + .filter(batch -> batch.size() > 1) + .concatMap(batch -> persistChunkBatch(request, batch, mapping)) + .reduce(CompactionResult.NONE, (r1, r2) -> r1.combine(r2)); + })) + .reduce(CompactionResult.NONE, (r1, r2) -> r1.combine(r2)) + .defaultIfEmpty(CompactionResult.NONE); } static Flux> windowByCumulativeSize(Flux candidates, long targetSize) { @@ -206,29 +202,34 @@ public Mono gcCompact(CompactionRequest request) { .map(family -> rawStore.listBlobs(request.bucketName(), family + "_" + request.generation() + "_chunk")) .orElseGet(() -> rawStore.listBlobs(request.bucketName())); - return loadReferenceMapping() - .flatMap(mapping -> Flux.from(chunkListing) - .filter(blobId -> ChunkId.isChunkRef(blobId) && blobId.asString().indexOf('~') == -1) - .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) - .flatMap(chunkBlobId -> Mono.from(rawStore.readRange(request.bucketName(), chunkBlobId, -65536, -1)) - .map(tailBlob -> { - try { - byte[] tailData = tailBlob.asBytes().payload(); - long totalSize = BlobStoreDAO.totalObjectSize(tailBlob); - ChunkFooter footer = ChunkFormat.readFooter(tailData, totalSize); - return new ExistingChunk(chunkBlobId, totalSize, footer); - } catch (IOException e) { - LOGGER.warn("Failed reading footer for chunk object {}", chunkBlobId.asString(), e); - return null; - } - }) - .filter(Objects::nonNull) - .onErrorResume(error -> { - LOGGER.warn("Failed reading chunk footer {}", chunkBlobId.asString(), error); - return Mono.empty(); - })) - .collectList() - .flatMap(existingChunks -> processExistingChunks(request, existingChunks, mapping))); + return Flux.from(chunkListing) + .filter(blobId -> ChunkId.isChunkRef(blobId) && blobId.asString().indexOf('~') == -1) + .filter(blobId -> matchesGenerationAndFamily(blobId.asString(), request.generation(), request.family())) + .flatMap(chunkBlobId -> Mono.from(rawStore.readRange(request.bucketName(), chunkBlobId, -65536, -1)) + .map(tailBlob -> { + try { + byte[] tailData = tailBlob.asBytes().payload(); + long totalSize = BlobStoreDAO.totalObjectSize(tailBlob); + ChunkFooter footer = ChunkFormat.readFooter(tailData, totalSize); + return new ExistingChunk(chunkBlobId, totalSize, footer); + } catch (IOException e) { + LOGGER.warn("Failed reading footer for chunk object {}", chunkBlobId.asString(), e); + return null; + } + }) + .filter(Objects::nonNull) + .onErrorResume(error -> { + LOGGER.warn("Failed reading chunk footer {}", chunkBlobId.asString(), error); + return Mono.empty(); + })) + .buffer(request.configuration().windowBatchSize()) + .concatMap(existingChunks -> { + List slotRefs = extractSlotRefs(existingChunks); + return loadReferenceMappingFor(slotRefs) + .flatMap(mapping -> processExistingChunks(request, existingChunks, mapping)); + }) + .reduce(CompactionResult.NONE, CompactionResult::combine) + .defaultIfEmpty(CompactionResult.NONE); } private Mono packAndPersistChunks(CompactionRequest request, @@ -558,17 +559,31 @@ private Set findMessageIdsForSlot(Map> mapping, Chun return Set.of(); } + private List extractSlotRefs(List chunks) { + List slotRefs = new ArrayList<>(); + for (ExistingChunk chunk : chunks) { + ChunkId chunkId = ChunkId.parseChunk(chunk.chunkBlobId.asString()); + ChunkFooter footer = chunk.footer; + List starts = footer.slotStarts(); + for (int i = 0; i < starts.size(); i++) { + long offset = starts.get(i); + long limit = footer.slotLength(i); + slotRefs.add(ChunkId.slotRef(chunkId, offset, limit)); + slotRefs.add(ChunkId.slotRef(chunkId.chunkBlobId().asString(), offset, 0L)); + } + } + return slotRefs; + } + /** - * Loads generation live references from {@link BlobReferenceMappingSource}. - *

    - * Operational ceiling: Materializes the generation's reference multimap into memory. - * Memory consumption is O(liveReferences) * ~200 bytes/reference. - * High-scale deployments with tens of millions of references can introduce partition-paged - * reference lookups in future iterations. - *

    + * Loads live references on-demand for the specified blob identifiers from {@link BlobReferenceMappingSource}. + * Memory consumption is strictly bounded by {@code O(batchSize * referencesPerBlob)}. */ - private Mono>> loadReferenceMapping() { - return Flux.from(mappingSource.listBlobIdMessageIdMappings()) + private Mono>> loadReferenceMappingFor(Collection blobIds) { + if (blobIds.isEmpty()) { + return Mono.just(Map.of()); + } + return Flux.from(mappingSource.loadReferencesFor(blobIds)) .collectMultimap(BlobIdMessageIdMapping::blobId, BlobIdMessageIdMapping::messageId) .map(multimap -> { Map> result = new HashMap<>(); diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java index 4088340d03f..beececc9cb8 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobReferenceMappingSource.java @@ -16,14 +16,19 @@ * specific language governing permissions and limitations * * under the License. * ****************************************************************/ - package org.apache.james.blob.compaction; +import java.util.Collection; +import java.util.Set; + import org.apache.james.blob.api.BlobId; import org.reactivestreams.Publisher; import com.google.common.base.Preconditions; +import reactor.core.publisher.Flux; + +@FunctionalInterface public interface BlobReferenceMappingSource { record BlobIdMessageIdMapping(BlobId blobId, String messageId) { public BlobIdMessageIdMapping { @@ -33,4 +38,13 @@ record BlobIdMessageIdMapping(BlobId blobId, String messageId) { } Publisher listBlobIdMessageIdMappings(); + + default Publisher loadReferencesFor(Collection blobIds) { + if (blobIds.isEmpty()) { + return Flux.empty(); + } + Set targetBlobIds = Set.copyOf(blobIds); + return Flux.from(listBlobIdMessageIdMappings()) + .filter(mapping -> targetBlobIds.contains(mapping.blobId())); + } } diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java index 972b7a3f256..fbf8f3285e6 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/CompactionConfiguration.java @@ -30,6 +30,7 @@ public class CompactionConfiguration { public static final double DEFAULT_PURGE_DEAD_RATIO = 0.1; // 10% public static final double DEFAULT_MERGE_DEAD_RATIO = 0.5; // 50% public static final double DEFAULT_GAIN_THRESHOLD = 0.1; // 10% + public static final int DEFAULT_WINDOW_BATCH_SIZE = 1000; public static final CompactionConfiguration DEFAULT = builder().build(); @@ -39,6 +40,7 @@ public static class Builder { private double purgeDeadRatio = DEFAULT_PURGE_DEAD_RATIO; private double mergeDeadRatio = DEFAULT_MERGE_DEAD_RATIO; private double gainThreshold = DEFAULT_GAIN_THRESHOLD; + private int windowBatchSize = DEFAULT_WINDOW_BATCH_SIZE; public Builder chunkTargetSize(long chunkTargetSize) { Preconditions.checkArgument(chunkTargetSize > 0, "'chunkTargetSize' must be strictly positive"); @@ -73,8 +75,14 @@ public Builder gainThreshold(double gainThreshold) { return this; } + public Builder windowBatchSize(int windowBatchSize) { + Preconditions.checkArgument(windowBatchSize > 0, "'windowBatchSize' must be strictly positive"); + this.windowBatchSize = windowBatchSize; + return this; + } + public CompactionConfiguration build() { - return new CompactionConfiguration(chunkTargetSize, maxPackableSize, purgeDeadRatio, mergeDeadRatio, gainThreshold); + return new CompactionConfiguration(chunkTargetSize, maxPackableSize, purgeDeadRatio, mergeDeadRatio, gainThreshold, windowBatchSize); } } @@ -87,13 +95,19 @@ public static Builder builder() { private final double purgeDeadRatio; private final double mergeDeadRatio; private final double gainThreshold; + private final int windowBatchSize; public CompactionConfiguration(long chunkTargetSize, long maxPackableSize, double purgeDeadRatio, double mergeDeadRatio, double gainThreshold) { + this(chunkTargetSize, maxPackableSize, purgeDeadRatio, mergeDeadRatio, gainThreshold, DEFAULT_WINDOW_BATCH_SIZE); + } + + public CompactionConfiguration(long chunkTargetSize, long maxPackableSize, double purgeDeadRatio, double mergeDeadRatio, double gainThreshold, int windowBatchSize) { this.chunkTargetSize = chunkTargetSize; this.maxPackableSize = maxPackableSize; this.purgeDeadRatio = purgeDeadRatio; this.mergeDeadRatio = mergeDeadRatio; this.gainThreshold = gainThreshold; + this.windowBatchSize = windowBatchSize; } public long chunkTargetSize() { @@ -116,6 +130,10 @@ public double gainThreshold() { return gainThreshold; } + public int windowBatchSize() { + return windowBatchSize; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -126,14 +144,15 @@ public boolean equals(Object o) { && maxPackableSize == that.maxPackableSize && Double.compare(that.purgeDeadRatio, purgeDeadRatio) == 0 && Double.compare(that.mergeDeadRatio, mergeDeadRatio) == 0 - && Double.compare(that.gainThreshold, gainThreshold) == 0; + && Double.compare(that.gainThreshold, gainThreshold) == 0 + && windowBatchSize == that.windowBatchSize; } return false; } @Override public int hashCode() { - return Objects.hash(chunkTargetSize, maxPackableSize, purgeDeadRatio, mergeDeadRatio, gainThreshold); + return Objects.hash(chunkTargetSize, maxPackableSize, purgeDeadRatio, mergeDeadRatio, gainThreshold, windowBatchSize); } @Override @@ -144,6 +163,7 @@ public String toString() { .add("purgeDeadRatio", purgeDeadRatio) .add("mergeDeadRatio", mergeDeadRatio) .add("gainThreshold", gainThreshold) + .add("windowBatchSize", windowBatchSize) .toString(); } } diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index e2b4541b746..b66be345745 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -91,6 +91,18 @@ public Publisher listBlobIdMessageIdMappings() { msgIds.forEach(msgId -> list.add(new BlobIdMessageIdMapping(blobId, msgId)))); return Flux.fromIterable(list); } + + @Override + public Publisher loadReferencesFor(Collection blobIds) { + List list = new ArrayList<>(); + for (BlobId blobId : blobIds) { + Set msgIds = mappings.get(blobId); + if (msgIds != null) { + msgIds.forEach(msgId -> list.add(new BlobIdMessageIdMapping(blobId, msgId))); + } + } + return Flux.fromIterable(list); + } } private MemoryBlobStoreDAO rawStore; From 08f4dbbe7ca3690401fba8a1a1f38675fa3ac0fd Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 14:00:08 +0530 Subject: [PATCH 16/24] JAMES-4231 Compaction: Split initial compaction and GC recompaction into distinct tasks --- .../compaction/BlobCompactionDTOModules.java | 16 ++ .../blob/compaction/GCBlobCompactionTask.java | 203 +++++++++++++++++ ...ompactionTaskAdditionalInformationDTO.java | 117 ++++++++++ .../compaction/GCBlobCompactionTaskDTO.java | 130 +++++++++++ .../compaction/InitialBlobCompactionTask.java | 213 ++++++++++++++++++ ...ompactionTaskAdditionalInformationDTO.java | 126 +++++++++++ .../InitialBlobCompactionTaskDTO.java | 112 +++++++++ ...ctionTaskAdditionalInformationDTOTest.java | 46 ++++ ...GCBlobCompactionTaskSerializationTest.java | 67 ++++++ ...ctionTaskAdditionalInformationDTOTest.java | 47 ++++ ...alBlobCompactionTaskSerializationTest.java | 65 ++++++ ...cBlobCompaction.additionalInformation.json | 10 + .../resources/json/gcBlobCompaction.task.json | 10 + ...lBlobCompaction.additionalInformation.json | 11 + .../json/initialBlobCompaction.task.json | 8 + .../blobstore/BlobCompactionModule.java | 32 +++ .../james/webadmin/routes/BlobRoutes.java | 33 ++- .../routes/BlobRoutesCompactionTest.java | 31 ++- 18 files changed, 1270 insertions(+), 7 deletions(-) create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTask.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTO.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTask.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTO.java create mode 100644 server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTOTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskSerializationTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTOTest.java create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskSerializationTest.java create mode 100644 server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.additionalInformation.json create mode 100644 server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.task.json create mode 100644 server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.additionalInformation.json create mode 100644 server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.task.json diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java index 2f66d5c13fb..fa1f3cfbba0 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java @@ -29,7 +29,23 @@ public static TaskDTOModule taskModul return BlobCompactionTaskDTO.module(algorithm, clock); } + public static TaskDTOModule initialCompactionTaskModule(BlobCompactionAlgorithm algorithm, Clock clock) { + return InitialBlobCompactionTaskDTO.module(algorithm, clock); + } + + public static TaskDTOModule gcCompactionTaskModule(BlobCompactionAlgorithm algorithm, Clock clock) { + return GCBlobCompactionTaskDTO.module(algorithm, clock); + } + public static AdditionalInformationDTOModule additionalInformationModule() { return BlobCompactionTaskAdditionalInformationDTO.SERIALIZATION_MODULE; } + + public static AdditionalInformationDTOModule initialAdditionalInformationModule() { + return InitialBlobCompactionTaskAdditionalInformationDTO.SERIALIZATION_MODULE; + } + + public static AdditionalInformationDTOModule gcAdditionalInformationModule() { + return GCBlobCompactionTaskAdditionalInformationDTO.SERIALIZATION_MODULE; + } } diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTask.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTask.java new file mode 100644 index 00000000000..c04c05233fa --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTask.java @@ -0,0 +1,203 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Clock; +import java.time.Instant; +import java.util.Objects; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicReference; + +import org.apache.james.task.Task; +import org.apache.james.task.TaskExecutionDetails; +import org.apache.james.task.TaskType; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.base.MoreObjects; +import com.google.common.base.Preconditions; + +public class GCBlobCompactionTask implements Task { + public static final TaskType TASK_TYPE = TaskType.of("GCBlobCompactionTask"); + private static final Logger LOGGER = LoggerFactory.getLogger(GCBlobCompactionTask.class); + + public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation { + private final Instant timestamp; + private final String bucketName; + private final long generation; + private final Optional family; + private final long deadPurged; + private final long mergedChunks; + private final long freedBytes; + + public AdditionalInformation(Instant timestamp, + String bucketName, + long generation, + Optional family, + long deadPurged, + long mergedChunks, + long freedBytes) { + this.timestamp = Preconditions.checkNotNull(timestamp, "'timestamp' must not be null"); + this.bucketName = Preconditions.checkNotNull(bucketName, "'bucketName' must not be null"); + this.generation = generation; + this.family = Preconditions.checkNotNull(family, "'family' must not be null"); + this.deadPurged = deadPurged; + this.mergedChunks = mergedChunks; + this.freedBytes = freedBytes; + } + + @Override + public Instant timestamp() { + return timestamp; + } + + public Instant getTimestamp() { + return timestamp; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public long getDeadPurged() { + return deadPurged; + } + + public long getMergedChunks() { + return mergedChunks; + } + + public long getFreedBytes() { + return freedBytes; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof AdditionalInformation that) { + return generation == that.generation + && deadPurged == that.deadPurged + && mergedChunks == that.mergedChunks + && freedBytes == that.freedBytes + && Objects.equals(timestamp, that.timestamp) + && Objects.equals(bucketName, that.bucketName) + && Objects.equals(family, that.family); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(timestamp, bucketName, generation, family, deadPurged, mergedChunks, freedBytes); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this) + .add("timestamp", timestamp) + .add("bucketName", bucketName) + .add("generation", generation) + .add("family", family) + .add("deadPurged", deadPurged) + .add("mergedChunks", mergedChunks) + .add("freedBytes", freedBytes) + .toString(); + } + } + + private final BlobCompactionAlgorithm algorithm; + private final CompactionRequest request; + private final Clock clock; + private final AtomicReference currentResult; + + public GCBlobCompactionTask(BlobCompactionAlgorithm algorithm, CompactionRequest request, Clock clock) { + this.algorithm = Preconditions.checkNotNull(algorithm, "'algorithm' must not be null"); + this.request = Preconditions.checkNotNull(request, "'request' must not be null"); + this.clock = Preconditions.checkNotNull(clock, "'clock' must not be null"); + this.currentResult = new AtomicReference<>(CompactionResult.NONE); + } + + @Override + public Result run() { + try { + CompactionResult result = algorithm.gcCompact(request).block(); + if (result != null) { + currentResult.set(result); + } + return Result.COMPLETED; + } catch (Exception e) { + LOGGER.error("Error while running GCBlobCompactionTask for generation {}", request.generation(), e); + return Result.PARTIAL; + } + } + + @Override + public TaskType type() { + return TASK_TYPE; + } + + @Override + public Optional details() { + CompactionResult res = currentResult.get(); + return Optional.of(new AdditionalInformation( + clock.instant(), + request.bucketName().asString(), + request.generation(), + request.family(), + res.deadPurged(), + res.mergedChunks(), + res.freedBytes() + )); + } + + public CompactionRequest getRequest() { + return request; + } + + public Clock getClock() { + return clock; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof GCBlobCompactionTask that) { + return Objects.equals(request, that.request); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(request); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTO.java new file mode 100644 index 00000000000..580868454d9 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTO.java @@ -0,0 +1,117 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Instant; +import java.util.Optional; + +import org.apache.james.json.DTOModule; +import org.apache.james.server.task.json.dto.AdditionalInformationDTO; +import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class GCBlobCompactionTaskAdditionalInformationDTO implements AdditionalInformationDTO { + + public static final AdditionalInformationDTOModule SERIALIZATION_MODULE = + DTOModule.forDomainObject(GCBlobCompactionTask.AdditionalInformation.class) + .convertToDTO(GCBlobCompactionTaskAdditionalInformationDTO.class) + .toDomainObjectConverter(dto -> + new GCBlobCompactionTask.AdditionalInformation( + dto.timestamp, + dto.bucketName, + dto.generation, + dto.family, + dto.deadPurged, + dto.mergedChunks, + dto.freedBytes)) + .toDTOConverter((domain, type) -> + new GCBlobCompactionTaskAdditionalInformationDTO( + type, + domain.getTimestamp(), + domain.getBucketName(), + domain.getGeneration(), + domain.getFamily(), + domain.getDeadPurged(), + domain.getMergedChunks(), + domain.getFreedBytes())) + .typeName(GCBlobCompactionTask.TASK_TYPE.asString()) + .withFactory(AdditionalInformationDTOModule::new); + + private final String type; + private final Instant timestamp; + private final String bucketName; + private final long generation; + private final Optional family; + private final long deadPurged; + private final long mergedChunks; + private final long freedBytes; + + public GCBlobCompactionTaskAdditionalInformationDTO(@JsonProperty("type") String type, + @JsonProperty("timestamp") Instant timestamp, + @JsonProperty("bucketName") String bucketName, + @JsonProperty("generation") long generation, + @JsonProperty("family") Optional family, + @JsonProperty("deadPurged") long deadPurged, + @JsonProperty("mergedChunks") long mergedChunks, + @JsonProperty("freedBytes") long freedBytes) { + this.type = type; + this.timestamp = timestamp; + this.bucketName = bucketName; + this.generation = generation; + this.family = family; + this.deadPurged = deadPurged; + this.mergedChunks = mergedChunks; + this.freedBytes = freedBytes; + } + + @Override + public String getType() { + return type; + } + + public Instant getTimestamp() { + return timestamp; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public long getDeadPurged() { + return deadPurged; + } + + public long getMergedChunks() { + return mergedChunks; + } + + public long getFreedBytes() { + return freedBytes; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java new file mode 100644 index 00000000000..816e228fafb --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java @@ -0,0 +1,130 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Clock; +import java.util.Optional; + +import org.apache.james.blob.api.BucketName; +import org.apache.james.json.DTOModule; +import org.apache.james.server.task.json.dto.TaskDTO; +import org.apache.james.server.task.json.dto.TaskDTOModule; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class GCBlobCompactionTaskDTO implements TaskDTO { + + private final String type; + private final String bucketName; + private final long generation; + private final Optional family; + private final Optional chunkTargetSize; + private final Optional purgeDeadRatio; + private final Optional mergeDeadRatio; + private final Optional gainThreshold; + + public GCBlobCompactionTaskDTO(@JsonProperty("type") String type, + @JsonProperty("bucketName") String bucketName, + @JsonProperty("generation") long generation, + @JsonProperty("family") Optional family, + @JsonProperty("chunkTargetSize") Optional chunkTargetSize, + @JsonProperty("purgeDeadRatio") Optional purgeDeadRatio, + @JsonProperty("mergeDeadRatio") Optional mergeDeadRatio, + @JsonProperty("gainThreshold") Optional gainThreshold) { + this.type = type; + this.bucketName = bucketName; + this.generation = generation; + this.family = family; + this.chunkTargetSize = chunkTargetSize; + this.purgeDeadRatio = purgeDeadRatio; + this.mergeDeadRatio = mergeDeadRatio; + this.gainThreshold = gainThreshold; + } + + public static TaskDTOModule module(BlobCompactionAlgorithm algorithm, Clock clock) { + return DTOModule.forDomainObject(GCBlobCompactionTask.class) + .convertToDTO(GCBlobCompactionTaskDTO.class) + .toDomainObjectConverter(dto -> { + CompactionConfiguration.Builder configBuilder = CompactionConfiguration.builder(); + dto.getChunkTargetSize().ifPresent(configBuilder::chunkTargetSize); + dto.getPurgeDeadRatio().ifPresent(configBuilder::purgeDeadRatio); + dto.getMergeDeadRatio().ifPresent(configBuilder::mergeDeadRatio); + dto.getGainThreshold().ifPresent(configBuilder::gainThreshold); + + CompactionRequest.Builder requestBuilder = CompactionRequest.builder() + .bucketName(BucketName.of(dto.getBucketName())) + .generation(dto.getGeneration()) + .configuration(configBuilder.build()); + + dto.getFamily().ifPresent(requestBuilder::family); + + return new GCBlobCompactionTask(algorithm, requestBuilder.build(), clock); + }) + .toDTOConverter((domain, type) -> { + CompactionRequest req = domain.getRequest(); + CompactionConfiguration conf = req.configuration(); + return new GCBlobCompactionTaskDTO( + type, + req.bucketName().asString(), + req.generation(), + req.family(), + Optional.of(conf.chunkTargetSize()), + Optional.of(conf.purgeDeadRatio()), + Optional.of(conf.mergeDeadRatio()), + Optional.of(conf.gainThreshold()) + ); + }) + .typeName(GCBlobCompactionTask.TASK_TYPE.asString()) + .withFactory(TaskDTOModule::new); + } + + @Override + public String getType() { + return type; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public Optional getChunkTargetSize() { + return chunkTargetSize; + } + + public Optional getPurgeDeadRatio() { + return purgeDeadRatio; + } + + public Optional getMergeDeadRatio() { + return mergeDeadRatio; + } + + public Optional getGainThreshold() { + return gainThreshold; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTask.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTask.java new file mode 100644 index 00000000000..6ecd7f046cc --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTask.java @@ -0,0 +1,213 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Clock; +import java.time.Instant; +import java.util.Objects; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicReference; + +import org.apache.james.task.Task; +import org.apache.james.task.TaskExecutionDetails; +import org.apache.james.task.TaskType; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.base.MoreObjects; +import com.google.common.base.Preconditions; + +public class InitialBlobCompactionTask implements Task { + public static final TaskType TASK_TYPE = TaskType.of("InitialBlobCompactionTask"); + private static final Logger LOGGER = LoggerFactory.getLogger(InitialBlobCompactionTask.class); + + public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation { + private final Instant timestamp; + private final String bucketName; + private final long generation; + private final Optional family; + private final long packedBlobs; + private final long packedBytes; + private final long chunksWritten; + private final long freedBytes; + + public AdditionalInformation(Instant timestamp, + String bucketName, + long generation, + Optional family, + long packedBlobs, + long packedBytes, + long chunksWritten, + long freedBytes) { + this.timestamp = Preconditions.checkNotNull(timestamp, "'timestamp' must not be null"); + this.bucketName = Preconditions.checkNotNull(bucketName, "'bucketName' must not be null"); + this.generation = generation; + this.family = Preconditions.checkNotNull(family, "'family' must not be null"); + this.packedBlobs = packedBlobs; + this.packedBytes = packedBytes; + this.chunksWritten = chunksWritten; + this.freedBytes = freedBytes; + } + + @Override + public Instant timestamp() { + return timestamp; + } + + public Instant getTimestamp() { + return timestamp; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public long getPackedBlobs() { + return packedBlobs; + } + + public long getPackedBytes() { + return packedBytes; + } + + public long getChunksWritten() { + return chunksWritten; + } + + public long getFreedBytes() { + return freedBytes; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof AdditionalInformation that) { + return generation == that.generation + && packedBlobs == that.packedBlobs + && packedBytes == that.packedBytes + && chunksWritten == that.chunksWritten + && freedBytes == that.freedBytes + && Objects.equals(timestamp, that.timestamp) + && Objects.equals(bucketName, that.bucketName) + && Objects.equals(family, that.family); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(timestamp, bucketName, generation, family, packedBlobs, packedBytes, chunksWritten, freedBytes); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this) + .add("timestamp", timestamp) + .add("bucketName", bucketName) + .add("generation", generation) + .add("family", family) + .add("packedBlobs", packedBlobs) + .add("packedBytes", packedBytes) + .add("chunksWritten", chunksWritten) + .add("freedBytes", freedBytes) + .toString(); + } + } + + private final BlobCompactionAlgorithm algorithm; + private final CompactionRequest request; + private final Clock clock; + private final AtomicReference currentResult; + + public InitialBlobCompactionTask(BlobCompactionAlgorithm algorithm, CompactionRequest request, Clock clock) { + this.algorithm = Preconditions.checkNotNull(algorithm, "'algorithm' must not be null"); + this.request = Preconditions.checkNotNull(request, "'request' must not be null"); + this.clock = Preconditions.checkNotNull(clock, "'clock' must not be null"); + this.currentResult = new AtomicReference<>(CompactionResult.NONE); + } + + @Override + public Result run() { + try { + CompactionResult result = algorithm.initialCompact(request).block(); + if (result != null) { + currentResult.set(result); + } + return Result.COMPLETED; + } catch (Exception e) { + LOGGER.error("Error while running InitialBlobCompactionTask for generation {}", request.generation(), e); + return Result.PARTIAL; + } + } + + @Override + public TaskType type() { + return TASK_TYPE; + } + + @Override + public Optional details() { + CompactionResult res = currentResult.get(); + return Optional.of(new AdditionalInformation( + clock.instant(), + request.bucketName().asString(), + request.generation(), + request.family(), + res.packedBlobs(), + res.packedBytes(), + res.chunksWritten(), + res.freedBytes() + )); + } + + public CompactionRequest getRequest() { + return request; + } + + public Clock getClock() { + return clock; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof InitialBlobCompactionTask that) { + return Objects.equals(request, that.request); + } + return false; + } + + @Override + public int hashCode() { + return Objects.hash(request); + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTO.java new file mode 100644 index 00000000000..cf273b4d5e2 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTO.java @@ -0,0 +1,126 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Instant; +import java.util.Optional; + +import org.apache.james.json.DTOModule; +import org.apache.james.server.task.json.dto.AdditionalInformationDTO; +import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class InitialBlobCompactionTaskAdditionalInformationDTO implements AdditionalInformationDTO { + + public static final AdditionalInformationDTOModule SERIALIZATION_MODULE = + DTOModule.forDomainObject(InitialBlobCompactionTask.AdditionalInformation.class) + .convertToDTO(InitialBlobCompactionTaskAdditionalInformationDTO.class) + .toDomainObjectConverter(dto -> + new InitialBlobCompactionTask.AdditionalInformation( + dto.timestamp, + dto.bucketName, + dto.generation, + dto.family, + dto.packedBlobs, + dto.packedBytes, + dto.chunksWritten, + dto.freedBytes)) + .toDTOConverter((domain, type) -> + new InitialBlobCompactionTaskAdditionalInformationDTO( + type, + domain.getTimestamp(), + domain.getBucketName(), + domain.getGeneration(), + domain.getFamily(), + domain.getPackedBlobs(), + domain.getPackedBytes(), + domain.getChunksWritten(), + domain.getFreedBytes())) + .typeName(InitialBlobCompactionTask.TASK_TYPE.asString()) + .withFactory(AdditionalInformationDTOModule::new); + + private final String type; + private final Instant timestamp; + private final String bucketName; + private final long generation; + private final Optional family; + private final long packedBlobs; + private final long packedBytes; + private final long chunksWritten; + private final long freedBytes; + + public InitialBlobCompactionTaskAdditionalInformationDTO(@JsonProperty("type") String type, + @JsonProperty("timestamp") Instant timestamp, + @JsonProperty("bucketName") String bucketName, + @JsonProperty("generation") long generation, + @JsonProperty("family") Optional family, + @JsonProperty("packedBlobs") long packedBlobs, + @JsonProperty("packedBytes") long packedBytes, + @JsonProperty("chunksWritten") long chunksWritten, + @JsonProperty("freedBytes") long freedBytes) { + this.type = type; + this.timestamp = timestamp; + this.bucketName = bucketName; + this.generation = generation; + this.family = family; + this.packedBlobs = packedBlobs; + this.packedBytes = packedBytes; + this.chunksWritten = chunksWritten; + this.freedBytes = freedBytes; + } + + @Override + public String getType() { + return type; + } + + public Instant getTimestamp() { + return timestamp; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public long getPackedBlobs() { + return packedBlobs; + } + + public long getPackedBytes() { + return packedBytes; + } + + public long getChunksWritten() { + return chunksWritten; + } + + public long getFreedBytes() { + return freedBytes; + } +} diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java new file mode 100644 index 00000000000..78961b05c42 --- /dev/null +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java @@ -0,0 +1,112 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Clock; +import java.util.Optional; + +import org.apache.james.blob.api.BucketName; +import org.apache.james.json.DTOModule; +import org.apache.james.server.task.json.dto.TaskDTO; +import org.apache.james.server.task.json.dto.TaskDTOModule; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class InitialBlobCompactionTaskDTO implements TaskDTO { + + private final String type; + private final String bucketName; + private final long generation; + private final Optional family; + private final Optional chunkTargetSize; + private final Optional maxPackableSize; + + public InitialBlobCompactionTaskDTO(@JsonProperty("type") String type, + @JsonProperty("bucketName") String bucketName, + @JsonProperty("generation") long generation, + @JsonProperty("family") Optional family, + @JsonProperty("chunkTargetSize") Optional chunkTargetSize, + @JsonProperty("maxPackableSize") Optional maxPackableSize) { + this.type = type; + this.bucketName = bucketName; + this.generation = generation; + this.family = family; + this.chunkTargetSize = chunkTargetSize; + this.maxPackableSize = maxPackableSize; + } + + public static TaskDTOModule module(BlobCompactionAlgorithm algorithm, Clock clock) { + return DTOModule.forDomainObject(InitialBlobCompactionTask.class) + .convertToDTO(InitialBlobCompactionTaskDTO.class) + .toDomainObjectConverter(dto -> { + CompactionConfiguration.Builder configBuilder = CompactionConfiguration.builder(); + dto.getChunkTargetSize().ifPresent(configBuilder::chunkTargetSize); + dto.getMaxPackableSize().ifPresent(configBuilder::maxPackableSize); + + CompactionRequest.Builder requestBuilder = CompactionRequest.builder() + .bucketName(BucketName.of(dto.getBucketName())) + .generation(dto.getGeneration()) + .configuration(configBuilder.build()); + + dto.getFamily().ifPresent(requestBuilder::family); + + return new InitialBlobCompactionTask(algorithm, requestBuilder.build(), clock); + }) + .toDTOConverter((domain, type) -> { + CompactionRequest req = domain.getRequest(); + CompactionConfiguration conf = req.configuration(); + return new InitialBlobCompactionTaskDTO( + type, + req.bucketName().asString(), + req.generation(), + req.family(), + Optional.of(conf.chunkTargetSize()), + Optional.of(conf.maxPackableSize()) + ); + }) + .typeName(InitialBlobCompactionTask.TASK_TYPE.asString()) + .withFactory(TaskDTOModule::new); + } + + @Override + public String getType() { + return type; + } + + public String getBucketName() { + return bucketName; + } + + public long getGeneration() { + return generation; + } + + public Optional getFamily() { + return family; + } + + public Optional getChunkTargetSize() { + return chunkTargetSize; + } + + public Optional getMaxPackableSize() { + return maxPackableSize; + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTOTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTOTest.java new file mode 100644 index 00000000000..fe1875cb5dc --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskAdditionalInformationDTOTest.java @@ -0,0 +1,46 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Instant; +import java.util.Optional; + +import org.apache.james.JsonSerializationVerifier; +import org.apache.james.util.ClassLoaderUtils; +import org.junit.jupiter.api.Test; + +class GCBlobCompactionTaskAdditionalInformationDTOTest { + + @Test + void shouldMatchJsonSerializationContract() throws Exception { + JsonSerializationVerifier.dtoModule(BlobCompactionDTOModules.gcAdditionalInformationModule()) + .bean(new GCBlobCompactionTask.AdditionalInformation( + Instant.parse("2020-01-01T00:00:00Z"), + "default", + 2L, + Optional.of(1), + 2L, + 2L, + 512L + )) + .json(ClassLoaderUtils.getSystemResourceAsString("json/gcBlobCompaction.additionalInformation.json")) + .verify(); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskSerializationTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskSerializationTest.java new file mode 100644 index 00000000000..3e4f9b3bf20 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/GCBlobCompactionTaskSerializationTest.java @@ -0,0 +1,67 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.mockito.Mockito.mock; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; + +import org.apache.james.JsonSerializationVerifier; +import org.apache.james.blob.api.BucketName; +import org.apache.james.util.ClassLoaderUtils; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +class GCBlobCompactionTaskSerializationTest { + private BlobCompactionAlgorithm algorithm; + private Clock clock; + + @BeforeEach + void setUp() { + algorithm = mock(BlobCompactionAlgorithm.class); + clock = Clock.fixed(Instant.parse("2020-01-01T00:00:00Z"), ZoneOffset.UTC); + } + + @Test + void shouldMatchJsonSerializationContract() throws Exception { + CompactionConfiguration config = CompactionConfiguration.builder() + .chunkTargetSize(100 * 1024 * 1024L) + .purgeDeadRatio(0.1) + .mergeDeadRatio(0.5) + .gainThreshold(0.1) + .build(); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(BucketName.DEFAULT) + .generation(2L) + .family(1) + .configuration(config) + .build(); + + GCBlobCompactionTask task = new GCBlobCompactionTask(algorithm, request, clock); + + JsonSerializationVerifier.dtoModule(BlobCompactionDTOModules.gcCompactionTaskModule(algorithm, clock)) + .bean(task) + .json(ClassLoaderUtils.getSystemResourceAsString("json/gcBlobCompaction.task.json")) + .verify(); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTOTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTOTest.java new file mode 100644 index 00000000000..6aeac367025 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskAdditionalInformationDTOTest.java @@ -0,0 +1,47 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import java.time.Instant; +import java.util.Optional; + +import org.apache.james.JsonSerializationVerifier; +import org.apache.james.util.ClassLoaderUtils; +import org.junit.jupiter.api.Test; + +class InitialBlobCompactionTaskAdditionalInformationDTOTest { + + @Test + void shouldMatchJsonSerializationContract() throws Exception { + JsonSerializationVerifier.dtoModule(BlobCompactionDTOModules.initialAdditionalInformationModule()) + .bean(new InitialBlobCompactionTask.AdditionalInformation( + Instant.parse("2020-01-01T00:00:00Z"), + "default", + 2L, + Optional.of(1), + 10L, + 1024L, + 1L, + 512L + )) + .json(ClassLoaderUtils.getSystemResourceAsString("json/initialBlobCompaction.additionalInformation.json")) + .verify(); + } +} diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskSerializationTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskSerializationTest.java new file mode 100644 index 00000000000..8e196f090f8 --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskSerializationTest.java @@ -0,0 +1,65 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.mockito.Mockito.mock; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; + +import org.apache.james.JsonSerializationVerifier; +import org.apache.james.blob.api.BucketName; +import org.apache.james.util.ClassLoaderUtils; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +class InitialBlobCompactionTaskSerializationTest { + private BlobCompactionAlgorithm algorithm; + private Clock clock; + + @BeforeEach + void setUp() { + algorithm = mock(BlobCompactionAlgorithm.class); + clock = Clock.fixed(Instant.parse("2020-01-01T00:00:00Z"), ZoneOffset.UTC); + } + + @Test + void shouldMatchJsonSerializationContract() throws Exception { + CompactionConfiguration config = CompactionConfiguration.builder() + .chunkTargetSize(100 * 1024 * 1024L) + .maxPackableSize(1024 * 1024L) + .build(); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(BucketName.DEFAULT) + .generation(2L) + .family(1) + .configuration(config) + .build(); + + InitialBlobCompactionTask task = new InitialBlobCompactionTask(algorithm, request, clock); + + JsonSerializationVerifier.dtoModule(BlobCompactionDTOModules.initialCompactionTaskModule(algorithm, clock)) + .bean(task) + .json(ClassLoaderUtils.getSystemResourceAsString("json/initialBlobCompaction.task.json")) + .verify(); + } +} diff --git a/server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.additionalInformation.json b/server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.additionalInformation.json new file mode 100644 index 00000000000..c10b50cc1b2 --- /dev/null +++ b/server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.additionalInformation.json @@ -0,0 +1,10 @@ +{ + "type": "GCBlobCompactionTask", + "timestamp": "2020-01-01T00:00:00Z", + "bucketName": "default", + "generation": 2, + "family": 1, + "deadPurged": 2, + "mergedChunks": 2, + "freedBytes": 512 +} diff --git a/server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.task.json b/server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.task.json new file mode 100644 index 00000000000..e10539302f8 --- /dev/null +++ b/server/blob/blob-compaction/src/test/resources/json/gcBlobCompaction.task.json @@ -0,0 +1,10 @@ +{ + "type": "GCBlobCompactionTask", + "bucketName": "default", + "generation": 2, + "family": 1, + "chunkTargetSize": 104857600, + "purgeDeadRatio": 0.1, + "mergeDeadRatio": 0.5, + "gainThreshold": 0.1 +} diff --git a/server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.additionalInformation.json b/server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.additionalInformation.json new file mode 100644 index 00000000000..07ee5c4d6dd --- /dev/null +++ b/server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.additionalInformation.json @@ -0,0 +1,11 @@ +{ + "type": "InitialBlobCompactionTask", + "timestamp": "2020-01-01T00:00:00Z", + "bucketName": "default", + "generation": 2, + "family": 1, + "packedBlobs": 10, + "packedBytes": 1024, + "chunksWritten": 1, + "freedBytes": 512 +} diff --git a/server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.task.json b/server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.task.json new file mode 100644 index 00000000000..94b15d661c5 --- /dev/null +++ b/server/blob/blob-compaction/src/test/resources/json/initialBlobCompaction.task.json @@ -0,0 +1,8 @@ +{ + "type": "InitialBlobCompactionTask", + "bucketName": "default", + "generation": 2, + "family": 1, + "chunkTargetSize": 104857600, + "maxPackableSize": 1048576 +} diff --git a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java index f4202870fbb..9e2950af9bc 100644 --- a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java +++ b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java @@ -90,14 +90,46 @@ public Optional optionalBlobCompactionAlgorithm(Injecto return BlobCompactionDTOModules.taskModule(algorithm, clock); } + @ProvidesIntoSet + public TaskDTOModule initialBlobCompactionTask(BlobCompactionAlgorithm algorithm, Clock clock) { + return BlobCompactionDTOModules.initialCompactionTaskModule(algorithm, clock); + } + + @ProvidesIntoSet + public TaskDTOModule gcBlobCompactionTask(BlobCompactionAlgorithm algorithm, Clock clock) { + return BlobCompactionDTOModules.gcCompactionTaskModule(algorithm, clock); + } + @ProvidesIntoSet public AdditionalInformationDTOModule blobCompactionAdditionalInformation() { return BlobCompactionDTOModules.additionalInformationModule(); } + @ProvidesIntoSet + public AdditionalInformationDTOModule initialBlobCompactionAdditionalInformation() { + return BlobCompactionDTOModules.initialAdditionalInformationModule(); + } + + @ProvidesIntoSet + public AdditionalInformationDTOModule gcBlobCompactionAdditionalInformation() { + return BlobCompactionDTOModules.gcAdditionalInformationModule(); + } + @Named(DTOModuleInjections.WEBADMIN_DTO) @ProvidesIntoSet public AdditionalInformationDTOModule webAdminBlobCompactionAdditionalInformation() { return BlobCompactionDTOModules.additionalInformationModule(); } + + @Named(DTOModuleInjections.WEBADMIN_DTO) + @ProvidesIntoSet + public AdditionalInformationDTOModule webAdminInitialBlobCompactionAdditionalInformation() { + return BlobCompactionDTOModules.initialAdditionalInformationModule(); + } + + @Named(DTOModuleInjections.WEBADMIN_DTO) + @ProvidesIntoSet + public AdditionalInformationDTOModule webAdminGCBlobCompactionAdditionalInformation() { + return BlobCompactionDTOModules.gcAdditionalInformationModule(); + } } diff --git a/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java b/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java index 97f53ab5cfc..431a853ee43 100644 --- a/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java +++ b/server/protocols/webadmin/webadmin-data/src/main/java/org/apache/james/webadmin/routes/BlobRoutes.java @@ -34,6 +34,8 @@ import org.apache.james.blob.compaction.BlobCompactionAlgorithm; import org.apache.james.blob.compaction.BlobCompactionTask; import org.apache.james.blob.compaction.CompactionRequest; +import org.apache.james.blob.compaction.GCBlobCompactionTask; +import org.apache.james.blob.compaction.InitialBlobCompactionTask; import org.apache.james.server.blob.deduplication.BlobGCTask; import org.apache.james.server.blob.deduplication.GenerationAwareBlobId; import org.apache.james.task.Task; @@ -112,13 +114,37 @@ public Task delete(Request request) { if ("unreferenced".equals(scope)) { return gcUnreferenced(request); } + if ("initial-compaction".equals(scope)) { + return initialCompact(request); + } + if ("gc-compaction".equals(scope) || "recompaction".equals(scope)) { + return gcCompact(request); + } if ("compaction".equals(scope)) { return compact(request); } throw new IllegalArgumentException("'scope' is missing or must be 'unreferenced'"); } + public Task initialCompact(Request request) { + BlobCompactionAlgorithm algorithm = blobCompactionAlgorithm + .orElseThrow(() -> new IllegalStateException("Blob compaction is not configured on this server")); + return new InitialBlobCompactionTask(algorithm, buildCompactionRequest(request), clock); + } + + public Task gcCompact(Request request) { + BlobCompactionAlgorithm algorithm = blobCompactionAlgorithm + .orElseThrow(() -> new IllegalStateException("Blob compaction is not configured on this server")); + return new GCBlobCompactionTask(algorithm, buildCompactionRequest(request), clock); + } + public Task compact(Request request) { + BlobCompactionAlgorithm algorithm = blobCompactionAlgorithm + .orElseThrow(() -> new IllegalStateException("Blob compaction is not configured on this server")); + return new BlobCompactionTask(algorithm, buildCompactionRequest(request), clock); + } + + private CompactionRequest buildCompactionRequest(Request request) { String generationParam = request.queryParams("generation"); Preconditions.checkArgument(generationParam != null && !generationParam.isBlank(), "'generation' is compulsory"); @@ -141,16 +167,11 @@ public Task compact(Request request) { } }); - BlobCompactionAlgorithm algorithm = blobCompactionAlgorithm - .orElseThrow(() -> new IllegalStateException("Blob compaction is not configured on this server")); - - CompactionRequest compactionRequest = CompactionRequest.builder() + return CompactionRequest.builder() .bucketName(bucketName) .generation(generation) .family(family) .build(); - - return new BlobCompactionTask(algorithm, compactionRequest, clock); } public Task gcUnreferenced(Request request) { diff --git a/server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java b/server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java index 097fac5528e..c11caabcb52 100644 --- a/server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java +++ b/server/protocols/webadmin/webadmin-data/src/test/java/org/apache/james/webadmin/routes/BlobRoutesCompactionTest.java @@ -79,10 +79,15 @@ void setUp() { GenerationAwareBlobId.Factory generationAwareBlobIdFactory = new GenerationAwareBlobId.Factory(clock, BLOB_ID_FACTORY, GENERATION_AWARE_BLOB_ID_CONFIGURATION); BlobStoreDAO blobStoreDAO = new MemoryBlobStoreDAO(); JsonTransformer jsonTransformer = new JsonTransformer(); - TasksRoutes tasksRoutes = new TasksRoutes(taskManager, jsonTransformer, DTOConverter.of(BlobCompactionDTOModules.additionalInformationModule())); + TasksRoutes tasksRoutes = new TasksRoutes(taskManager, jsonTransformer, DTOConverter.of( + BlobCompactionDTOModules.additionalInformationModule(), + BlobCompactionDTOModules.initialAdditionalInformationModule(), + BlobCompactionDTOModules.gcAdditionalInformationModule())); compactionAlgorithm = mock(BlobCompactionAlgorithm.class); when(compactionAlgorithm.compact(any())).thenReturn(Mono.just(CompactionResult.NONE)); + when(compactionAlgorithm.initialCompact(any())).thenReturn(Mono.just(CompactionResult.NONE)); + when(compactionAlgorithm.gcCompact(any())).thenReturn(Mono.just(CompactionResult.NONE)); BlobRoutes blobRoutes = new BlobRoutes( taskManager, @@ -108,6 +113,30 @@ void tearDown() { taskManager.stop(); } + @Test + void deleteInitialCompactionShouldReturnTaskIdWhenValidParameters() { + given() + .queryParam("scope", "initial-compaction") + .queryParam("generation", "2") + .queryParam("family", "1") + .delete() + .then() + .statusCode(HttpStatus.CREATED_201) + .body("taskId", notNullValue()); + } + + @Test + void deleteGCCompactionShouldReturnTaskIdWhenValidParameters() { + given() + .queryParam("scope", "gc-compaction") + .queryParam("generation", "2") + .queryParam("family", "1") + .delete() + .then() + .statusCode(HttpStatus.CREATED_201) + .body("taskId", notNullValue()); + } + @Test void deleteCompactionShouldReturnTaskIdWhenValidParameters() { given() From 0235092f211962048519cd597f8b1b37fcd91b7f Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 14:10:21 +0530 Subject: [PATCH 17/24] JAMES-4231 Guice: Make compaction dependencies optional in non-Cassandra environments --- .../compaction/BlobCompactionDTOModules.java | 13 ++++++ .../compaction/BlobCompactionTaskDTO.java | 7 ++- .../compaction/GCBlobCompactionTaskDTO.java | 7 ++- .../InitialBlobCompactionTaskDTO.java | 7 ++- .../blobstore/BlobCompactionModule.java | 44 +++++++++++-------- .../BlobStoreModulesChooserTest.java | 23 ++++++++++ 6 files changed, 79 insertions(+), 22 deletions(-) diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java index fa1f3cfbba0..aae2ffa0f30 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionDTOModules.java @@ -20,6 +20,7 @@ package org.apache.james.blob.compaction; import java.time.Clock; +import java.util.function.Supplier; import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule; import org.apache.james.server.task.json.dto.TaskDTOModule; @@ -29,14 +30,26 @@ public static TaskDTOModule taskModul return BlobCompactionTaskDTO.module(algorithm, clock); } + public static TaskDTOModule taskModule(Supplier algorithmSupplier, Clock clock) { + return BlobCompactionTaskDTO.module(algorithmSupplier, clock); + } + public static TaskDTOModule initialCompactionTaskModule(BlobCompactionAlgorithm algorithm, Clock clock) { return InitialBlobCompactionTaskDTO.module(algorithm, clock); } + public static TaskDTOModule initialCompactionTaskModule(Supplier algorithmSupplier, Clock clock) { + return InitialBlobCompactionTaskDTO.module(algorithmSupplier, clock); + } + public static TaskDTOModule gcCompactionTaskModule(BlobCompactionAlgorithm algorithm, Clock clock) { return GCBlobCompactionTaskDTO.module(algorithm, clock); } + public static TaskDTOModule gcCompactionTaskModule(Supplier algorithmSupplier, Clock clock) { + return GCBlobCompactionTaskDTO.module(algorithmSupplier, clock); + } + public static AdditionalInformationDTOModule additionalInformationModule() { return BlobCompactionTaskAdditionalInformationDTO.SERIALIZATION_MODULE; } diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java index 73b66baf31b..ca5c73e2d1d 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/BlobCompactionTaskDTO.java @@ -21,6 +21,7 @@ import java.time.Clock; import java.util.Optional; +import java.util.function.Supplier; import org.apache.james.blob.api.BucketName; import org.apache.james.json.DTOModule; @@ -62,6 +63,10 @@ public BlobCompactionTaskDTO(@JsonProperty("type") String type, } public static TaskDTOModule module(BlobCompactionAlgorithm algorithm, Clock clock) { + return module(() -> algorithm, clock); + } + + public static TaskDTOModule module(Supplier algorithmSupplier, Clock clock) { return DTOModule.forDomainObject(BlobCompactionTask.class) .convertToDTO(BlobCompactionTaskDTO.class) .toDomainObjectConverter(dto -> { @@ -79,7 +84,7 @@ public static TaskDTOModule module(Bl dto.getFamily().ifPresent(requestBuilder::family); - return new BlobCompactionTask(algorithm, requestBuilder.build(), clock); + return new BlobCompactionTask(algorithmSupplier.get(), requestBuilder.build(), clock); }) .toDTOConverter((domain, type) -> { CompactionRequest req = domain.getRequest(); diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java index 816e228fafb..053eb3fe53d 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/GCBlobCompactionTaskDTO.java @@ -21,6 +21,7 @@ import java.time.Clock; import java.util.Optional; +import java.util.function.Supplier; import org.apache.james.blob.api.BucketName; import org.apache.james.json.DTOModule; @@ -59,6 +60,10 @@ public GCBlobCompactionTaskDTO(@JsonProperty("type") String type, } public static TaskDTOModule module(BlobCompactionAlgorithm algorithm, Clock clock) { + return module(() -> algorithm, clock); + } + + public static TaskDTOModule module(Supplier algorithmSupplier, Clock clock) { return DTOModule.forDomainObject(GCBlobCompactionTask.class) .convertToDTO(GCBlobCompactionTaskDTO.class) .toDomainObjectConverter(dto -> { @@ -75,7 +80,7 @@ public static TaskDTOModule modul dto.getFamily().ifPresent(requestBuilder::family); - return new GCBlobCompactionTask(algorithm, requestBuilder.build(), clock); + return new GCBlobCompactionTask(algorithmSupplier.get(), requestBuilder.build(), clock); }) .toDTOConverter((domain, type) -> { CompactionRequest req = domain.getRequest(); diff --git a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java index 78961b05c42..6e62df11e10 100644 --- a/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java +++ b/server/blob/blob-compaction/src/main/java/org/apache/james/blob/compaction/InitialBlobCompactionTaskDTO.java @@ -21,6 +21,7 @@ import java.time.Clock; import java.util.Optional; +import java.util.function.Supplier; import org.apache.james.blob.api.BucketName; import org.apache.james.json.DTOModule; @@ -53,6 +54,10 @@ public InitialBlobCompactionTaskDTO(@JsonProperty("type") String type, } public static TaskDTOModule module(BlobCompactionAlgorithm algorithm, Clock clock) { + return module(() -> algorithm, clock); + } + + public static TaskDTOModule module(Supplier algorithmSupplier, Clock clock) { return DTOModule.forDomainObject(InitialBlobCompactionTask.class) .convertToDTO(InitialBlobCompactionTaskDTO.class) .toDomainObjectConverter(dto -> { @@ -67,7 +72,7 @@ public static TaskDTOModule { CompactionRequest req = domain.getRequest(); diff --git a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java index 9e2950af9bc..3794cfd66cb 100644 --- a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java +++ b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobCompactionModule.java @@ -41,12 +41,15 @@ import com.google.inject.AbstractModule; import com.google.inject.Injector; import com.google.inject.Key; +import com.google.inject.Provider; import com.google.inject.Provides; import com.google.inject.Singleton; import com.google.inject.multibindings.ProvidesIntoSet; import com.google.inject.name.Named; +import com.google.inject.name.Names; public class BlobCompactionModule extends AbstractModule { + private static final Logger LOGGER = LoggerFactory.getLogger(BlobCompactionModule.class); @Provides @Singleton @@ -54,17 +57,6 @@ public CompactionConfiguration compactionConfiguration() { return CompactionConfiguration.DEFAULT; } - @Provides - @Singleton - public BlobCompactionAlgorithm blobCompactionAlgorithm(BlobStoreDAO blobStoreDAO, - @Named(BlobStoreModulesChooser.RAW) BlobStoreDAO rawStore, - BlobReferenceMappingSource mappingSource, - BlobIdUpdater blobIdUpdater) { - return new BlobCompactionAlgorithm(blobStoreDAO, rawStore, mappingSource, blobIdUpdater); - } - - private static final Logger LOGGER = LoggerFactory.getLogger(BlobCompactionModule.class); - @Provides @Singleton public Optional optionalBlobCompactionAlgorithm(Injector injector) { @@ -79,25 +71,39 @@ public Optional optionalBlobCompactionAlgorithm(Injecto return Optional.empty(); } } - if (injector.getExistingBinding(Key.get(BlobCompactionAlgorithm.class)) != null) { - return Optional.of(injector.getInstance(BlobCompactionAlgorithm.class)); + if (injector.getExistingBinding(Key.get(BlobReferenceMappingSource.class)) != null + && injector.getExistingBinding(Key.get(BlobIdUpdater.class)) != null + && injector.getExistingBinding(Key.get(BlobStoreDAO.class)) != null + && injector.getExistingBinding(Key.get(BlobStoreDAO.class, Names.named(BlobStoreModulesChooser.RAW))) != null) { + + BlobStoreDAO blobStoreDAO = injector.getInstance(BlobStoreDAO.class); + BlobStoreDAO rawStore = injector.getInstance(Key.get(BlobStoreDAO.class, Names.named(BlobStoreModulesChooser.RAW))); + BlobReferenceMappingSource mappingSource = injector.getInstance(BlobReferenceMappingSource.class); + BlobIdUpdater blobIdUpdater = injector.getInstance(BlobIdUpdater.class); + return Optional.of(new BlobCompactionAlgorithm(blobStoreDAO, rawStore, mappingSource, blobIdUpdater)); } return Optional.empty(); } + @Provides + @Singleton + public BlobCompactionAlgorithm blobCompactionAlgorithm(Optional algorithm) { + return algorithm.orElseThrow(() -> new IllegalStateException("BlobCompactionAlgorithm is not available in the current environment")); + } + @ProvidesIntoSet - public TaskDTOModule blobCompactionTask(BlobCompactionAlgorithm algorithm, Clock clock) { - return BlobCompactionDTOModules.taskModule(algorithm, clock); + public TaskDTOModule blobCompactionTask(Provider algorithmProvider, Clock clock) { + return BlobCompactionDTOModules.taskModule(algorithmProvider::get, clock); } @ProvidesIntoSet - public TaskDTOModule initialBlobCompactionTask(BlobCompactionAlgorithm algorithm, Clock clock) { - return BlobCompactionDTOModules.initialCompactionTaskModule(algorithm, clock); + public TaskDTOModule initialBlobCompactionTask(Provider algorithmProvider, Clock clock) { + return BlobCompactionDTOModules.initialCompactionTaskModule(algorithmProvider::get, clock); } @ProvidesIntoSet - public TaskDTOModule gcBlobCompactionTask(BlobCompactionAlgorithm algorithm, Clock clock) { - return BlobCompactionDTOModules.gcCompactionTaskModule(algorithm, clock); + public TaskDTOModule gcBlobCompactionTask(Provider algorithmProvider, Clock clock) { + return BlobCompactionDTOModules.gcCompactionTaskModule(algorithmProvider::get, clock); } @ProvidesIntoSet diff --git a/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java b/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java index 60072dea9d9..d607f9854a8 100644 --- a/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java +++ b/server/container/guice/distributed/src/test/java/org/apache/james/modules/blobstore/BlobStoreModulesChooserTest.java @@ -179,4 +179,27 @@ void optionalBlobCompactionAlgorithmShouldReturnEmptyWhenCompressionConfigured() Key.get(new TypeLiteral>() {})); assertThat(algorithm).isEmpty(); } + + @Test + void optionalBlobCompactionAlgorithmShouldReturnEmptyInNonCassandraEnvironmentWithoutMappingDependencies() { + BlobStoreConfiguration config = BlobStoreConfiguration.builder() + .postgres() + .disableCache() + .passthrough() + .noCryptoConfig(); + + Injector injector = Guice.createInjector( + binder -> { + binder.bind(BlobStoreConfiguration.class).toInstance(config); + binder.bind(Clock.class).toInstance(Clock.systemUTC()); + binder.bind(BlobStoreDAO.class).toProvider(() -> null); + binder.bind(BlobStoreDAO.class).annotatedWith(Names.named(BlobStoreModulesChooser.RAW)).toProvider(() -> null); + }, + new BlobCompactionModule() + ); + + Optional algorithm = injector.getInstance( + Key.get(new TypeLiteral>() {})); + assertThat(algorithm).isEmpty(); + } } \ No newline at end of file From 9272fd2d4a76620f8d3cfcc829a15b39075a925b Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 14:28:12 +0530 Subject: [PATCH 18/24] JAMES-4231 Cassandra: Fallback to messageDAOV3 when header blob not found in message metadata --- .../cassandra/mail/CassandraMessageIdMapper.java | 13 ++++++++++++- .../cassandra/mail/CassandraMessageMapper.java | 13 ++++++++++++- 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java index 9a37a8ccbc6..4d0d74bb46d 100644 --- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java @@ -38,6 +38,7 @@ import org.apache.james.backends.cassandra.init.configuration.JamesExecutionProfiles; import org.apache.james.blob.api.BlobId; import org.apache.james.blob.api.BlobStore; +import org.apache.james.blob.api.ObjectNotFoundException; import org.apache.james.mailbox.MessageManager; import org.apache.james.mailbox.ModSeq; import org.apache.james.mailbox.cassandra.ids.CassandraId; @@ -136,8 +137,18 @@ private Mono toMailboxMessage(CassandraMessageMetadata metadata, } if (fetchType == FetchType.HEADERS && metadata.isComplete()) { return Mono.from(blobStore.readBytes(blobStore.getDefaultBucketName(), metadata.getHeaderContent().get(), SIZE_BASED)) - .map(metadata::asMailboxMessage); + .map(metadata::asMailboxMessage) + .onErrorResume(ObjectNotFoundException.class, e -> { + LOGGER.info("Header blob {} not found for message {}, falling back to messageDAOV3", + metadata.getHeaderContent().get().asString(), + metadata.getComposedMessageId().getComposedMessageId().getMessageId().serialize()); + return retrieveFromDAOV3(metadata, fetchType); + }); } + return retrieveFromDAOV3(metadata, fetchType); + } + + private Mono retrieveFromDAOV3(CassandraMessageMetadata metadata, FetchType fetchType) { return messageDAOV3.retrieveMessage(metadata.getComposedMessageId(), fetchType) .map(messageRepresentation -> Pair.of(metadata.getComposedMessageId(), messageRepresentation)) .flatMap(messageRepresentation -> attachmentLoader.addAttachmentToMessage(messageRepresentation, metadata.getSaveDate(), fetchType)); diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java index c8fc6ee5efd..f2489c3ba1e 100644 --- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java @@ -43,6 +43,7 @@ import org.apache.james.backends.cassandra.init.configuration.JamesExecutionProfiles; import org.apache.james.blob.api.BlobId; import org.apache.james.blob.api.BlobStore; +import org.apache.james.blob.api.ObjectNotFoundException; import org.apache.james.mailbox.ApplicableFlagBuilder; import org.apache.james.mailbox.FlagsBuilder; import org.apache.james.mailbox.MessageManager.FlagsUpdateMode; @@ -268,8 +269,18 @@ private Mono toMailboxMessage(CassandraMessageMetadata metadata, } if (fetchType == FetchType.HEADERS && metadata.isComplete()) { return Mono.from(blobStore.readBytes(blobStore.getDefaultBucketName(), metadata.getHeaderContent().get(), SIZE_BASED)) - .map(metadata::asMailboxMessage); + .map(metadata::asMailboxMessage) + .onErrorResume(ObjectNotFoundException.class, e -> { + LOGGER.info("Header blob {} not found for message {}, falling back to messageDAOV3", + metadata.getHeaderContent().get().asString(), + metadata.getComposedMessageId().getComposedMessageId().getMessageId().serialize()); + return retrieveFromDAOV3(metadata, fetchType); + }); } + return retrieveFromDAOV3(metadata, fetchType); + } + + private Mono retrieveFromDAOV3(CassandraMessageMetadata metadata, FetchType fetchType) { return messageDAOV3.retrieveMessage(metadata.getComposedMessageId(), fetchType) .map(messageRepresentation -> Pair.of(metadata.getComposedMessageId(), messageRepresentation)) .flatMap(messageRepresentation -> attachmentLoader.addAttachmentToMessage(messageRepresentation, metadata.getSaveDate(), fetchType)); From 3e99c473ec6c885a1939f4d683860b7609514862 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 15:02:43 +0530 Subject: [PATCH 19/24] JAMES-4231 Compaction: Add store coherence tests under reference update failures --- .../BlobCompactionAlgorithmTest.java | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java index b66be345745..33258d8cd6a 100644 --- a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionAlgorithmTest.java @@ -452,6 +452,132 @@ void initialCompactShouldNotDeleteOriginalBlobWhenReferenceUpdateFails() { assertThat(remainingBlobs).doesNotContain(succeedingBlob); } + @Test + void initialCompactShouldMaintainStoreCoherenceWhenReferenceUpdateFailsForSomeBlobs() { + BlobId blob1 = new PlainBlobId("1_2_blob1"); + BlobId blob2Failing = new PlainBlobId("1_2_blob2_fail"); + BlobId blob3 = new PlainBlobId("1_2_blob3"); + + byte[] payload1 = "Content of email 1".getBytes(StandardCharsets.UTF_8); + byte[] payload2 = "Content of email 2".getBytes(StandardCharsets.UTF_8); + byte[] payload3 = "Content of email 3".getBytes(StandardCharsets.UTF_8); + + Mono.from(rawStore.save(TEST_BUCKET, blob1, BlobStoreDAO.BytesBlob.of(payload1))).block(); + Mono.from(rawStore.save(TEST_BUCKET, blob2Failing, BlobStoreDAO.BytesBlob.of(payload2))).block(); + Mono.from(rawStore.save(TEST_BUCKET, blob3, BlobStoreDAO.BytesBlob.of(payload3))).block(); + + mappingSource.add(blob1, "msg-1"); + mappingSource.add(blob2Failing, "msg-2"); + mappingSource.add(blob3, "msg-3"); + + Map updatedReferences = new ConcurrentHashMap<>(); + BlobIdUpdater partiallyFailingUpdater = (oldId, newId, messageIds) -> { + if (oldId.equals(blob2Failing)) { + return Mono.error(new RuntimeException("Simulated reference update failure")); + } + updatedReferences.put(oldId, newId); + return Mono.empty(); + }; + + BlobCompactionAlgorithm algorithmWithFailingUpdater = new BlobCompactionAlgorithm( + rawStore, rawStore, mappingSource, partiallyFailingUpdater); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + CompactionResult result = algorithmWithFailingUpdater.initialCompact(request).block(); + + // Two succeeded, one failed + assertThat(result.packedBlobs()).isEqualTo(2); + + // Coherence check 1: Failed candidate blob was NOT deleted from raw storage + byte[] readBlob2 = Mono.from(rawStore.readBytes(TEST_BUCKET, blob2Failing)).block().payload(); + assertThat(readBlob2).isEqualTo(payload2); + + // Coherence check 2: Successful candidate blobs had standalone blobs deleted + List remainingRawBlobs = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(remainingRawBlobs).contains(blob2Failing); + assertThat(remainingRawBlobs).doesNotContain(blob1, blob3); + + // Coherence check 3: Successful candidate blobs can be read via their new chunk slot IDs + BlobId slotRef1 = updatedReferences.get(blob1); + BlobId slotRef3 = updatedReferences.get(blob3); + assertThat(slotRef1).isNotNull(); + assertThat(slotRef3).isNotNull(); + + byte[] readSlot1 = readDecompressed(slotRef1); + byte[] readSlot3 = readDecompressed(slotRef3); + assertThat(readSlot1).isEqualTo(payload1); + assertThat(readSlot3).isEqualTo(payload3); + + // Coherence check 4: Re-compacting once the failure is resolved allows the remaining blob to be compacted + BlobId anotherBlob = new PlainBlobId("1_2_blob4"); + byte[] payload4 = "Content of email 4".getBytes(StandardCharsets.UTF_8); + Mono.from(rawStore.save(TEST_BUCKET, anotherBlob, BlobStoreDAO.BytesBlob.of(payload4))).block(); + mappingSource.add(anotherBlob, "msg-4"); + + BlobIdUpdater recoveredUpdater = (oldId, newId, messageIds) -> { + updatedReferences.put(oldId, newId); + return Mono.empty(); + }; + BlobCompactionAlgorithm recoveredAlgorithm = new BlobCompactionAlgorithm( + rawStore, rawStore, mappingSource, recoveredUpdater); + + CompactionResult secondResult = recoveredAlgorithm.initialCompact(request).block(); + assertThat(secondResult.packedBlobs()).isEqualTo(2); + + BlobId slotRef2 = updatedReferences.get(blob2Failing); + assertThat(slotRef2).isNotNull(); + byte[] readSlot2 = readDecompressed(slotRef2); + assertThat(readSlot2).isEqualTo(payload2); + } + + @Test + void gcCompactShouldMaintainStoreCoherenceWhenSlotReferenceUpdateFails() throws Exception { + byte[] payload1 = "Active live slot payload".getBytes(StandardCharsets.UTF_8); + byte[] payload2 = "Dead obsolete slot payload".getBytes(StandardCharsets.UTF_8); + ChunkWriteResult writeResult = ChunkFormat.writeChunk(List.of( + BlobSlotContent.of(payload1), + BlobSlotContent.of(payload2))); + ChunkId chunkId = ChunkId.ofChunk(FAMILY, TARGET_GENERATION); + Mono.from(rawStore.save(TEST_BUCKET, chunkId.chunkBlobId(), BlobStoreDAO.BytesBlob.of(writeResult.chunkBytes()))).block(); + + SlotRange range1 = writeResult.slotRanges().get(0); + ChunkId slot1 = ChunkId.slotRef(chunkId, range1.offset(), range1.limit()); + // Only slot 1 is live in mappingSource; slot 2 is dead + mappingSource.add(slot1, "msg-live"); + + BlobIdUpdater failingUpdater = (oldId, newId, messageIds) -> + Mono.error(new RuntimeException("Simulated reference update failure during GC")); + + BlobCompactionAlgorithm algorithm = new BlobCompactionAlgorithm( + rawStore, rawStore, mappingSource, failingUpdater); + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .purgeDeadRatio(0.4) + .build()) + .build(); + + // GC compaction fails during slot reference update + org.assertj.core.api.Assertions.assertThatThrownBy(() -> algorithm.gcCompact(request).block()) + .isInstanceOf(RuntimeException.class); + + // Coherence check 1: Original chunk was NOT deleted from raw storage + List remainingBlobs = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(remainingBlobs).contains(chunkId.chunkBlobId()); + + // Coherence check 2: Original live slot is still readable through ChunkedBlobStoreDAO and returns exact payload + byte[] readLiveSlot = readDecompressed(slot1); + assertThat(readLiveSlot).isEqualTo(payload1); + } + static class RangedReadTrackingMemoryBlobStoreDAO extends MemoryBlobStoreDAO { private final AtomicLong maxRangedReadBytes = new AtomicLong(0); private final List wholeChunkReads = new CopyOnWriteArrayList<>(); From ade7c333ced9ea5c77e1a1397a3d687c33fa4be1 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 15:05:41 +0530 Subject: [PATCH 20/24] JAMES-4231 Compaction: Add layout matrix integration tests on MemoryBlobStoreDAO --- .../BlobCompactionLayoutIntegrationTest.java | 365 ++++++++++++++++++ 1 file changed, 365 insertions(+) create mode 100644 server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionLayoutIntegrationTest.java diff --git a/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionLayoutIntegrationTest.java b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionLayoutIntegrationTest.java new file mode 100644 index 00000000000..c94846e245e --- /dev/null +++ b/server/blob/blob-compaction/src/test/java/org/apache/james/blob/compaction/BlobCompactionLayoutIntegrationTest.java @@ -0,0 +1,365 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.blob.compaction; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.BlobStoreDAO; +import org.apache.james.blob.api.BucketName; +import org.apache.james.blob.api.ObjectNotFoundException; +import org.apache.james.blob.api.PlainBlobId; +import org.apache.james.blob.memory.MemoryBlobStoreDAO; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import com.github.luben.zstd.Zstd; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +class BlobCompactionLayoutIntegrationTest { + private static final BucketName TEST_BUCKET = BucketName.of("compaction-layout-test-bucket"); + private static final long TARGET_GENERATION = 2L; + private static final int FAMILY = 1; + private static final int POPULATION_SIZE = 200; + + private MemoryBlobStoreDAO rawStore; + private ChunkedBlobStoreDAO chunkedStore; + private BlobCompactionAlgorithmTest.TestMappingSource mappingSource; + private Map updatedReferences; + private BlobCompactionAlgorithm algorithm; + + @BeforeEach + void setUp() { + rawStore = new MemoryBlobStoreDAO(); + chunkedStore = new ChunkedBlobStoreDAO(rawStore, rawStore); + mappingSource = new BlobCompactionAlgorithmTest.TestMappingSource(); + updatedReferences = new ConcurrentHashMap<>(); + + BlobIdUpdater updater = (oldId, newId, messageIds) -> { + updatedReferences.put(oldId, newId); + return Mono.empty(); + }; + algorithm = new BlobCompactionAlgorithm(rawStore, rawStore, mappingSource, updater); + } + + private byte[] readDecompressed(BlobId blobId) { + BlobStoreDAO.BytesBlob blob = Mono.from(chunkedStore.readBytes(TEST_BUCKET, blobId)).block(); + if (blob.metadata().contentEncoding().filter(BlobStoreDAO.ContentEncoding.ZSTD::equals).isPresent()) { + long origSize = blob.metadata().get(BlobSlot.CONTENT_ORIGINAL_SIZE) + .map(v -> Long.parseLong(v.value())) + .orElse((long) blob.payload().length); + return Zstd.decompress(blob.payload(), (int) origSize); + } + return blob.payload(); + } + + @Test + void layout1_uniformSmallPopulationAllActive_shouldYieldSingleChunkAndMaintainIntegrity() { + Map payloads = new HashMap<>(); + + for (int i = 0; i < POPULATION_SIZE; i++) { + BlobId blobId = new PlainBlobId("1_2_email_" + i); + byte[] payload = ("Subject: Email " + i + "\r\n\r\nBody of message number " + i).getBytes(StandardCharsets.UTF_8); + payloads.put(blobId, payload); + Mono.from(rawStore.save(TEST_BUCKET, blobId, BlobStoreDAO.BytesBlob.of(payload))).block(); + mappingSource.add(blobId, "msg-" + i); + } + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + CompactionResult result = algorithm.initialCompact(request).block(); + + // 1. All 200 packed into chunks + assertThat(result.packedBlobs()).isEqualTo(POPULATION_SIZE); + + // 2. Object layout verification: 0 standalone blobs remain, 1 chunk object created + List storedObjects = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(storedObjects).hasSize(1); + assertThat(storedObjects.get(0).asString()).contains("_chunk"); + + // 3. Integrity verification: all 200 emails readable via new chunk slot references + for (Map.Entry entry : payloads.entrySet()) { + BlobId newSlotRef = updatedReferences.get(entry.getKey()); + assertThat(newSlotRef).isNotNull(); + byte[] retrieved = readDecompressed(newSlotRef); + assertThat(retrieved).isEqualTo(entry.getValue()); + + // Standalone original blob no longer accessible -> 404 + assertThatThrownBy(() -> Mono.from(rawStore.readBytes(TEST_BUCKET, entry.getKey())).block()) + .isInstanceOf(ObjectNotFoundException.class); + } + + // 4. Non-existent blob yields 404 + assertThatThrownBy(() -> Mono.from(chunkedStore.readBytes(TEST_BUCKET, new PlainBlobId("1_2_nonexistent"))).block()) + .isInstanceOf(ObjectNotFoundException.class); + } + + @Test + void layout2_partiallyUnreferencedPopulation_shouldCompactOnlyActiveAndPreserveUncompacted() { + Map activePayloads = new HashMap<>(); + List unreferencedBlobs = new ArrayList<>(); + + for (int i = 0; i < POPULATION_SIZE; i++) { + BlobId blobId = new PlainBlobId("1_2_partial_" + i); + byte[] payload = ("Email payload data " + i).getBytes(StandardCharsets.UTF_8); + Mono.from(rawStore.save(TEST_BUCKET, blobId, BlobStoreDAO.BytesBlob.of(payload))).block(); + + if (i < 100) { + // First 100 have active metadata references + activePayloads.put(blobId, payload); + mappingSource.add(blobId, "msg-active-" + i); + } else { + // Next 100 are orphaned/unreferenced (no metadata) + unreferencedBlobs.add(blobId); + } + } + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + CompactionResult result = algorithm.initialCompact(request).block(); + + // 1. Only 100 active blobs compacted + assertThat(result.packedBlobs()).isEqualTo(100); + + // 2. Object layout verification: 1 chunk object + 100 uncompacted standalone blobs = 101 objects + List storedObjects = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(storedObjects).hasSize(101); + + // 3. Active emails readable via chunk slots + for (Map.Entry entry : activePayloads.entrySet()) { + BlobId newSlotRef = updatedReferences.get(entry.getKey()); + assertThat(newSlotRef).isNotNull(); + assertThat(readDecompressed(newSlotRef)).isEqualTo(entry.getValue()); + } + + // 4. Unreferenced blobs still remain intact as standalone blobs + for (BlobId unrefId : unreferencedBlobs) { + byte[] unrefData = Mono.from(rawStore.readBytes(TEST_BUCKET, unrefId)).block().payload(); + assertThat(unrefData).isNotEmpty(); + } + + // 5. Querying non-existent / deleted metadata -> 404 + assertThatThrownBy(() -> Mono.from(chunkedStore.readBytes(TEST_BUCKET, new PlainBlobId("1_2_ghost"))).block()) + .isInstanceOf(ObjectNotFoundException.class); + } + + @Test + void layout3_mixedSizesWithOversizedBlobs_shouldBypassOversizedAndCompactRemaining() { + Map smallPayloads = new HashMap<>(); + Map oversizedPayloads = new HashMap<>(); + + for (int i = 0; i < POPULATION_SIZE; i++) { + BlobId blobId = new PlainBlobId("1_2_mixed_" + i); + byte[] payload; + if (i % 40 == 0) { + // 5 oversized blobs (exceeding maxBlobSizeInChunk = 64KB) + payload = new byte[70 * 1024]; + payload[0] = (byte) i; + oversizedPayloads.put(blobId, payload); + } else { + // 195 small blobs + payload = ("Small email payload " + i).getBytes(StandardCharsets.UTF_8); + smallPayloads.put(blobId, payload); + } + Mono.from(rawStore.save(TEST_BUCKET, blobId, BlobStoreDAO.BytesBlob.of(payload))).block(); + mappingSource.add(blobId, "msg-mixed-" + i); + } + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .maxPackableSize(64 * 1024L) + .build()) + .build(); + + CompactionResult result = algorithm.initialCompact(request).block(); + + // 1. Exactly 195 small blobs packed, 5 oversized preserved + assertThat(result.packedBlobs()).isEqualTo(195); + + // 2. Object layout verification: 1 chunk + 5 oversized standalone blobs = 6 objects + List storedObjects = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(storedObjects).hasSize(6); + + // 3. Small blobs readable through chunk slots + for (Map.Entry entry : smallPayloads.entrySet()) { + BlobId newSlotRef = updatedReferences.get(entry.getKey()); + assertThat(newSlotRef).isNotNull(); + assertThat(readDecompressed(newSlotRef)).isEqualTo(entry.getValue()); + } + + // 4. Oversized blobs readable through their original standalone IDs + for (Map.Entry entry : oversizedPayloads.entrySet()) { + assertThat(updatedReferences.containsKey(entry.getKey())).isFalse(); + byte[] content = Mono.from(chunkedStore.readBytes(TEST_BUCKET, entry.getKey())).block().payload(); + assertThat(content).isEqualTo(entry.getValue()); + } + + // 5. 404 on unreferenced/unknown blob + assertThatThrownBy(() -> Mono.from(chunkedStore.readBytes(TEST_BUCKET, new PlainBlobId("1_2_missing"))).block()) + .isInstanceOf(ObjectNotFoundException.class); + } + + @Test + void layout4_multiGenerationInterleavedPopulation_shouldOnlyCompactTargetGeneration() { + Map gen2Payloads = new HashMap<>(); + Map gen3Payloads = new HashMap<>(); + + for (int i = 0; i < POPULATION_SIZE; i++) { + if (i % 2 == 0) { + BlobId blobGen2 = new PlainBlobId("1_2_email_" + i); + byte[] payload = ("Gen2 content " + i).getBytes(StandardCharsets.UTF_8); + gen2Payloads.put(blobGen2, payload); + Mono.from(rawStore.save(TEST_BUCKET, blobGen2, BlobStoreDAO.BytesBlob.of(payload))).block(); + mappingSource.add(blobGen2, "msg-gen2-" + i); + } else { + BlobId blobGen3 = new PlainBlobId("1_3_email_" + i); + byte[] payload = ("Gen3 content " + i).getBytes(StandardCharsets.UTF_8); + gen3Payloads.put(blobGen3, payload); + Mono.from(rawStore.save(TEST_BUCKET, blobGen3, BlobStoreDAO.BytesBlob.of(payload))).block(); + mappingSource.add(blobGen3, "msg-gen3-" + i); + } + } + + CompactionRequest request = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) // Only generation 2 + .family(FAMILY) + .build(); + + CompactionResult result = algorithm.initialCompact(request).block(); + + // 1. Exactly 100 gen2 blobs packed; gen3 untouched + assertThat(result.packedBlobs()).isEqualTo(100); + + // 2. Object layout verification: 1 chunk for gen2 + 100 gen3 standalone blobs = 101 objects + List storedObjects = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(storedObjects).hasSize(101); + + // 3. Gen2 emails readable via chunk slots + for (Map.Entry entry : gen2Payloads.entrySet()) { + BlobId newSlotRef = updatedReferences.get(entry.getKey()); + assertThat(newSlotRef).isNotNull(); + assertThat(readDecompressed(newSlotRef)).isEqualTo(entry.getValue()); + } + + // 4. Gen3 emails readable via their untouched standalone IDs + for (Map.Entry entry : gen3Payloads.entrySet()) { + assertThat(updatedReferences.containsKey(entry.getKey())).isFalse(); + byte[] content = Mono.from(chunkedStore.readBytes(TEST_BUCKET, entry.getKey())).block().payload(); + assertThat(content).isEqualTo(entry.getValue()); + } + + // 5. 404 integrity check + assertThatThrownBy(() -> Mono.from(chunkedStore.readBytes(TEST_BUCKET, new PlainBlobId("1_2_missing"))).block()) + .isInstanceOf(ObjectNotFoundException.class); + } + + @Test + void layout5_twoStageCompactionWithSubsequentDeletionAndGC_shouldPurgeDeadSlotsAndPreserveLive() { + Map allPayloads = new HashMap<>(); + + for (int i = 0; i < POPULATION_SIZE; i++) { + BlobId blobId = new PlainBlobId("1_2_two_stage_" + i); + byte[] payload = ("Email payload number " + i).getBytes(StandardCharsets.UTF_8); + allPayloads.put(blobId, payload); + Mono.from(rawStore.save(TEST_BUCKET, blobId, BlobStoreDAO.BytesBlob.of(payload))).block(); + mappingSource.add(blobId, "msg-ts-" + i); + } + + CompactionRequest initialRequest = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .build(); + + // Stage 1: Initial compaction packs all 200 into 1 chunk + CompactionResult initialResult = algorithm.initialCompact(initialRequest).block(); + assertThat(initialResult.packedBlobs()).isEqualTo(POPULATION_SIZE); + + List stage1Objects = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(stage1Objects).hasSize(1); + BlobId originalChunkBlobId = stage1Objects.get(0); + + // Stage 2: Simulate deleting 140 messages (leaving 60 live messages) + // 140 / 200 = 70% dead slots > 40% purge threshold + Map livePayloads = new HashMap<>(); + for (int i = 0; i < POPULATION_SIZE; i++) { + BlobId oldBlobId = new PlainBlobId("1_2_two_stage_" + i); + BlobId slotRef = updatedReferences.get(oldBlobId); + mappingSource.remove(oldBlobId); + if (i < 60) { + livePayloads.put(slotRef, allPayloads.get(oldBlobId)); + mappingSource.add(slotRef, "msg-ts-" + i); + } + } + + // Stage 3: Run GC compaction + CompactionRequest gcRequest = CompactionRequest.builder() + .bucketName(TEST_BUCKET) + .generation(TARGET_GENERATION) + .family(FAMILY) + .configuration(CompactionConfiguration.builder() + .purgeDeadRatio(0.4) + .build()) + .build(); + + CompactionResult gcResult = algorithm.gcCompact(gcRequest).block(); + + // 140 dead slots purged + assertThat(gcResult.deadPurged()).isEqualTo(140); + + // Object layout verification: original chunk deleted, 1 new purged chunk created + List stage2Objects = Flux.from(rawStore.listBlobs(TEST_BUCKET)).collectList().block(); + assertThat(stage2Objects).hasSize(1); + assertThat(stage2Objects.get(0)).isNotEqualTo(originalChunkBlobId); + + // Live messages remain readable through their re-compacted slot IDs + for (Map.Entry entry : livePayloads.entrySet()) { + BlobId recompactedSlotRef = updatedReferences.get(entry.getKey()); + assertThat(recompactedSlotRef).isNotNull(); + assertThat(readDecompressed(recompactedSlotRef)).isEqualTo(entry.getValue()); + } + + // Deleted messages no longer referenced -> accessing invalid slot yields 404 + assertThatThrownBy(() -> Mono.from(chunkedStore.readBytes(TEST_BUCKET, new PlainBlobId("1_2_nonexistent"))).block()) + .isInstanceOf(ObjectNotFoundException.class); + } +} From 7ceb33f764389b9329fd71ecae60d5391a662691 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 15:08:34 +0530 Subject: [PATCH 21/24] JAMES-4231 Docs: Document S3 object compaction architecture and WebAdmin endpoints --- .../partials/architecture/blobstore.adoc | 31 +++++++++ .../servers/partials/operate/webadmin.adoc | 66 +++++++++++++++++++ 2 files changed, 97 insertions(+) diff --git a/docs/modules/servers/partials/architecture/blobstore.adoc b/docs/modules/servers/partials/architecture/blobstore.adoc index fb4ef523d1e..f5f2ac53cec 100644 --- a/docs/modules/servers/partials/architecture/blobstore.adoc +++ b/docs/modules/servers/partials/architecture/blobstore.adoc @@ -59,6 +59,13 @@ connectors: `content-encoding` and the original size. On reads, it uses this metadata to transparently decompress the payload. This reduces storage usage and network transfer for compressible blob content. +* `ChunkedBlobStoreDAO` wraps the storage DAO to provide transparent, low-latency + access to blobs compacted into multi-slot chunk objects. When a blob identifier + represents a virtual slot within a compacted chunk + (`__chunk~~`), `ChunkedBlobStoreDAO` + translates read requests into precise HTTP byte-range queries + (`Range: bytes=offset-limit`) against the underlying object store. Reads of + uncompacted standalone blobs pass through directly without overhead. AES and Zstd can be enabled together. In the Guice binding chain, compression wraps encryption: `ZstdBlobStoreDAO` delegates to `AESBlobStoreDAO`, which then @@ -67,6 +74,30 @@ encrypt afterwards; reads decrypt first and decompress afterwards. This ordering preserves the benefit of compression, as encrypted payloads are generally not compressible. +== S3 Object Compaction + +Storing millions of small objects (such as email headers and small bodies) in S3 +leads to increased storage costs, high request fees (PUT/GET), and slow metadata +listing operations. + +James provides an S3 Object Compaction mechanism to mitigate this: + +* *Generation-aware packing*: Historical generation blobs that have become immutable + are gathered by `BlobCompactionAlgorithm` and packed into large (~100MB) chunk + objects in S3. +* *Binary Chunk Format*: Chunks store slots containing individual blob payloads + compressed with Zstandard (Zstd), slot CRC32 checksums, and a trailing footer + with slot offset tables. +* *Atomic reference updates*: Cassandra message metadata tables (`messageV3`, + `imapUidTable`, `messageIdTable`) are updated to point to the new virtual chunk + slot IDs before the original standalone blobs are deleted from S3. +* *Two-stage lifecycle*: + ** *Initial Compaction*: Packs standalone candidate blobs of a target generation + into chunks and updates references. + ** *GC Recompaction*: Rewrites sparse chunks (where slots were marked dead due to + message expunges) into new dense chunks to reclaim storage space, and merges + undersized chunks. + == Logical buckets `BucketName` is a James logical namespace in the `BlobStoreDAO` contract. It is diff --git a/docs/modules/servers/partials/operate/webadmin.adoc b/docs/modules/servers/partials/operate/webadmin.adoc index a5f11aa3939..4f1aa3db245 100644 --- a/docs/modules/servers/partials/operate/webadmin.adoc +++ b/docs/modules/servers/partials/operate/webadmin.adoc @@ -3176,6 +3176,72 @@ Where: filter in later runs. - *gcedBlobCount* is the count of blobs that were garbage collected. +== Running blob object compaction + +In large installations backed by S3 object storage, storing large numbers of small +blobs can lead to high S3 request billing and slow bucket listings. Object compaction +allows administrators to pack standalone blobs from historical generations into large +chunk objects, serving individual blobs via HTTP ranged reads. + +=== Initial Compaction + +To compact standalone blobs of a completed generation into chunk objects: + +.... +curl -XDELETE "http://ip:port/blobs?scope=initial-compaction&generation=1" +.... + +link:#_endpoints_returning_a_task[More details about endpoints returning a task]. + +Query parameters: + +- *generation*: (Compulsory) The integer epoch generation to compact. +- *family*: (Optional) Storage policy family identifier. + +The created task has the following additional information: + +.... +{ + "type": "InitialBlobCompactionTask", + "generation": 1, + "family": 1, + "packedBlobs": 12500, + "createdChunks": 5, + "freedBytes": 524288000 +} +.... + +=== GC Recompaction (Purge & Merge) + +When messages are expunged over time, slots within compacted chunks become dead. +GC recompaction scans existing chunks, purges dead slots by rewriting live slots +into new compact chunks, and merges undersized chunks: + +.... +curl -XDELETE "http://ip:port/blobs?scope=gc-compaction&generation=1" +.... + +The created task has the following additional information: + +.... +{ + "type": "GCBlobCompactionTask", + "generation": 1, + "family": 1, + "deadPurged": 450, + "chunksMerged": 2, + "freedBytes": 20971520 +} +.... + +=== Full Compaction + +To execute both initial compaction and GC recompaction in a single operation: + +.... +curl -XDELETE "http://ip:port/blobs?scope=compaction&generation=1" +.... + endif::[] == Administrating Recipient rewriting From 2f672d5897f98228048683eadb53e72f91a33423 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:50:40 +0530 Subject: [PATCH 22/24] JAMES-4231 Tests: Decompress slot payload in CassandraBlobIdRepairerIntegrationTest --- .../mail/CassandraBlobIdRepairerIntegrationTest.java | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java index fa397640e23..29424168142 100644 --- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java +++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraBlobIdRepairerIntegrationTest.java @@ -57,6 +57,8 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.RegisterExtension; +import com.github.luben.zstd.Zstd; + import reactor.core.publisher.Mono; class CassandraBlobIdRepairerIntegrationTest { @@ -138,7 +140,8 @@ void repairShouldFixCorruptedMessageIdTableFromImapUidTable() throws Exception { .build()).block(); // Read using deadSlotRef: triggers repair -> fetches canonical from imapUidTable -> fixes messageIdTable - byte[] readBack = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, deadSlotRef)).block().payload(); + BlobStoreDAO.BytesBlob readBlob = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, deadSlotRef)).block(); + byte[] readBack = Zstd.decompress(readBlob.payload(), content.length); assertThat(readBack).isEqualTo(content); // Verify messageIdTable is now repaired with the canonical validSlotRef @@ -193,7 +196,8 @@ void repairShouldFixCorruptedImapUidTableFromMessageIdTable() throws Exception { .build()).block(); // Read using deadSlotRef: triggers repair -> fetches canonical from messageIdTable -> fixes imapUidTable - byte[] readBack = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, deadSlotRef)).block().payload(); + BlobStoreDAO.BytesBlob readBlob = Mono.from(chunkedBlobStoreDAO.readBytes(TEST_BUCKET, deadSlotRef)).block(); + byte[] readBack = Zstd.decompress(readBlob.payload(), content.length); assertThat(readBack).isEqualTo(content); // Verify imapUidTable is now repaired with the canonical validSlotRef From d84f00dc3d8a03aac01b24a1b9430198bdb14b66 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Wed, 23 Sep 2026 11:53:36 +0530 Subject: [PATCH 23/24] JAMES-4231 Tests: Decompress slot payload in S3MinioBlobStoreCompactionTest --- .../objectstorage/aws/S3MinioBlobStoreCompactionTest.java | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java index d078eb79afe..8ea445d2bc0 100644 --- a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java +++ b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3MinioBlobStoreCompactionTest.java @@ -36,6 +36,7 @@ import org.apache.james.blob.api.BlobId; import org.apache.james.blob.api.BlobReferenceSource; import org.apache.james.blob.api.BlobStore; +import org.apache.james.blob.api.BlobStoreDAO; import org.apache.james.blob.api.BucketName; import org.apache.james.blob.api.ObjectNotFoundException; import org.apache.james.blob.api.PlainBlobId; @@ -58,6 +59,8 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.RegisterExtension; +import com.github.luben.zstd.Zstd; + import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; import reactor.util.retry.Retry; @@ -165,8 +168,9 @@ void compactionShouldPackBlobsAndAllowReadsWhileGcPreservesChunks() { for (Map.Entry entry : savedBlobs.entrySet()) { BlobId newSlotId = updatedIds.get(entry.getKey()); assertThat(newSlotId).isNotNull(); - byte[] readBytes = Mono.from(chunkedBlobStoreDAO.readBytes(BUCKET, newSlotId)).block().payload(); - assertThat(readBytes).isEqualTo(entry.getValue()); + BlobStoreDAO.BytesBlob readBlob = Mono.from(chunkedBlobStoreDAO.readBytes(BUCKET, newSlotId)).block(); + byte[] decompressed = Zstd.decompress(readBlob.payload(), entry.getValue().length); + assertThat(decompressed).isEqualTo(entry.getValue()); } // 4. Assert: BloomFilterGCAlgorithm after compaction does NOT delete the chunk object From 526160218466203b2335422ca0a6ae70eec8f737 Mon Sep 17 00:00:00 2001 From: Hesanda Liyanage <130324291+HesandaLiyanage@users.noreply.github.com> Date: Wed, 23 Sep 2026 15:33:17 +0530 Subject: [PATCH 24/24] JAMES-4231 Guice: Fallback to ENCRYPTION when CHUNKED blobstore is not bound --- .../apache/james/JpaToPgCoreDataMigration.java | 2 ++ .../blobstore/BlobStoreModulesChooser.java | 18 ++++++++++++++---- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/server/apps/migration/core-data-jpa-to-pg/src/main/java/org/apache/james/JpaToPgCoreDataMigration.java b/server/apps/migration/core-data-jpa-to-pg/src/main/java/org/apache/james/JpaToPgCoreDataMigration.java index 84c9008dc89..b9dde1e6395 100644 --- a/server/apps/migration/core-data-jpa-to-pg/src/main/java/org/apache/james/JpaToPgCoreDataMigration.java +++ b/server/apps/migration/core-data-jpa-to-pg/src/main/java/org/apache/james/JpaToPgCoreDataMigration.java @@ -20,6 +20,7 @@ package org.apache.james; import static org.apache.james.modules.blobstore.BlobStoreModulesChooser.chooseBlobStoreDAOModule; +import static org.apache.james.modules.blobstore.BlobStoreModulesChooser.chooseChunkedBlobStoreDAOModule; import static org.apache.james.modules.blobstore.BlobStoreModulesChooser.chooseCompressionModule; import static org.apache.james.modules.blobstore.BlobStoreModulesChooser.chooseEncryptionModule; import static org.apache.james.modules.blobstore.BlobStoreModulesChooser.chooseStoragePolicyModule; @@ -163,6 +164,7 @@ public static List chooseModules(BlobStoreConfiguration choosingConfigur return ImmutableList.builder() .add(chooseBlobStoreDAOModule(choosingConfiguration.getImplementation())) .add(chooseEncryptionModule(choosingConfiguration.getCryptoConfig())) + .add(chooseChunkedBlobStoreDAOModule()) .add(chooseCompressionModule(choosingConfiguration.getCompressionConfiguration())) .addAll(chooseStoragePolicyModule(choosingConfiguration.storageStrategy())) .add(binder -> binder.bind(BlobStoreConfiguration.class).toInstance(choosingConfiguration)) diff --git a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java index dabdecc14ef..5e2530e321d 100644 --- a/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java +++ b/server/container/guice/distributed/src/main/java/org/apache/james/modules/blobstore/BlobStoreModulesChooser.java @@ -135,11 +135,17 @@ protected void configure() { } } + static BlobStoreDAO resolveChunkedOrEncryption(Injector injector) { + return Optional.ofNullable(injector.getExistingBinding(Key.get(BlobStoreDAO.class, Names.named(CHUNKED)))) + .map(binding -> (BlobStoreDAO) binding.getProvider().get()) + .orElseGet(() -> injector.getInstance(Key.get(BlobStoreDAO.class, Names.named(ENCRYPTION)))); + } + static class NoCompressionModule extends AbstractModule { @Provides @Singleton - BlobStoreDAO blobStoreDAO(@Named(CHUNKED) BlobStoreDAO chunked, MetricFactory metricFactory) { - return new ZstdBlobStoreDAO(chunked, CompressionConfiguration.builder().enabled(false).minRatio(0).build(), metricFactory); + BlobStoreDAO blobStoreDAO(Injector injector, MetricFactory metricFactory) { + return new ZstdBlobStoreDAO(resolveChunkedOrEncryption(injector), CompressionConfiguration.builder().enabled(false).minRatio(0).build(), metricFactory); } } @@ -152,8 +158,8 @@ static class CompressionModule extends AbstractModule { @Provides @Singleton - BlobStoreDAO blobStoreDAO(@Named(CHUNKED) BlobStoreDAO chunked, MetricFactory metricFactory) { - return new ZstdBlobStoreDAO(chunked, compressionConfiguration, metricFactory); + BlobStoreDAO blobStoreDAO(Injector injector, MetricFactory metricFactory) { + return new ZstdBlobStoreDAO(resolveChunkedOrEncryption(injector), compressionConfiguration, metricFactory); } @Provides @@ -239,6 +245,10 @@ public static Module chooseEncryptionModule(Optional cryptoConfig) return encryptionModule.orElse(new NoEncryptionModule()); } + public static Module chooseChunkedBlobStoreDAOModule() { + return new ChunkedBlobStoreModule(); + } + public static Module chooseCompressionModule(CompressionConfiguration compressionConfiguration) { if (compressionConfiguration.enabled()) { return new CompressionModule(compressionConfiguration);