Repository navigation
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #866 +/- ##
===========================================
+ Coverage 64.53% 64.74% +0.21%
===========================================
Files 760 765 +5
Lines 53314 53936 +622
Branches 8453 8538 +85
===========================================
+ Hits 34404 34922 +518
- Misses 17258 17362 +104
Partials 1652 1652 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…x/tsfile-tde-page-aead # Conflicts: # java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java # java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java
HTHou
marked this pull request as ready for review
October 9, 2026 09:35
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open findings
What changed in this PR
This PR introduces pluggable page-level AEAD encryption for TsFile by adding a self-describing file encryption header, a provider-neutral encryption SPI/registry, and integrating page body encrypt/decrypt across write and read paths while preserving unencrypted TsFile compatibility.
Changes:
- Added
FileEncryptionHeader(written after the TsFile version byte) and propagated encryption context through writers/readers, append, and recovery flows. - Introduced provider-neutral AEAD SPI (
IEncryptProvider,EncryptionProviderRegistry) and page-associated-data binding (PageCryptoContext) with per-chunk ordinals and per-page indices. - Added/updated tests to validate round-trips, tamper detection, ordinal continuity across append/recovery, and reader behavior.
| File | Description |
|---|---|
| java/tsfile/src/test/java/org/apache/tsfile/write/writer/TimePageWriterTest.java | Adds an AEAD encrypt/decrypt page round-trip test using the test provider. |
| java/tsfile/src/test/java/org/apache/tsfile/write/writer/RestorableTsFileIOWriterTest.java | Adds recovery/ownership tests for encrypted headers and chunk ordinal continuation. |
| java/tsfile/src/test/java/org/apache/tsfile/write/writer/ForceAppendTsFileWriterTest.java | Adds force-append coverage for encrypted files and chunk ordinal uniqueness. |
| java/tsfile/src/test/java/org/apache/tsfile/write/TDEPageAeadTsFileTest.java | New integration tests for encrypted read/write paths, tamper detection, and chunk reuse rejection. |
| java/tsfile/src/test/java/org/apache/tsfile/read/UnClosedTsFileReaderTest.java | Adds reading encrypted pages from file header in unclosed reader scenario. |
| java/tsfile/src/test/java/org/apache/tsfile/file/header/FileEncryptionHeaderTest.java | New round-trip and validation tests for the file encryption header format. |
| java/tsfile/src/test/java/org/apache/tsfile/encrypt/TestAeadEncryptionProvider.java | Adds a JDK-only AES-GCM test provider implementing the new SPI. |
| java/tsfile/src/test/java/org/apache/tsfile/encrypt/PageCryptoContextTest.java | Adds tests for AAD binding/tamper detection and ordinal allocation behavior. |
| java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java | Writes encryption header before data, threads encryption context through chunk headers, and blocks encrypted chunk copy. |
| java/tsfile/src/main/java/org/apache/tsfile/write/writer/RestorableTsFileIOWriter.java | Loads encryption header on recovery, resumes ordinal allocation, and closes owned AEAD parameters. |
| java/tsfile/src/main/java/org/apache/tsfile/write/writer/ForceAppendTsFileWriter.java | Loads encryption header for append, resumes ordinal allocation, and closes owned AEAD parameters. |
| java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java | Propagates file-level AEAD context and ensures owned AEAD parameters are closed. |
| java/tsfile/src/main/java/org/apache/tsfile/write/page/ValuePageWriter.java | Uses centralized PageBodyEncoder and adds pageIndex/chunkOrdinal support. |
| java/tsfile/src/main/java/org/apache/tsfile/write/page/TimePageWriter.java | Uses centralized PageBodyEncoder and adds pageIndex/chunkOrdinal support. |
| java/tsfile/src/main/java/org/apache/tsfile/write/page/PageWriter.java | Uses centralized PageBodyEncoder and adds pageIndex/chunkOrdinal support. |
| java/tsfile/src/main/java/org/apache/tsfile/write/page/PageBodyEncoder.java | New helper to compress + encrypt page bodies (including AEAD AAD binding). |
| java/tsfile/src/main/java/org/apache/tsfile/write/page/EncodedPageBody.java | New lightweight holder for encoded page body bytes (data/offset/size). |
| java/tsfile/src/main/java/org/apache/tsfile/write/chunk/ValueChunkWriter.java | Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers. |
| java/tsfile/src/main/java/org/apache/tsfile/write/chunk/TimeChunkWriter.java | Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers. |
| java/tsfile/src/main/java/org/apache/tsfile/write/chunk/ChunkWriterImpl.java | Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers. |
| java/tsfile/src/main/java/org/apache/tsfile/write/TsFileWriter.java | Propagates file-level AEAD context and closes owned AEAD parameters on close. |
| java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileSketchTool.java | Uses reader.getDataStartOffset() to handle optional encryption header. |
| java/tsfile/src/main/java/org/apache/tsfile/read/reader/page/LazyLoadPageData.java | Adds AEAD decrypt path using pageIndex/chunkOrdinal and validates plaintext sizes. |
| java/tsfile/src/main/java/org/apache/tsfile/read/reader/chunk/ChunkReader.java | Threads pageIndex/chunkOrdinal into page readers and adds AEAD page decrypt/uncompress overloads. |
| java/tsfile/src/main/java/org/apache/tsfile/read/reader/chunk/AbstractAlignedChunkReader.java | Threads pageIndex/chunkOrdinal through aligned chunk page reads. |
| java/tsfile/src/main/java/org/apache/tsfile/read/common/Chunk.java | Rejects merging AEAD-encrypted chunks and ensures ordinal allocation during rewrite. |
| java/tsfile/src/main/java/org/apache/tsfile/read/UnClosedTsFileReader.java | Uses file-level encrypt param directly for unclosed reader. |
| java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java | Parses optional encryption header, tracks data start offset, records chunk ordinals, and supports AEAD page decryption with AAD. |
| java/tsfile/src/main/java/org/apache/tsfile/file/header/FileEncryptionHeader.java | New persisted encryption header format (versioned, size-bounded, wrapped key only). |
| java/tsfile/src/main/java/org/apache/tsfile/file/header/ChunkHeader.java | Adds optional chunk ordinal field for AEAD binding and updates (de)serialization accordingly. |
| java/tsfile/src/main/java/org/apache/tsfile/file/MetaMarker.java | Adds ENCRYPTION_HEADER marker for file-level encryption header. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/PageCryptoContext.java | New canonical AEAD page context with stable associated data construction. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncryptor.java | Adds encryptor selection for AEAD params and encryptPage default method. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncryptProvider.java | New SPI interface for provider implementations. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncrypt.java | Extends AutoCloseable and adds getPageBodyOverhead contract for AEAD profiles. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/IDecryptor.java | Adds decryptor selection for AEAD params and decryptPage default method. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptionProviderRegistry.java | New registry (and service loader integration) for stable provider ids. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptUtils.java | Adds AEAD-aware getEncryptParameter and getEncrypt overload. |
| java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptParameter.java | Extends parameters for AEAD (provider/profile ids, wrapped key, file id, chunk ordinal allocation, destroy/close). |
| java/common/src/main/resources/org/apache/tsfile/i18n/messages_zh.properties | Adds i18n strings for new encryption header/provider/AAD errors. |
| java/common/src/main/resources/org/apache/tsfile/i18n/messages.properties | Adds i18n strings for new encryption header/provider/AAD errors. |
🧠 Review effort: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+243
to
+247
| byte[] ordinalBytes = inputStream.readNBytes(Long.BYTES); | ||
| if (ordinalBytes.length != Long.BYTES) { | ||
| throw new EOFException(); | ||
| } | ||
| chunkOrdinal = ByteBuffer.wrap(ordinalBytes).getLong(); |
Comment on lines
+129
to
+131
| public byte[] getAssociatedData() { | ||
| return Arrays.copyOf(associatedData, associatedData.length); | ||
| } |
Comment on lines
2303
to
+2309
|
|
||
| public ByteBuffer readPage(PageHeader header, CompressionType type) throws IOException { | ||
| return readPage(header, type, 0); | ||
| } | ||
|
|
||
| public ByteBuffer readPage(PageHeader header, CompressionType type, int pageIndex) | ||
| throws IOException { |
Comment on lines
+50
to
+52
| if (providerId != null) { | ||
| PROVIDERS.remove(providerId); | ||
| } |
Copilot stopped reviewing on behalf of
HTHou due to an error
October 9, 2026 10:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Motivation
TsFile currently has legacy encryption interfaces but no self-describing file-level context for pluggable page-level authenticated encryption. This change establishes the format and I/O integration points while leaving algorithm and key-management implementations to external providers.
Compatibility
Validation
./mvnw spotless:check -P with-java -pl java/tsfile -am./mvnw test -P with-java -pl java/tsfile -am -Dtest='PageCryptoContextTest,FileEncryptionHeaderTest,TDEPageAeadTsFileTest,UnClosedTsFileReaderTest,ForceAppendTsFileWriterTest,RestorableTsFileIOWriterTest,TimePageWriterTest' -Dsurefire.failIfNoSpecifiedTests=falseDraft discussion points