Skip to content

feat(realtime): support compaction for deduplicate tables - #449

Merged
SteNicholas merged 4 commits into
apache:mainfrom
lxy-9602:rt-dv-compact
Oct 31, 2026
Merged

SteNicholas merged 4 commits into
apache:mainfrom
lxy-9602:rt-dv-compact

Conversation

@lxy-9602

Copy link
Copy Markdown
Member

Purpose

Linked issue: #158

Support realtime deduplicate compaction by remapping the latest deletions to compacted files at commit time, preventing deleted rows from reappearing. Share DV materialization between write and compaction providers.

Tests

  • Unit tests for rewrite normalization, DV materialization and cleanup.
  • Integration tests for concurrent commits/retries, rolling outputs, restart, full deletion, partition/bucket isolation and no-hit commits.

API and Format

No public API, storage format or protocol changes.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex

@SteNicholas SteNicholas left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lxy-9602, I left one comment for this pull request. PTAL.

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@SteNicholas
SteNicholas merged commit 03d291f into apache:main Oct 31, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants