From 4ac78dcbfd6ab1d79ca29c5d7c2eaeb40cbcfd1f Mon Sep 17 00:00:00 2001 From: An Nguyen Date: Mon, 29 Jun 2026 19:49:45 +0000 Subject: [PATCH 1/3] docs: add RFC draft for Cloud Bigtable storage backend Proposes the design of a production-grade Cloud Bigtable storage backend for ExtendDB. Key items include a Lock-then-Read 2PC transaction flow, sharded TTL indexing shadow table, GSI projection enforcement, and secure GCP authentication config. --- docs/rfcs/0000-bigtable-backend.md | 168 +++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 docs/rfcs/0000-bigtable-backend.md diff --git a/docs/rfcs/0000-bigtable-backend.md b/docs/rfcs/0000-bigtable-backend.md new file mode 100644 index 00000000..a7f4c172 --- /dev/null +++ b/docs/rfcs/0000-bigtable-backend.md @@ -0,0 +1,168 @@ +# RFC-0003: Cloud Bigtable Storage Backend for ExtendDB + +- Status: Draft +- Author: @annguy3n +- Created: 2026-06-29 +- Tracking issue: #185 + +## Summary + +This RFC defines the design and implementation of a production-grade Cloud Bigtable storage backend for ExtendDB. It addresses critical architectural requirements for this new backend: transaction isolation, stream emissions in transactions, Global Secondary Index (GSI) projections, GSI write consistency, Time-to-Live (TTL) eviction scalability, and secure Google Cloud Platform (GCP) connectivity. + +## Motivation + +A naive mapping of DynamoDB structures to Bigtable introduces several architectural challenges that must be addressed to ensure production-readiness: +1. **Transaction Isolation Leak:** Non-transactional (single-row) writes bypass 2-Phase Commit (2PC) locks (intents), enabling dirty writes and lost updates. +2. **2PC Race Condition (Read-then-Lock):** The 2PC coordinator reads row data before placing locks, exposing the system to Time-of-Check to Time-of-Use (TOCTOU) races. +3. **Missing Transaction Streams:** Multi-row transactions (`TransactWriteItems`) do not emit DynamoDB Stream records. +4. **Inefficient TTL Eviction:** The TTL worker performs full-table scans to identify expired items, which degrades performance on large datasets. +5. **GSI Consistency Gap:** GSI shadow table write failures are logged without retry or repair, causing permanent base-to-shadow divergence. +6. **Auth Configuration Constraints:** Lack of support for explicit credentials file configuration. + +This RFC provides the technical specification to resolve these challenges. + +## Detailed Design + +### 1. Table Mapping & Naming Constraints + +* **Catalog:** System metadata, accounts, and policies are stored in a single table named `__extenddb_catalog__`. +* **Data Tables:** DynamoDB tables map to Bigtable tables named `t` (where `table_id_hex` is a 32-character hex UUID). Data attributes are stored in column family `d`. +* **GSIs:** Shadow tables are named `t_g` (where `idx_hash` is an 8-character hash of the index name). +* **Transaction Intents:** Column family `m` in data tables stores lock/intent cells. +* **TTL Markers:** Column family `t` (reserved). + +> [!NOTE] +> **Naming Constraints:** Table IDs in Cloud Bigtable are limited to 50 characters. To prevent overflow, GSIs are named `t_g` (43 characters total) rather than reproducing the full index name. + +### 2. Transaction Isolation & 2PC Refactoring + +#### A. Lock-then-Read 2PC Flow +Refactor the `TransactWriteItems` coordinator to acquire locks *before* reading data and evaluating condition expressions. + +```mermaid +sequenceDiagram + participant C as Coordinator (ExtendDB) + participant B as Bigtable + + C->>B: Phase 1: Open Transaction (Write PENDING to __extenddb_txn_log__) + Note over C,B: Place Intents (Locks) + loop For each participant row + C->>B: Place Intent (CheckAndMutateRow: fail if other fresh intent exists) + end + alt Any Lock Fails + C->>B: Rollback (Clear placed intents, drop txn row) + Note over C: Return TransactionConflict + end + Note over C,B: Read & Verify + C->>B: Read participant rows (now locked by us) + Note over C: Evaluate Condition Expressions + alt Any Condition Fails + C->>B: Rollback (Clear intents, drop txn row) + Note over C: Return TransactionCanceled + end + C->>B: Phase 2: Commit Transaction (Write COMMITTED to __extenddb_txn_log__) + Note over C,B: Apply Mutations + loop For each participant row + C->>B: Apply Data Mutations & Clear Intent (Atomic MutateRow) + end + C->>B: Phase 3: Clean up (Write CLEANED to log, then drop txn row) +``` + +#### B. Guarding Single-Row Writes +All single-row mutations (`PutItem`, `UpdateItem`, `DeleteItem`) must run via `CheckAndMutateRow` to prevent overwriting active 2PC locks: +* **Predicate:** Matches if a cell exists in family `m` with qualifier `intent:*` and a timestamp $\ge$ `now - intent_timeout`. +* **False Mutations (No Lock):** Execute the mutation. +* **True Mutations (Locked):** None. Returns `TransactionConflict` (client retries). +* **Deletes:** Use `DeleteFromFamily(d)` instead of `DeleteFromRow` to preserve the intent family `m`. + +#### C. Recovery Sweeper +A background worker periodically scans `__extenddb_txn_log__` for transactions older than `intent_timeout`: +* **`PENDING` State:** Clear intents on participant rows and delete the coordinator row (rollback). +* **`COMMITTED` State:** Re-apply mutations to participant rows, write stream records, clear intents, and delete the coordinator row (roll-forward). + +### 3. Stream Emissions in Transactions + +To ensure stream record atomicity: +1. **Phase 1 Resolution:** Fetch the full `TableDescription` (including `StreamSpecification` and `latest_stream_arn`) and pre-generate the `StreamRecord` payloads. +2. **Log Enrichment:** Store the pre-generated stream record payloads in the coordinator row of `__extenddb_txn_log__` in Phase 2. +3. **Phase 5 Commit:** Write the pre-generated stream records to the stream table alongside the data mutations. +4. **Recovery:** The recovery sweeper uses the payloads stored in the log to roll forward stream writes on crash. + +### 4. Scalable TTL Indexing + +Avoid full-table scans by introducing a sharded TTL index table. + +#### A. TTL Index Table Schema +* **Table Name:** `__extenddb_ttl_index__` +* **Row Key format (Binary):** `[shard_id:1] [expiry_timestamp_be:8] [account_id_len:1] [account_id] [table_name_len:1] [table_name] [encoded_base_row_key]` + * `shard_id` (1 byte): `hash(base_row_key) % 16` (prevents write hotspotting). + * `expiry_timestamp_be` (8 bytes): Big-Endian `u64` representing the TTL epoch second. + * `encoded_base_row_key` (variable): Raw row key of the target item. +* **Payload:** None (empty values). + +#### B. Maintenance +* **Writes:** Insert an empty row in `__extenddb_ttl_index__` when writing an item with a TTL attribute. +* **Updates/Deletes:** Delete the old index entry using the item's prior image. + +#### C. Sweeper Flow +1. Perform parallel range scans across all 16 shards from `start_key = [S] [0]` to `end_key = [S] [current_time_epoch_s + 1]`. +2. For each expired entry, read the base row and verify its current TTL matches the index entry. +3. If valid, execute `delete_item` on the base table. +4. If invalid (stale index), delete the index entry. + +### 5. GSI Projections & Consistency + +#### A. Projection Validation +Enforce DynamoDB GSI projection behavior: +* **Read Path:** Queries on GSIs with `KEYS_ONLY` or `INCLUDE` projections return only the projected attributes. +* **Validation:** If the client requests `Select: ALL_ATTRIBUTES` (or requests non-projected attributes) on a non-`ALL` GSI, return `ValidationException`. Do not perform base table fetches. +* **Defaulting:** If `Select` is omitted in a GSI query, default to `Select::AllProjectedAttributes`. + +#### B. Background Reconciler +Implement a reconciler as an internal background worker spawned via `RuntimeHooks`. The worker scans GSIs, compares them with the base tables, and repairs missing, orphaned, or mismatched shadow rows. + +### 6. GCP Credentials Configuration + +Extend `BigtableStorageConfig` to support explicit service account files: + +```toml +[storage.bigtable] +project_id = "my-project" +instance_id = "my-instance" +data_instance_id = "my-data-instance" # Optional +credentials_path = "/path/to/sa-key.json" # Optional +pool_size = 20 +``` + +* **Behavior:** If `credentials_path` is specified, the server programmatically sets the `GOOGLE_APPLICATION_CREDENTIALS` environment variable before client initialization. Configure `tonic` channels with `ClientTlsConfig` using native system roots. + +### 7. Implementation Sketch + +* **`config.rs`**: Add `credentials_path` and `data_instance_id` to config. +* **`client.rs` & `admin.rs`**: Configure TLS and inject bearer tokens. +* **`item_ops.rs`**: Update mutations to use `CheckAndMutateRow` and target deletes to the `d` family. +* **`transact.rs`**: Refactor `TxnCoordinator` to Lock-then-Read and implement the recovery sweeper. +* **`engine.rs`**: Update `transact_write_items`, GSI retry tasks, and stream record writes. +* **`ttl_worker.rs`**: Implement sharded TTL sweeps. +* **`gsi_reconciler.rs`**: Implement background GSI reconciliation. + +## Testing Strategy + +1. **Unit Tests (Emulator-based):** + * Verify type encoding/decoding and lexicographical sorting of numeric keys. + * Mock Bigtable client to verify 2PC coordinator state transitions. + * Assert `ValidationException` is returned for invalid GSI attribute requests. +2. **Failure Integration Tests (Real Bigtable & Emulator):** + * Inject crashes during the 2PC flow and assert that the recovery sweeper rolls forward/back and preserves stream record writes. + * Run concurrent mixed workloads to verify transactional serializability. + * Verify GSI reconciler heals base-to-shadow mismatches. + * Verify TTL sweeper ignores updated items with old index entries. +3. **E2E & Chaos Tests:** + * Run the Python integration suite (`devtools/run-tests --extenddb --pytest`) against production Bigtable. + * Inject network partitions during `TransactWriteItems` to verify 2PC resilience. + +## Alternatives Considered + +* **Unprotected Single-Row Writes:** Bypassing intent checks for single-row writes allows dirty writes, violating serializability. Rejected. +* **Scan-Based TTL Sweeper:** Scanning the entire base table to evict items degrades performance at scale. Rejected. +* **Cloud Spanner Backend:** Cloud Spanner supports multi-row transactions natively, eliminating the need for application-layer 2PC. This is a viable alternative to be explored as a separate backend connector. From aa6a7a5a8831a56296560f95e95c382b6a23701b Mon Sep 17 00:00:00 2001 From: An Nguyen Date: Tue, 11 Aug 2026 13:13:04 +0000 Subject: [PATCH 2/3] docs(rfc): update RFC with maintainer feedback clarifications - Promoted exact decimal number key encoding to core design specification. - Documented TransactWriteItems ClientRequestToken idempotency (10m) and duplicate item rejection rules. - Clarified read isolation guarantees (no dirty reads via CF 'd' snapshot writes and serializable TransactGetItems). - Documented TTL sweeper REMOVE stream emission with service principal identity. - Added explicit ConsistentRead=true rejection on GSIs. - Called out cross-row visibility in apply window and PITR retention differences. --- docs/rfcs/0000-bigtable-backend.md | 64 +++++++++++++++++++----------- 1 file changed, 40 insertions(+), 24 deletions(-) diff --git a/docs/rfcs/0000-bigtable-backend.md b/docs/rfcs/0000-bigtable-backend.md index a7f4c172..dc59fd6e 100644 --- a/docs/rfcs/0000-bigtable-backend.md +++ b/docs/rfcs/0000-bigtable-backend.md @@ -23,8 +23,9 @@ This RFC provides the technical specification to resolve these challenges. ## Detailed Design -### 1. Table Mapping & Naming Constraints +### 1. Table Mapping & Key Encoding +#### A. Table Structure & Naming * **Catalog:** System metadata, accounts, and policies are stored in a single table named `__extenddb_catalog__`. * **Data Tables:** DynamoDB tables map to Bigtable tables named `t` (where `table_id_hex` is a 32-character hex UUID). Data attributes are stored in column family `d`. * **GSIs:** Shadow tables are named `t_g` (where `idx_hash` is an 8-character hash of the index name). @@ -34,10 +35,23 @@ This RFC provides the technical specification to resolve these challenges. > [!NOTE] > **Naming Constraints:** Table IDs in Cloud Bigtable are limited to 50 characters. To prevent overflow, GSIs are named `t_g` (43 characters total) rather than reproducing the full index name. -### 2. Transaction Isolation & 2PC Refactoring +#### B. Exact Decimal & Order-Preserving Key Encoding +Cloud Bigtable sorts row keys by raw byte lexicographical order. To maintain exact DynamoDB semantics: +* **String & Binary Keys:** Encoded verbatim with null-byte / length-prefixed terminators to preserve lexicographical sort order. +* **Number Keys:** DynamoDB supports numeric values with up to 38 significant digits and requires numerical sorting (e.g., `-10 < -2 < 0 < 1.5 < 2 < 100`). Numbers are stored as exact arbitrary-precision decimals (never floating point floats) using an order-preserving byte encoding: + * **Negative numbers:** Sign byte `0x00`, followed by bit-inverted exponent and complement mantissa bytes. + * **Zero:** Exactly `[0x80]`. + * **Positive numbers:** Sign byte `0x81`, followed by big-endian exponent bytes and binary-coded decimal mantissa bytes. +* **GSI Index Row Keys:** Primary keys in GSI shadow tables are formatted as `[pk_val]\0[sk_val]\0[base_pk_val]\0[base_sk_val]`, ensuring exact numeric/string sort parity and uniqueness. -#### A. Lock-then-Read 2PC Flow -Refactor the `TransactWriteItems` coordinator to acquire locks *before* reading data and evaluating condition expressions. +### 2. Transaction Rules & Isolation Guarantees + +#### A. Transaction Validation & Idempotency Rules +* **`ClientRequestToken` Idempotency:** `TransactWriteItems` enforces idempotency within a 10-minute sliding window. Requests with an active or recently committed token replay the committed result; conflicting requests with the same token but mismatched payload return `IdempotentParameterMismatchException`. +* **Duplicate Participant Rejection:** A single `TransactWriteItems` request cannot contain multiple operations on the same item. Any transaction with duplicate primary keys is rejected upfront with `ValidationException`. + +#### B. Lock-then-Read 2PC Flow +Refactor the `TransactWriteItems` coordinator to acquire locks *before* reading data and evaluating condition expressions: ```mermaid sequenceDiagram @@ -68,14 +82,19 @@ sequenceDiagram C->>B: Phase 3: Clean up (Write CLEANED to log, then drop txn row) ``` -#### B. Guarding Single-Row Writes +#### C. Read Isolation & Concurrency Guarantees +* **Read Committed (No Dirty Reads):** In Phase 3, the coordinator only writes intent lock markers to column family `m`. Actual data mutations in column family `d` are **only applied in Phase 7** after the coordinator's `COMMITTED` record is durable in `__extenddb_txn_log__`. Standard reads (`GetItem`, `Query`, `Scan`, `BatchGetItem`) read solely from column family `d` and therefore never observe uncommitted candidate writes that could roll back. +* **Serializable `TransactGetItems`:** `TransactGetItems` observes the committed set atomically. Before reading data rows, it checks for active intent markers in column family `m`. If an active lock is present, it coordinates with `__extenddb_txn_log__` to observe the snapshot at the transaction commit boundary, preventing dirty or partial reads. +* **Cross-Row Visibility for Standard Reads:** During the 2PC apply window (Phase 7), standard `Query` or `Scan` operations may observe some participant rows updated before others across separate Bigtable tables. This is fully compliant with DynamoDB's `Read Committed` isolation model, as every observed row reflects a durable, committed state. + +#### D. Guarding Single-Row Writes All single-row mutations (`PutItem`, `UpdateItem`, `DeleteItem`) must run via `CheckAndMutateRow` to prevent overwriting active 2PC locks: * **Predicate:** Matches if a cell exists in family `m` with qualifier `intent:*` and a timestamp $\ge$ `now - intent_timeout`. * **False Mutations (No Lock):** Execute the mutation. * **True Mutations (Locked):** None. Returns `TransactionConflict` (client retries). * **Deletes:** Use `DeleteFromFamily(d)` instead of `DeleteFromRow` to preserve the intent family `m`. -#### C. Recovery Sweeper +#### E. Recovery Sweeper A background worker periodically scans `__extenddb_txn_log__` for transactions older than `intent_timeout`: * **`PENDING` State:** Clear intents on participant rows and delete the coordinator row (rollback). * **`COMMITTED` State:** Re-apply mutations to participant rows, write stream records, clear intents, and delete the coordinator row (roll-forward). @@ -88,7 +107,7 @@ To ensure stream record atomicity: 3. **Phase 5 Commit:** Write the pre-generated stream records to the stream table alongside the data mutations. 4. **Recovery:** The recovery sweeper uses the payloads stored in the log to roll forward stream writes on crash. -### 4. Scalable TTL Indexing +### 4. Scalable TTL Indexing & Streams Integration Avoid full-table scans by introducing a sharded TTL index table. @@ -104,22 +123,24 @@ Avoid full-table scans by introducing a sharded TTL index table. * **Writes:** Insert an empty row in `__extenddb_ttl_index__` when writing an item with a TTL attribute. * **Updates/Deletes:** Delete the old index entry using the item's prior image. -#### C. Sweeper Flow +#### C. Sweeper Flow & Stream Emission 1. Perform parallel range scans across all 16 shards from `start_key = [S] [0]` to `end_key = [S] [current_time_epoch_s + 1]`. 2. For each expired entry, read the base row and verify its current TTL matches the index entry. 3. If valid, execute `delete_item` on the base table. -4. If invalid (stale index), delete the index entry. +4. **TTL Stream Records:** When DynamoDB Streams are enabled on the table, TTL deletions emit a `REMOVE` stream record containing `userIdentity.type = "Service"` and `userIdentity.principalId = "dynamodb.amazonaws.com"`, matching native DynamoDB behavior. +5. If invalid (stale index), delete the index entry. ### 5. GSI Projections & Consistency -#### A. Projection Validation +#### A. Projection Validation & ConsistentRead Rejection Enforce DynamoDB GSI projection behavior: * **Read Path:** Queries on GSIs with `KEYS_ONLY` or `INCLUDE` projections return only the projected attributes. * **Validation:** If the client requests `Select: ALL_ATTRIBUTES` (or requests non-projected attributes) on a non-`ALL` GSI, return `ValidationException`. Do not perform base table fetches. +* **ConsistentRead Rejection:** Queries on GSIs with `ConsistentRead = true` are rejected with `ValidationException` (matching DynamoDB API rules). * **Defaulting:** If `Select` is omitted in a GSI query, default to `Select::AllProjectedAttributes`. #### B. Background Reconciler -Implement a reconciler as an internal background worker spawned via `RuntimeHooks`. The worker scans GSIs, compares them with the base tables, and repairs missing, orphaned, or mismatched shadow rows. +Implement a reconciler as an internal background worker spawned via `RuntimeHooks`. The worker scans GSIs, compares them with the base tables, and repairs missing, orphaned, or mismatched shadow rows, preventing phantom records from returning during divergence windows. ### 6. GCP Credentials Configuration @@ -134,29 +155,23 @@ credentials_path = "/path/to/sa-key.json" # Optional pool_size = 20 ``` -* **Behavior:** If `credentials_path` is specified, the server programmatically sets the `GOOGLE_APPLICATION_CREDENTIALS` environment variable before client initialization. Configure `tonic` channels with `ClientTlsConfig` using native system roots. +* **Behavior:** If `credentials_path` is specified, the server programmatically acquires tokens from the service account key. Configure `tonic` channels with `ClientTlsConfig` using native system roots. -### 7. Implementation Sketch +### 7. Future Work & Non-Goals -* **`config.rs`**: Add `credentials_path` and `data_instance_id` to config. -* **`client.rs` & `admin.rs`**: Configure TLS and inject bearer tokens. -* **`item_ops.rs`**: Update mutations to use `CheckAndMutateRow` and target deletes to the `d` family. -* **`transact.rs`**: Refactor `TxnCoordinator` to Lock-then-Read and implement the recovery sweeper. -* **`engine.rs`**: Update `transact_write_items`, GSI retry tasks, and stream record writes. -* **`ttl_worker.rs`**: Implement sharded TTL sweeps. -* **`gsi_reconciler.rs`**: Implement background GSI reconciliation. +* **Backup / PITR / Export:** Point-in-Time Recovery (PITR) and automated backup snapshots are deferred to a follow-up RFC. Note that Cloud Bigtable's native backup retention windows and export capabilities will have different retention characteristics from DynamoDB's fixed 35-day PITR window. ## Testing Strategy 1. **Unit Tests (Emulator-based):** - * Verify type encoding/decoding and lexicographical sorting of numeric keys. - * Mock Bigtable client to verify 2PC coordinator state transitions. - * Assert `ValidationException` is returned for invalid GSI attribute requests. + * Verify exact decimal number key encoding/decoding and numerical sorting up to 38 significant digits. + * Mock Bigtable client to verify 2PC coordinator state transitions and idempotency token deduplication. + * Assert `ValidationException` is returned for invalid GSI attribute requests and `ConsistentRead = true` on GSIs. 2. **Failure Integration Tests (Real Bigtable & Emulator):** * Inject crashes during the 2PC flow and assert that the recovery sweeper rolls forward/back and preserves stream record writes. * Run concurrent mixed workloads to verify transactional serializability. * Verify GSI reconciler heals base-to-shadow mismatches. - * Verify TTL sweeper ignores updated items with old index entries. + * Verify TTL sweeper emits `REMOVE` stream records with the TTL principal and ignores updated items with old index entries. 3. **E2E & Chaos Tests:** * Run the Python integration suite (`devtools/run-tests --extenddb --pytest`) against production Bigtable. * Inject network partitions during `TransactWriteItems` to verify 2PC resilience. @@ -166,3 +181,4 @@ pool_size = 20 * **Unprotected Single-Row Writes:** Bypassing intent checks for single-row writes allows dirty writes, violating serializability. Rejected. * **Scan-Based TTL Sweeper:** Scanning the entire base table to evict items degrades performance at scale. Rejected. * **Cloud Spanner Backend:** Cloud Spanner supports multi-row transactions natively, eliminating the need for application-layer 2PC. This is a viable alternative to be explored as a separate backend connector. + From 23f6956ff40b2ced6de488aa98ffac27eaaa4400 Mon Sep 17 00:00:00 2001 From: An Nguyen Date: Fri, 14 Aug 2026 14:30:33 +0000 Subject: [PATCH 3/3] docs(rfc): address maintainer feedback on phases, isolation, and reverse-sync spec --- docs/rfcs/0000-bigtable-backend.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/rfcs/0000-bigtable-backend.md b/docs/rfcs/0000-bigtable-backend.md index dc59fd6e..2919d36b 100644 --- a/docs/rfcs/0000-bigtable-backend.md +++ b/docs/rfcs/0000-bigtable-backend.md @@ -27,22 +27,22 @@ This RFC provides the technical specification to resolve these challenges. #### A. Table Structure & Naming * **Catalog:** System metadata, accounts, and policies are stored in a single table named `__extenddb_catalog__`. -* **Data Tables:** DynamoDB tables map to Bigtable tables named `t` (where `table_id_hex` is a 32-character hex UUID). Data attributes are stored in column family `d`. -* **GSIs:** Shadow tables are named `t_g` (where `idx_hash` is an 8-character hash of the index name). +* **Data Tables:** DynamoDB tables map to Bigtable tables named `t` (where `table_id_hex` is the first 16 hex characters of the table UUID, 17 characters total). Data attributes are stored in column family `d`. +* **GSIs:** Shadow tables are named `t_g` (where `idx_hash` is an 8-character hash of the index name, 27 characters total). * **Transaction Intents:** Column family `m` in data tables stores lock/intent cells. * **TTL Markers:** Column family `t` (reserved). > [!NOTE] -> **Naming Constraints:** Table IDs in Cloud Bigtable are limited to 50 characters. To prevent overflow, GSIs are named `t_g` (43 characters total) rather than reproducing the full index name. +> **Naming Constraints:** Table IDs in Cloud Bigtable are limited to 50 characters. To prevent overflow, GSIs are named `t_g` (27 characters total) rather than reproducing the full index name. #### B. Exact Decimal & Order-Preserving Key Encoding Cloud Bigtable sorts row keys by raw byte lexicographical order. To maintain exact DynamoDB semantics: * **String & Binary Keys:** Encoded verbatim with null-byte / length-prefixed terminators to preserve lexicographical sort order. * **Number Keys:** DynamoDB supports numeric values with up to 38 significant digits and requires numerical sorting (e.g., `-10 < -2 < 0 < 1.5 < 2 < 100`). Numbers are stored as exact arbitrary-precision decimals (never floating point floats) using an order-preserving byte encoding: - * **Negative numbers:** Sign byte `0x00`, followed by bit-inverted exponent and complement mantissa bytes. + * **Negative numbers:** Sign byte `0x00`, followed by 1-byte bit-inverted biased exponent (`~exp_biased`), and 38-byte inverted mantissa digits (`9 - digit`) padded with `0x09`. * **Zero:** Exactly `[0x80]`. - * **Positive numbers:** Sign byte `0x81`, followed by big-endian exponent bytes and binary-coded decimal mantissa bytes. -* **GSI Index Row Keys:** Primary keys in GSI shadow tables are formatted as `[pk_val]\0[sk_val]\0[base_pk_val]\0[base_sk_val]`, ensuring exact numeric/string sort parity and uniqueness. + * **Positive numbers:** Sign byte `0xFF`, followed by 1-byte biased exponent (`exp_biased = sci_exp + 130`), and 38-byte normalized decimal digits padded with `0x00`. +* **GSI Index Row Keys:** Primary keys in GSI shadow tables are formatted as `[encoded_gsi_pk][encoded_gsi_sk]\xFE[encoded_base_pk][encoded_base_sk]`, reusing the exact order-preserving/length-prefixed encoding of the base table. The `0xFE` byte acts as a structural separator between the secondary and base key components, avoiding bare null-byte collisions and preserving uniqueness and sort parity. ### 2. Transaction Rules & Isolation Guarantees @@ -83,9 +83,10 @@ sequenceDiagram ``` #### C. Read Isolation & Concurrency Guarantees -* **Read Committed (No Dirty Reads):** In Phase 3, the coordinator only writes intent lock markers to column family `m`. Actual data mutations in column family `d` are **only applied in Phase 7** after the coordinator's `COMMITTED` record is durable in `__extenddb_txn_log__`. Standard reads (`GetItem`, `Query`, `Scan`, `BatchGetItem`) read solely from column family `d` and therefore never observe uncommitted candidate writes that could roll back. -* **Serializable `TransactGetItems`:** `TransactGetItems` observes the committed set atomically. Before reading data rows, it checks for active intent markers in column family `m`. If an active lock is present, it coordinates with `__extenddb_txn_log__` to observe the snapshot at the transaction commit boundary, preventing dirty or partial reads. -* **Cross-Row Visibility for Standard Reads:** During the 2PC apply window (Phase 7), standard `Query` or `Scan` operations may observe some participant rows updated before others across separate Bigtable tables. This is fully compliant with DynamoDB's `Read Committed` isolation model, as every observed row reflects a durable, committed state. +* **Read Committed (No Dirty Reads):** In Phase 1, the coordinator only writes intent lock markers to column family `m`. Actual data mutations in column family `d` are **only applied in Phase 2** after the coordinator's `COMMITTED` record is durable in `__extenddb_txn_log__`. Standard reads (`GetItem`, `Query`, `Scan`, `BatchGetItem`) read solely from column family `d` and therefore never observe uncommitted candidate writes that could roll back. +* **Serializable `TransactGetItems`:** `TransactGetItems` observes the committed set atomically by reading all requested rows and checking for active intent markers in column family `m`. Because Bigtable lacks cross-row atomic snapshots, if *any* requested row has an active intent lock, the request immediately blocks and retries. By deferring the read until all participant locks are cleared (Phase 3), the read never observes partial transaction state, guaranteeing Serializable visibility. +* **Read-Your-Writes (Ack Point):** To guarantee that strongly-consistent `GetItem` or `Query` requests observe applied transaction data, the coordinator defers the HTTP 200 OK success acknowledgment (client ack) until *after* all Phase 2 mutations (data and streams) are definitively applied to column family `d`. +* **Cross-Row Visibility for Standard Reads:** During the 2PC apply window (Phase 2), standard `Query` or `Scan` operations may observe some participant rows updated before others across separate Bigtable tables. This is fully compliant with DynamoDB's `Read Committed` isolation model, as every observed row reflects a durable, committed state. #### D. Guarding Single-Row Writes All single-row mutations (`PutItem`, `UpdateItem`, `DeleteItem`) must run via `CheckAndMutateRow` to prevent overwriting active 2PC locks: @@ -102,10 +103,9 @@ A background worker periodically scans `__extenddb_txn_log__` for transactions o ### 3. Stream Emissions in Transactions To ensure stream record atomicity: -1. **Phase 1 Resolution:** Fetch the full `TableDescription` (including `StreamSpecification` and `latest_stream_arn`) and pre-generate the `StreamRecord` payloads. -2. **Log Enrichment:** Store the pre-generated stream record payloads in the coordinator row of `__extenddb_txn_log__` in Phase 2. -3. **Phase 5 Commit:** Write the pre-generated stream records to the stream table alongside the data mutations. -4. **Recovery:** The recovery sweeper uses the payloads stored in the log to roll forward stream writes on crash. +1. **Phase 1 (Prepare):** Fetch the full `TableDescription` (including `StreamSpecification` and `latest_stream_arn`) and pre-generate the `StreamRecord` payloads. +2. **Phase 2 (Log Enrichment & Commit):** Store the pre-generated stream record payloads and participant mutation payloads in the coordinator row of `__extenddb_txn_log__` alongside `COMMITTED`, and apply stream records to `__extenddb_streams__` concurrently with data mutations. +3. **Recovery:** The recovery sweeper uses the payloads stored in the log to roll forward stream writes on crash. ### 4. Scalable TTL Indexing & Streams Integration @@ -140,7 +140,7 @@ Enforce DynamoDB GSI projection behavior: * **Defaulting:** If `Select` is omitted in a GSI query, default to `Select::AllProjectedAttributes`. #### B. Background Reconciler -Implement a reconciler as an internal background worker spawned via `RuntimeHooks`. The worker scans GSIs, compares them with the base tables, and repairs missing, orphaned, or mismatched shadow rows, preventing phantom records from returning during divergence windows. +Implement a reconciler as an internal background worker spawned via `ServerRuntimeHooks::spawn_workers`. The worker scans GSIs, compares them with the base tables, and repairs missing, orphaned, or mismatched shadow rows, preventing phantom records from returning during divergence windows. ### 6. GCP Credentials Configuration