Skip to content

LOAD TsFile: use the Pair the resolver takes and cover the HA cases - #18901

Open
luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:load-tsfile-consensus-review-fix
Open

luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:load-tsfile-consensus-review-fix

Conversation

@luoluoyuyu

Copy link
Copy Markdown
Member

Description

The router and the resolver each wrapped a (device, time partition) pair into a private record only to unwrap it again on the way in or out. Pair already carries the value equality the deduplication maps need, so keep the pair itself and drop both SlotKey records, the record and its conversion.

Cover the two cluster cases the consensus LOAD path has to survive:

  • a LOAD that runs while one DataNode is down: restarting that DataNode must replay the LOAD commands it missed and serve the loaded file.
  • a LOAD whose region is migrated mid-flight: the LOAD has to follow the refreshed route or fall back to the tablet path, but every point of the file must be imported and the migration must finish.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

The router and the resolver each wrapped a (device, time partition) pair
into a private record only to unwrap it again on the way in or out. Pair
already carries the value equality the deduplication maps need, so keep
the pair itself and drop both SlotKey records, the record and its
conversion.

Cover the two cluster cases the consensus LOAD path has to survive:

- a LOAD that runs while one DataNode is down: restarting that DataNode
  must replay the LOAD commands it missed and serve the loaded file.
- a LOAD whose region is migrated mid-flight: the LOAD has to follow the
  refreshed route or fall back to the tablet path, but every point of the
  file must be imported and the migration must finish.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant