fix(ddc): wrap ufsTotal status update with retry on conflict#5695
fix(ddc): wrap ufsTotal status update with retry on conflict#5695jakharmonika364 wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical reliability issue in the DDC (Distributed Data Cache) component, specifically concerning the UfsTotal status updates for Alluxio and GooseFS runtimes. Previously, these updates could fail silently under concurrent modification scenarios, leading to incorrect metadata synchronization behavior. By implementing a retry.RetryOnConflict mechanism, the system now gracefully handles these conflicts, ensuring that UfsTotal status is consistently and accurately reflected, which is crucial for the correct functioning of metadata synchronization. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
Hi @jakharmonika364. Thanks for your PR. I'm waiting for a fluid-cloudnative member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request correctly identifies a race condition during dataset status updates and introduces retry.RetryOnConflict to make the update more robust. This is a good improvement. However, the implementation introduces a bug where the error from the retry block is overwritten and lost if the subsequent SyncMetadata call succeeds. I've added comments with suggestions to fix this by propagating the error. Additionally, I've pointed out the usage of context.TODO() which could be improved by passing a proper context from the caller.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5695 +/- ##
==========================================
- Coverage 64.86% 64.85% -0.01%
==========================================
Files 480 480
Lines 33514 33520 +6
==========================================
+ Hits 21738 21740 +2
- Misses 10051 10054 +3
- Partials 1725 1726 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jakharmonika364
force-pushed
the
bugfix/3377-metadata-update-conflict
branch
from
e7f9dc7 to
b6f3010
Compare
|
jakharmonika364
force-pushed
the
bugfix/3377-metadata-update-conflict
branch
from
b6f3010 to
dca14bc
Compare
|
@cheyang Can I get your review please?? |
jakharmonika364
force-pushed
the
bugfix/3377-metadata-update-conflict
branch
5 times, most recently
from
1d05d73 to
0e2975f
Compare
|
/retest |
|
@jakharmonika364: Cannot trigger testing until a trusted user reviews the PR and leaves an DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
jakharmonika364
force-pushed
the
bugfix/3377-metadata-update-conflict
branch
from
0e2975f to
51a82db
Compare
|
|
This PR has merge conflicts and cannot be merged in its current state. The change to wrap ufsTotal status update with retry on conflict looks reasonable in principle. Please rebase onto the latest master to resolve the conflicts, then this can proceed with review. |
Signed-off-by: Monika Jakhar <jakharmonika364@gmail.com>
jakharmonika364
force-pushed
the
bugfix/3377-metadata-update-conflict
branch
from
51a82db to
c4973d8
Compare
|
|
@cheyang Rebased onto master, conflicts resolved. Ready for review. |
2 participants
Ⅰ. Describe what this PR does
In processUpdatingUFS() for both AlluxioRuntime and GooseFSRuntime, the call that resets
UfsTotalto"Calculating"before triggering SyncMetadata() wasn't wrapped withretry.RetryOnConflict. If a concurrent update caused a conflict, it would silently fail and SyncMetadata() would skip syncing because it saw a non-emptyUfsTotal. This PR wraps that update withretry.RetryOnConflict, consistent with how similar status updates are handled elsewhere in the package.Ⅱ. Does this pull request fix one issue?
fixes #3377
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
No new tests added. The fix mirrors the existing retry pattern already used in the same package (e.g. metadata.go). Existing tests continue to pass.
Ⅳ. Describe how to verify it
UfsTotalin the dataset status updates correctly after the mount changeⅤ. Special notes for reviews
Same fix applied to both pkg/ddc/alluxio/ufs_internal.go and pkg/ddc/goosefs/ufs_internal.go since they had identical code paths with the same bug.