Skip to content

Fixed the silent-batch-drop bug where OpenSearch bulk indexing fails during shard reallocation - #27031

Merged
moesterheld merged 3 commits into
masterfrom
fix/retry-incomplete-bulk-response
Aug 20, 2026
Merged

Fixed the silent-batch-drop bug where OpenSearch bulk indexing fails during shard reallocation#27031
moesterheld merged 3 commits into
masterfrom
fix/retry-incomplete-bulk-response

Conversation

@todvora

@todvora todvora commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Description

During a rolling restart, OpenSearch can return a bulk response whose _shards.failures[] entry omits the shard number while a shard is still being promoted; the opensearch-java 3.x client treats that field as required and throws MissingRequiredPropertyException (see opensearch-java#551, fix proposed in #2023 but not yet merged), which didn't match any condition in Messages#createBulkRequestRetryerBuilder()'s retry predicate and caused the entire in-flight batch to be dropped instead of retried. MessagesAdapterOS now detects this specific failure (missing shard on ShardSearchFailure, to avoid masking unrelated parsing bugs) and converts it into a new IncompleteBulkResponseException, which Messages retries with the same exponential backoff as other transient indexing failures until the shard reallocation completes.

Motivation and Context

Fixes #26853

How Has This Been Tested?

Added unit tests

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Refactoring (non-breaking change)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have requested a documentation update.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

@todvora
todvora marked this pull request as ready for review August 20, 2026 05:16

@moesterheld moesterheld left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. for a backport, we will take into account the retyping from ShardFailure there to ShardSearchFailure

@moesterheld
moesterheld merged commit e83ae65 into master Aug 20, 2026
25 checks passed
@moesterheld
moesterheld deleted the fix/retry-incomplete-bulk-response branch August 20, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Silent potential message loss during OpenSearch primary reallocation, unhandled ShardFailure deserialization exception drops batch

2 participants