Repository navigation
[Bug]: BigQuery connector sets insane high maxRetryJobs for unbounded collections #28282
Description
What happened?
For unbounded collections and FileLoads writing method default value for maxRetryJobs is 1000. It delays step failure if the error is permanent, for example if TableRow contains timestamp field with invalid value (out of accepted range).
I found the following comment in the code:
// When running in streaming (unbounded mode) we want to retry failed load jobs
// indefinitely. Failing the bundle is expensive, so we set a fairly high limit on retries.
if (IsBounded.UNBOUNDED.equals(input.isBounded())) {
batchLoads.setMaxRetryJobs(getMaxRetryJobs());
}
public static <T> Write<T> write() {
return new AutoValue_BigQueryIO_Write.Builder<T>()
...
.setMaxRetryJobs(1000)
...
.build()
I would expect:
- no retries for persistent errors
- a few retries for transient errors but much less than 1000
In addition the Documentation is far for complete:
- No information that settings is only applicable for FileLoads
- No information about behaviour if maxRetryJobs is not specified
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
liferoad commentedon Sep 2, 2023
Please check more discussions here: #27495
liferoad commentedon Sep 2, 2023
@ahmedabu98 FYI.
ahmedabu98 commentedon Sep 5, 2023
@liferoad Note the discussion in #27495 is for
STORAGE_WRITE_APIwhereas this issue is forFILE_LOADS(two different code paths)ahmedabu98 commentedon Sep 5, 2023
The decision to retry a job is made in this block of code. Perhaps we can parse
jobStatus.getErrorResult()and fail the bundle for non-transient errors.@reuvenlax may have better ideas
ahmedabu98 commentedon Sep 5, 2023
FYI in BigQueryIO documentation, "batch loads" is generally synonymous with "file loads"
ahmedabu98 commentedon Sep 5, 2023
This may also get into the territory of needing to add a DLQ for file loads method.. some of these errors may be completely unretriable (e.g. schema mismatch), in which case even a bundle failure won't help
github-actions commentedon Aug 5, 2026
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
mxtymoshyk commentedon Aug 31, 2026
Still relevant, and the code quoted in the description has since moved, so here is where it stands.
The documentation half is addressed
Both doc gaps @mkuthan listed are fixed in #39938 (open, pending review), which rewrites the
withMaxRetryJobsJavadoc:That PR also removes the
.setMaxRetryJobs(1000)default fromBigQueryIO.write()that the description quotes.getMaxRetryJobs()is now a nullableIntegerthat is unset unless the user calls the setter, which is what let #28281 be fixed without raising the bounded default from 3 to 1000. The two defaults are now named constants,BatchLoads.DEFAULT_MAX_RETRY_JOBS(3) andBatchLoads.DEFAULT_MAX_RETRY_JOBS_UNBOUNDED(1000).The 1000 itself is deliberately untouched by that PR, because it is this issue.
What 1000 actually costs
The backoff between retries is configured in
PendingJobManager(BigQueryHelpers.java:120-128): 1s initial, exponent 1.5, capped at 60s,withMaxRetries(Integer.MAX_VALUE)and nowithMaxCumulativeBackoffoverride, soFluentBackoff's 1000-day cumulative default never binds.waitForDone(:161-168) sleeps once per polling round before re-running each still-pending job, andrunJobgives up whencurrentAttempt < maxRetries + 1fails (:212-221,:287-289).Nominal sleep sequence: 1s, 1.5s, 2.25s, ... reaching the 60s cap at the 12th sleep. The first 11 sum to about 171s; the remaining ~988 all sit at the cap for 59,280s.
Total: roughly 16.5 hours before a permanently failing load job surfaces as a bundle failure.
FluentBackoff's randomization factor of 0.5 puts the real range at roughly 8 to 25 hours. Each of those attempts is a load job that reached a terminalFAILEDstate, not a slow one still running, so this is genuinely 1000 doomed retries of the invalid-timestamp case in the description.The design question is still open
@ahmedabu98 suggested parsing
jobStatus.getErrorResult()and failing the bundle for non-transient errors, and asked @reuvenlax for better ideas. That was three years ago and there was no reply, so there has never been a decision to implement against.The options, as far as I can see:
Worth noting there is currently no test anywhere asserting the unbounded default or any retry count that would move, so whichever direction is taken would want a regression test alongside it.
I have the retry path paged in from working on #28281 and would be glad to implement whichever of these is wanted, but this changes streaming failure behaviour, so it seems like a call for the BigQueryIO maintainers rather than something to guess at. Is there a preferred direction, or should this go to dev@ first? Not claiming the issue in the meantime.