Skip to content

feat(py-client): Add automatic resumable uploads - #667

Open
lcian wants to merge 14 commits into
mainfrom
lcian/feat/python-automatic-resumable-uploads
Open

lcian wants to merge 14 commits into
mainfrom
lcian/feat/python-automatic-resumable-uploads

Conversation

@lcian

@lcian lcian commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Session.put now automatically uses resumable uploads under the hood for eligible uploads. An upload is eligible if:

  • Its known remaining source size is at least resumable_threshold_bytes (default 32 MiB), measured before any compression
  • It's a bytes buffer or a seekable stream
    • If it's a stream, it doesn't need on-the-fly compression (so either it needs no compression at all, or is precompressed)

We could later extend this to streams that need compression but that means we'll need a "staging area" either on disk or in memory to hold the compressed version, so that requires some more thought.
The same mechanism could be used if the stream is non-rewindable to make it rewindable, though I'm not sure that's something we want to do.

Usecase gains the following kwargs, shown with their defaults:

Usecase(
    "debug_files",
    resumable_threshold_bytes=32 * 1024 * 1024,   # ``None`` disables resumables
    resumable_retries=objectstore_client.ResumableRetryPolicy(
        retries=2,   # ``retries = 0`` disables resumables
        delay=2.0,
        jitter=1.0
    ),
)

Session.put also gains resumable_threshold_bytes for per-request overrides, defaulting to the value set on the Usecase.

The reason why I decided to make resumable_retries configurable is because we already make the urllib3 retry policy configurable, and resumable requests actually honor it, overriding just whole-body retries with our own logic.
Introducing such behavior with a non-overridable default doesn't seem like a good option to me.

Closes FS-389

@codecov

This comment has been minimized.

@lcian lcian changed the title feat(python): Add automatic resumable uploads feat(py-client): Add automatic resumable uploads Oct 5, 2026
@linear-code

linear-code Bot commented Oct 5, 2026

Copy link
Copy Markdown

FS-389

Base automatically changed from lcian/feat/python-resumable-uploads to main October 6, 2026 09:56
@lcian
lcian force-pushed the lcian/feat/python-automatic-resumable-uploads branch from 79e839b to 46d4611 Compare October 6, 2026 12:41
@lcian

This comment has been minimized.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0af32a3. Configure here.

Comment thread clients/python/src/objectstore_client/_resumable.py
@lcian
lcian marked this pull request as ready for review October 6, 2026 15:54
@lcian
lcian requested a review from a team as a code owner October 6, 2026 15:54
Comment thread clients/python/src/objectstore_client/_resumable.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant