fix(file-based): give each transferred file a unique staging path

[devin-ai-integration]

Summary

File transfer (use_file_transfer) staged every source file at {staging_dir}/{source_file_relative_path}. When two source files resolve to the same relative path, they share one staging file, which breaks syncs:

1. Source downloads file A to /staging/files/Untitled document.docx, emits a record referencing it.
2. Destination reads it, uploads it, then deletes the staging file (FileChunkTask → localFile.delete()).
3. Source downloads file B (a different source file) to the same path and emits another record.
4. Whichever side loses the race fails: the destination hits java.io.FileNotFoundException: /staging/files/Untitled document.docx (No such file or directory), or the destination reads bytes of B for record A.

This is easy to hit on Google Drive (duplicate document names are allowed, and exported Google-native docs are named {name}.{ext}) and on any source when preserve_directory_structure is disabled and the same file name exists in several folders.

Fix: namespace the local staging path under a unique subdirectory, leaving source_file_relative_path (and therefore the destination object key) unchanged:

-local_file_path = path.join(staging_directory, file_relative_path)
+local_file_path = path.join(staging_directory, uuid.uuid4().hex, file_relative_path)

Because LOCAL_FILE_PATH comes from AbstractFileBasedStreamReader._get_file_transfer_paths, this covers the base upload() as well as the readers that call the helper themselves (source-google-drive, source-s3, source-microsoft-sharepoint) with no connector-side change.

The Check: source-google-drive CI job here fails on test_upload_file, which asserts the exact staging path; the test-only update it needs is in airbytehq/airbyte#82754 (same for source-microsoft-sharepoint; source-s3 already asserts with endswith). That check should go green once the monorepo PR merges.

Out of scope but worth flagging: duplicate relative paths still map to a single destination object key, so one file overwrites the other in the destination. That is inherent to path-mirroring naming and is not what makes the sync fail.

Resolves https://github.com/airbytehq/oncall/issues/12851:

• https://github.com/airbytehq/oncall/issues/12851

Declarative-First Evaluation

Not applicable: the bug is in the Python CDK's file-based stream reader (staging path construction), not in a declarative connector manifest. No declarative feature can influence where the CDK stages a transferred file.

Test Coverage

• New test_staging_paths_are_unique_per_file in unit_tests/sources/file_based/test_file_based_stream_reader.py asserts that two files resolving to the same relative path get identical file_relative_path but distinct local_file_path (parametrized for the duplicate-name and preserve_directory_structure: false cases). It fails on main (both calls return the same local path).
• test_preserve_sub_directories_scenarios updated: it now asserts the local path is {staging_dir}/{unique_subdir}/{expected_relative_path} instead of a hardcoded path.
• poetry run pytest unit_tests/sources/file_based -q → 796 passed, 1 skipped. ruff check, ruff format --check, and mypy pass.

Versioning

No connector version bump or changelog entry: this repo publishes from release notes, and pyproject.toml version is computed dynamically. Non-breaking — no schema, spec, stream, or state change; only the ephemeral local staging path changes, and it is communicated to the destination via staging_file_url in the same record.

Link to Devin session: https://app.devin.ai/sessions/8ef791c2602147a2951781e0af756828

Important

Auto-merge enabled.

This PR is set to merge automatically when all requirements are met.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1785066655-unique-file-transfer-staging-paths#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1785066655-unique-file-transfer-staging-paths

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 229 tests  +2   4 217 ✅ +2   8m 24s ⏱️ +25s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 3e935ea. ± Comparison against base commit 5fc9b62.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 232 tests  +2   4 220 ✅ +2   10m 1s ⏱️ - 2m 47s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 3e935ea. ± Comparison against base commit 5fc9b62.

♻️ This comment has been updated with latest results.

[devin-ai-integration]

CI status on the two red connector checks:

• Check: source-google-drive — real consequence of this change, and expected. test_upload_file asserts the exact staging path (assert expected_paths["staging_file_url"] in file_reference.staging_file_url), which no longer holds once the uuid segment is inserted. The test-only update is in test(source-google-drive, source-microsoft-sharepoint): allow unique staging subdirectory airbyte#82754 (also covers source-microsoft-sharepoint; source-s3 already asserts with endswith). Verified locally against this branch: google-drive 7 passed, sharepoint 48 passed, s3 26 passed. This check should go green once that PR merges.
• Check: destination-motherduck — DockerHub connectivity, unrelated to this diff: failed to resolve source metadata for docker.io/docker/dockerfile:1: ... dial tcp 44.205.146.148:443: i/o timeout in test_docker_image_build_and_*. No CDK code involved; a re-run should clear it (I don't have permission to re-run jobs here).

All CDK checks — Pytest (all Python versions), Ruff, MyPy, package build — are green.

---

Devin session

[Copilot]

Pull request overview

This PR fixes a file-transfer staging-path collision in the Python CDK’s file-based stream reader by ensuring each transferred file is staged under a unique local subdirectory, preventing overwrites and race-driven FileNotFoundException failures when distinct source files resolve to the same relative path.

Changes:

• Namespace each staged local file path under a per-file UUID subdirectory while keeping source_file_relative_path unchanged.
• Update existing staging-path assertions to accommodate the new UUID subdirectory.
• Add a unit test asserting staging paths are unique even when relative paths collide.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
airbyte_cdk/sources/file_based/file_based_stream_reader.py	Adds UUID-based subdirectory to local staging paths to prevent collisions.
unit_tests/sources/file_based/test_file_based_stream_reader.py	Adjusts path assertions and adds a test ensuring staging-path uniqueness.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Daryna Ishchenko (darynaishchenko)]

Patrick Nilan (@pnilan) The change makes sense to me, so approving.
But looks like it affects unit tests for source-google-drive since its tests have checks for files paths. It should be fixed before/after the release of new cdk as follow up. Also can we double check that new pathes are the only reason of failed ci for google-drive?

[devin-ai-integration]

🙋 Human Input Needed: Daryna Ishchenko (@darynaishchenko) thanks for approving — answering your question, and the follow-up PR already exists, but it needs a maintainer to unblock two things. (Replying at top level since your review has no inline thread to reply to.)

Is the new path the only reason google-drive CI fails? Based on the job log, yes: all 7 failures in Check: source-google-drive are the same assertion in unit_tests/test_reader.py::test_upload_file, with 30 tests passing and no other errors. Every failure looks like:

assert '/tmp/airbyte-file-transfer/testdoc_pdf.pdf'
    in '/tmp/airbyte-file-transfer/c96ac3d31544485e952cff9939e52f60/testdoc_pdf.pdf'

i.e. the test asserts the exact staging path and only the new <uuid> segment differs. source_file_relative_path is unchanged in each of those records.

Follow-up PR for the connector tests: airbytehq/airbyte#82754 — it relaxes the exact-path assertions in source-google-drive and source-microsoft-sharepoint to prefix + relative-suffix while keeping the source_file_relative_path / file name / folder assertions intact. Test source-google-drive Connector is green there. Two things I can't do myself on it:

• source-microsoft-sharepoint unit tests pass (62) but the live docker check fails with Error listing files. Please check the credentials on 6 streams. My diff there only touches unit_tests/test_stream_reader.py, so it shouldn't be able to reach that path, but I couldn't find a clean master run to confirm — a second opinion would help.
• source-google-drive Progressive Rollout Gate needs a human to tick the "(Click to Approve:) Bypass the active progressive rollout warning" checkbox; I deliberately didn't self-approve that gate.

Also on this PR: Check: destination-motherduck fails at check with Your request is not authenticated. Please check your MotherDuck token plus PermissionError(13), which looks like a CI credential issue rather than this diff (motherduck doesn't use the file-based staging path). I get a 403 trying to re-run jobs, so someone with permissions may need to.

---

Devin session