fix(file-based): sample one file per path shape when capping schema inference

[devin-ai-integration]

Overview

👉 TL;DR: When a file-based stream has too many files to read them all during schema discovery, the connector now samples one recent file from each distinct folder/file-name pattern instead of just the newest files. This stops buckets that mix several file types from producing a different schema on every discovery, which was flagging spurious breaking schema changes and disabling connections. Specifically, DefaultFileBasedStream._get_raw_json_schema now selects files via the new _select_files_for_schema_inference / _schema_inference_group_key helpers when capping to get_max_n_files_for_schema_inference().

Requested by Stephen Ibuoye (Airbyte support) for a Cloud S3 → Snowflake connection with ~67K heterogeneous Avro files.

Changes

• Add _schema_inference_group_key(uri): directory segments containing a digit (dates, ids, uuids, hashes) become *; the file name is reduced to its leading alphabetic prefix + extension.
• Add _select_files_for_schema_inference(files, n): newest-first, one file per group key first, then fill remaining slots with the next-newest files.
• Use it only in the "Refusing to infer schema for N files" branch; candidates are first sliced to the newest first_n_files, so recent_n_files_to_read_for_schema_discovery still bounds the window. use_first_found_file_for_schema_discovery is unaffected.
• Log the selected file URIs at info level so discovery diffs can be debugged from job logs.
• Unit tests for group-key normalization, heterogeneous selection order, homogeneous fallback, under-limit passthrough, the _get_raw_json_schema integration, and the recent-window bound.

Review Spotlight

Reviewers with limited time, please review first:

• default_file_based_stream.py#L306-L320: the capped branch now calls the diverse selector on the recent window
• default_file_based_stream.py#L335-L374: grouping heuristic and selection

Risks

• For datasets with a single path shape the selection is identical to the previous newest-first behavior, so no change there. For heterogeneous datasets the inferred schema can gain columns on the first discovery after upgrade (a non-breaking "column added" change).
• The grouping is a heuristic; if a dataset has more distinct path shapes than the cap (10), some shapes are still not sampled.

Follow-ups

• Bump source-s3 (and other file-based connectors) to a CDK release containing this change.

Test plan

• poetry run pytest unit_tests/sources/file_based/stream/test_default_file_based_stream.py -q → 33 passed
• poetry run ruff check / ruff format --check on the two changed files → clean
• poetry run mypy --config-file mypy.ini airbyte_cdk/sources/file_based/stream/default_file_based_stream.py → no issues

Design notes: why the sampling was unstable

Show/Hide Content

The affected bucket is a Pendo export writing .../<uuid>/matchedEvents/Page/Page-<hash>.avro, .../matchedEvents/Feature/Feature-<hash>.avro, .../allguides.avro, .../allpages.avro, ... under one prefix. The 10 newest files are frequently all Page files, so columns present only in the other event types (guideId, pollId, uiElementId, ...) drop out of the inferred schema. Those columns are part of the user-defined composite primary key with append_dedup, so the platform (CatalogDiffHelpers.fieldTransformBreaksConnection) marks the removal as breaking and disables the connection. Reviewing triggers another discovery against a different 10 files where the columns are present again, so the change appears to "auto-resolve". Sampling one file per path shape makes the sampled set cover every file family regardless of which family was exported most recently.

Link to Devin session: https://app.devin.ai/sessions/1ccc4b821f6049e698ef5810f88b3383
Open in Devin Desktop: https://app.devin.ai/desktop/session/1ccc4b821f6049e698ef5810f88b3383?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1789433905-schema-inference-path-diversity#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1789433905-schema-inference-path-diversity

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[Copilot]

🟡 Changes recommended

The selector can sample files outside the configured recent window.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates file-based schema inference to sample recent files across distinct path shapes when the file cap is reached.

Changes:

• Adds path-shape grouping and diverse file selection.
• Logs selected files for debugging.
• Adds unit and integration coverage.

Review finding (moderate, 3 votes): candidates should be limited to the configured recent-file window before diversification.

File summaries

File	Summary
unit_tests/sources/file_based/stream/test_default_file_based_stream.py	Tests grouping, selection, and integration behavior.
airbyte_cdk/sources/file_based/stream/default_file_based_stream.py	Implements diverse capped schema sampling.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[devin-ai-integration]

👍 On it. Correct — when recent_n_files_to_read_for_schema_discovery exceeds the policy cap, the selector should only diversify within the newest first_n_files, not the whole listing. Will slice to that window before selecting and add a test for it.

[devin-ai-integration]

☑️ Resolved in 14b8466. Candidates are now sliced to the newest first_n_files before path-diverse selection, so recent_n_files_to_read_for_schema_discovery still bounds the window; added test_when_recent_n_files_exceeds_cap_then_selection_stays_within_recent_window.

[github-actions]

PyTest Results (Fast)

4 398 tests  +10   4 386 ✅ +10   7m 32s ⏱️ - 2m 6s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 14b8466. ± Comparison against base commit 96a7c0a.

[github-actions]

PyTest Results (Full)

4 401 tests  +10   4 388 ✅ +9   9m 37s ⏱️ - 4m 24s
    1 suites ± 0      12 💤 ±0 
    1 files   ± 0       1 ❌ +1 

For more details on these failures, see this check.

Results for commit 14b8466. ± Comparison against base commit 96a7c0a.

[devin-ai-integration]

Closing per maintainer request; the problem is being tracked as an internal issue instead of prescribing this fix.

---

Devin session