Skip to content

fix(cloud): tolerate undecodable mappers and unknown statuses in connection/job lookups - #1335

Merged
Aaron ("AJ") Steers (aaronsteers) merged 5 commits into
release-candidate/v1from
devin/1791249227-graceful-connection-decode
Oct 7, 2026
Merged

Aaron ("AJ") Steers (aaronsteers) merged 5 commits into
release-candidate/v1from
devin/1791249227-graceful-connection-decode

Conversation

@aaronsteers

@aaronsteers Aaron ("AJ") Steers (aaronsteers) commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Summary

👉 TL;DR: Cloud connection and job lookups no longer fail outright when the bundled airbyte-api SDK (0.53.0) can't decode part of the response. This covers 1Password's Marketo/PartnerStack connections (field-filtering mappers), locked connections, and job statuses the SDK doesn't define.

When the SDK raises ValueError while decoding, we log a warning, re-fetch the raw Public API JSON, and decode it leniently: we drop configurations.streams[*].mappers, and we keep any status value the SDK enum doesn't define as a plain string. This PR adds no SDK monkeypatching. The queued job status is now registered on the SDK enum by #1334 (merged), so queued decodes on the first request.

Changes

  1. get_connection / list_connections: on ValueError, fall back to GET /connections/{id} (or the per-page GET /connections) and _lenient_decode_connection.
  2. get_job_info / get_job_logs: on ValueError, fall back to GET /jobs/{id} (or the per-page GET /jobs) and _lenient_decode_job. Pagination is unchanged and reads from either source.
  3. New private helpers:
    • _get_public_api_json: a bounded-timeout requests GET using _config_api_headers. HTTP 404 raises AirbyteMissingResourceError and 403 keeps the existing forbidden guidance, both matching _wrap_sdk_error.
    • _decode_with_unknown_status: decodes with a known placeholder status, then calls dataclasses.replace(status=raw_status).
    • _lenient_decode_connection and _lenient_decode_job.
  4. Public enums: ConnectionStatus.LOCKED added. JobStatusEnum.QUEUED came from fix(cloud): support the queued job status #1334. Both are additive, and both values are in the platform spec.
  5. Unit tests (responses-mocked so the real SDK decode raises):
    • field-filtering mapper (get + list)
    • a job status unknown to the SDK (get + list)
    • locked connection
    • fallback 404 → AirbyteMissingResourceError
    • the no-fallback path

Review Spotlight

  1. _decode_with_unknown_status / _lenient_decode_connection / _lenient_decode_job at the bottom of airbyte/_util/api_util.py.
  2. get_job_logs restructure: SDK and fallback paths both feed page_data / next_page.

Risks

  1. Only in the fallback path: mappers are dropped (nothing in PyAirbyte reads them), and status may hold a raw str instead of an SDK enum. Public CloudConnectionInfo / CloudJobInfo deliberately keep closed status enums, so a status unknown to our public enums still raises there; we'll add new values as the platform introduces them.
  2. The fallback catches any ValueError from the SDK call. If the lenient re-decode also fails, that error propagates as before.
  3. No dependency or lock changes.

Follow-ups

  1. Durable fix: replace Speakeasy with airbyte-server-models ( refactor(cloud): replace Speakeasy airbyte-api SDK with generated airbyte-server-models #1148 ). The airbyte-api 1.0.1 bump ( chore: adapt PyAirbyte to airbyte-api==1.0.1 breaking changes #1048 ) still fails with Could not find discriminator field algorithm.

Test plan

  1. uv run ruff check . && uv run ruff format --check .
  2. uv run pyrefly check: 0 errors
  3. uv run pytest tests/unit_tests/test_cloud_api_util.py -q: 108 passed (after merging release-candidate/v1)

Link to Devin session: https://app.devin.ai/sessions/b71c6ee648ab4a67b4e68990fc713daf
Open in Devin Desktop: https://app.devin.ai/desktop/session/b71c6ee648ab4a67b4e68990fc713daf?variant=devin
Requested by: Aaron ("AJ") Steers (@aaronsteers)

@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This PyAirbyte Version

You can test this version of PyAirbyte using the following:

# Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1791249227-graceful-connection-decode' pyairbyte --help

# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1791249227-graceful-connection-decode'

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /fix-pr - Fixes most formatting and linting issues
  • /uv-lock - Updates uv.lock file
  • /test-pr - Runs tests with the updated PyAirbyte
  • /prerelease - Builds and publishes a prerelease version to PyPI
📚 Show Repo Guidance

Helpful Resources

Community Support

Questions? Join the #pyairbyte channel in our Slack workspace.

📝 Edit this welcome message.

This comment was marked as resolved.

Co-Authored-By: AJ Steers <aj@airbyte.io>
@github-code-quality

github-code-quality Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest-no-creds

The overall line coverage in commit 05673e8 in the devin/1791249227-gra... branch is 77%. The line coverage in commit d9f652f in the release-candidate/v1 branch is 65%.

Show a line coverage summary of the most impacted files.
File release-candidate/v1 d9f652f devin/1791249227-gra... 05673e8 +/-
airbyte/_util/api_util.py 36% 65% +29%
airbyte/mcp/cloud.py 52% 82% +30%
airbyte/cloud/connectors.py 34% 75% +41%
airbyte/mcp/int..._registry_ui.py 0% 92% +92%
airbyte/mcp/_user_identity.py 0% 93% +93%
airbyte/_direct...ctors/models.py 0% 94% +94%
airbyte/mcp/_telemetry.py 0% 94% +94%
airbyte/mcp/_sso_auth.py 0% 96% +96%
airbyte/cloud/models.py 0% 97% +97%
airbyte/mcp/int...nc_status_ui.py 0% 97% +97%

Python / code-coverage/pytest

The overall line coverage in commit 05673e8 in the devin/1791249227-gra... branch is 80%. The line coverage in commit d9f652f in the release-candidate/v1 branch is 71%.

Show a line coverage summary of the most impacted files.
File release-candidate/v1 d9f652f devin/1791249227-gra... 05673e8 +/-
airbyte/cloud/connectors.py 51% 80% +29%
airbyte/mcp/cloud.py 52% 82% +30%
airbyte/mcp/_otel.py 0% 78% +78%
airbyte/mcp/int..._registry_ui.py 0% 92% +92%
airbyte/mcp/_user_identity.py 0% 93% +93%
airbyte/_direct...ctors/models.py 0% 94% +94%
airbyte/mcp/_telemetry.py 0% 94% +94%
airbyte/mcp/_sso_auth.py 0% 96% +96%
airbyte/cloud/models.py 0% 97% +97%
airbyte/mcp/int...nc_status_ui.py 0% 97% +97%

Python / code-coverage/pytest-unit

The overall line coverage in commit 05673e8 in the devin/1791249227-gra... branch is 74%. Line coverage data for the release-candidate/v1 branch is not yet available.

Show a line coverage summary of the most covered files.
File release-candidate/v1 devin/1791249227-gra... 05673e8 +/-
airbyte/cloud/models.py — 97% —
airbyte/mcp/_sso_auth.py — 96% —
airbyte/_direct...ctors/models.py — 94% —
airbyte/cloud/client.py — 90% —
airbyte/exceptions.py — 85% —
airbyte/mcp/cloud.py — 82% —
airbyte/progress.py — 80% —
airbyte/cloud/connectors.py — 75% —
airbyte/shared/...ql_processor.py — 68% —
airbyte/_util/api_util.py — 65% —

Updated October 07, 2026 20:32 UTC

…cked)

Co-Authored-By: AJ Steers <aj@airbyte.io>
@devin-ai-integration devin-ai-integration Bot changed the title fix(cloud): tolerate undecodable stream mappers in get_connection / list_connections fix(cloud): tolerate undecodable mappers and unknown statuses in connection/job lookups Oct 6, 2026

This comment was marked as resolved.

@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review October 6, 2026 02:23

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread airbyte/_util/api_util.py
Comment thread airbyte/_util/api_util.py
Co-Authored-By: AJ Steers <aj@airbyte.io>
@aaronsteers

Copy link
Copy Markdown
Member Author

Devin, update this PR from the base branch. Note the related PR has mergec.

…/1791249227-graceful-connection-decode

Co-Authored-By: AJ Steers <aj@airbyte.io>

# Conflicts:
#	airbyte/mcp/cloud.py
#	tests/unit_tests/test_cloud_api_util.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The fallback behavior is bounded, preserves existing paths, and has focused regression coverage.

0 open findings

🧠 Review effort: Balanced

@aaronsteers
Aaron ("AJ") Steers (aaronsteers) merged commit 9bd5637 into release-candidate/v1 Oct 7, 2026
28 checks passed
@aaronsteers
Aaron ("AJ") Steers (aaronsteers) deleted the devin/1791249227-graceful-connection-decode branch October 7, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants