Skip to content

[Feature Request]: Create Apache Beam PostgreSQL I/O Connector with streaming support #39959

Open

Description

@jk-shah

What would you like to happen?

Feature: Support a native PostgreSQL I/O with streaming support.

Expectation: Create a a dedicated, high-performance PostgreSQL connector with native PostgreSQL streaming and write sinks that works with databases that are either self-hosted or managed.

Requirements: Main functionality will be to deliver the initial snapshot and stream continuity. It should be able to support native PostgreSQL binary protocol and handle transactional and non-transactional logical messages. Ability to support two-phase commit transactions. It must be able to handle TOAST columns in Beam runner-managed state without issuing point queries to the live database. Handling DDL changes in logical replication is hard but should be able to handle DDL changes or have guardrails around it. Finally it needs to do it at scale even for write sink to PostgreSQL.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Activity

  1. jk-shah commented on Sep 1, 2026

    @jk-shah
    ContributorAuthor

    Planning to .take-issue to develop it myself using modern tools.

  2. alisha-1000 commented on Oct 23, 2026

    @alisha-1000
    Contributor

    Hello,

    I’m interested in contributing to the PostgreSQL I/O connector initiative and would like to explore a well-defined area where I can make a meaningful contribution.

    Given the scope of this feature, I believe it would be valuable to first investigate the existing PostgreSQL I/O implementations and identify opportunities to improve streaming support, transaction handling, or write-sink reliability.

    I’d be happy to start with a focused investigation, document the relevant design considerations, and propose a concrete implementation plan before proceeding with any code changes.

    Would there be a particular component or outstanding requirement where an initial contribution would be most useful?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions