diff --git a/.github/workflows/hamilton-main.yml b/.github/workflows/hamilton-main.yml index fb055c5bf..af6e57832 100644 --- a/.github/workflows/hamilton-main.yml +++ b/.github/workflows/hamilton-main.yml @@ -51,6 +51,21 @@ jobs: - '3.14' env: HAMILTON_TELEMETRY_ENABLED: false + services: + # disposable server for the SQL lineage tests; they create and drop their own schemas + postgres: + image: postgres:16-alpine + env: + POSTGRES_USER: hamilton + POSTGRES_PASSWORD: hamilton + POSTGRES_DB: sales + ports: + - 5432:5432 + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 steps: - name: Install Graphviz on Linux @@ -96,6 +111,14 @@ jobs: uv sync --group test --extra mcp uv run pytest tests/plugins/test_h_mcp.py + - name: Test openlineage + env: + HAMILTON_TEST_POSTGRES_URL: postgresql+psycopg2://hamilton:hamilton@localhost:5432/sales + run: | + uv sync --group test --extra openlineage + uv pip install psycopg2-binary + uv run pytest tests/io/test_utils.py tests/plugins/test_pandas_extensions.py tests/plugins/test_h_openlineage.py + - name: Test pandas run: | uv sync --group test diff --git a/docs/concepts/materialization.rst b/docs/concepts/materialization.rst index 66ee41b76..8c9535b0a 100644 --- a/docs/concepts/materialization.rst +++ b/docs/concepts/materialization.rst @@ -146,3 +146,178 @@ Here are simplified snippets for saving and loading an XGBoost model to/from JSO +----------------------------------------------+-----------------------------------------------+ To define your own DataSaver and DataLoader, the Apache Hamilton `XGBoost extension `_ provides a good example + + +.. _sql-metadata-and-lineage: + +SQL metadata and lineage +------------------------ + +The built-in SQL materializers (``@load_from.sql``, ``@save_to.sql``, ``from_.sql``, ``to.sql`` and the +``PandasSqlReader`` / ``PandasSqlWriter`` behind them) return ``sql_metadata`` describing what was read +or written. Since version ``1.1.0`` of that metadata, they also record *where*: the database the +connection points at. Lineage consumers such as the :doc:`OpenLineage adapter +<../reference/lifecycle-hooks/OpenLineageAdapter>` use this to name the physical tables a query +reads, without any custom loader or hand-maintained mapping. + +Take a graph that reads a query joining ``orders`` and ``customers`` from a sales database, +aggregates daily revenue in Python, and writes ``daily_revenue`` to a reporting database: + +.. code-block:: python + + @load_from.sql(query_or_table=value(REVENUE_QUERY), db_connection=source("sales_db")) + def order_lines(df: pd.DataFrame) -> pd.DataFrame: + return df + + @save_to.sql(table_name=value("daily_revenue"), schema=value("reporting"), + db_connection=source("warehouse_db"), output_name_="saved_revenue") + def daily_revenue(order_lines: pd.DataFrame) -> pd.DataFrame: + ... + +With metadata version ``1.0.0``, the loader's metadata was ``{"rows": 3, "query": "...", "table_name": None}``: no table, no +server, no database. A join reported no inputs at all, and the saver's ``daily_revenue`` could not +be told apart from a table of the same name elsewhere. With version ``1.1.0``, the same code, with the same +connections, yields: + +.. code-block:: python + + {"sql_metadata": { + "rows": 3, + "query": "WITH paid AS (...) SELECT ... FROM paid p JOIN customers c ON ...", + "table_name": None, + "schema": None, + "operation": "read", + "source": {"dialect": "postgresql", "host": "source.example", "port": 5432, + "database": "sales", "default_schema": "public"}, + "notes": "", + "timestamp": 1758470400.0, + "__version__": "1.1.0", + }} + +and the OpenLineage adapter, with ``sql_dataset_identity="datasource"`` (see +:ref:`sql-dataset-identity-change`), reports ``sales.public.orders`` and ``sales.public.customers`` under +``postgres://source.example:5432`` as inputs, and ``analytics.reporting.daily_revenue`` under the +warehouse's namespace as output. The same module produces the same identities whether it runs from +a script, a notebook or an orchestrator. + +Fields +~~~~~~ + +The ``sql_metadata`` entry holds the following keys: + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - Key + - Meaning + * - ``rows`` + - Rows read (``len`` of the DataFrame) or the row count the write returned; ``None`` when unknown. + * - ``query`` + - The statement executed, or ``None`` when a bare table name was read or written. As in 1.0.0, + a string containing the upper-case text ``SELECT`` anywhere is filed here, and anything else + under ``table_name``. Since 1.1.0 a read that starts (after comments) with ``select`` or + ``with`` in any case is also filed here; 1.0.0 recorded a lower-case ``select ...`` as a table + name. Writes and the two-argument form of :func:`~hamilton.io.utils.get_sql_metadata` keep the + 1.0.0 rule. Lineage consumers should use ``operation``: the string a write names is the table + written, whichever of the two keys holds it (``"USER_SELECTIONS"`` is filed under ``query``). + A written name that is not a plain identifier (``daily revenue``, ``SELECT results``) is + parsed with ``openlineage-sql``: a statement that names tables contributes the tables it + writes (or is left out, with a note, if it writes none), and a string naming no table is the + table name. Without ``openlineage-sql``, such a name is left out of lineage with a note. + * - ``table_name`` + - The bare table name read or written, or ``None`` for a statement. + * - ``schema`` + - The schema explicitly passed to the writer (``PandasSqlWriter(schema=...)``), else ``None``. + *New in 1.1.0.* + * - ``operation`` + - ``"read"`` or ``"write"``; ``None`` when the helper was called in its original two-argument + form and the direction is unknown. *New in 1.1.0.* + * - ``source`` + - The datasource, or ``None`` when it could not be identified. It holds ``dialect``, the SQLAlchemy + backend name (``postgresql`` or ``sqlite``); ``host``; ``port``; ``database``, which is the + absolute file path for SQLite; and ``default_schema``, the schema unqualified names resolve + against. ``default_schema`` is set only when SQLAlchemy already established it on the connection, + and is ``None`` otherwise. SQLite sources also hold ``attached``, a mapping of attached database + name to absolute file path, or ``None`` when the attached databases cannot be known (see + below). *New in 1.1.0.* + * - ``notes`` + - Why ``source`` is ``None``, for example ``"In-memory SQLite database has no stable identity"`` + or ``"Unsupported connection type for SQL metadata: MyConn"``; empty otherwise. *New in 1.1.0.* + * - ``timestamp`` + - When the metadata was produced (POSIX seconds). + * - ``__version__`` + - ``"1.1.0"``. Added keys bump the minor version; a change to an existing key's meaning bumps the + major version. + +Supported connections +~~~~~~~~~~~~~~~~~~~~~ + +The connection object determines what ``source`` can hold: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - Connection passed as ``db_connection`` + - What ``source`` holds + * - SQLAlchemy ``Engine`` or ``Connection`` (PostgreSQL, SQLite) + - dialect, host, port, database from the URL; ``default_schema`` as SQLAlchemy determined it on + connect (PostgreSQL ``current_schema()``). + * - SQLAlchemy URL string (``"postgresql+psycopg2://..."``, ``"sqlite:///path.db"``) + - dialect, host, port, database from the parsed URL; ``default_schema`` is ``None`` because pandas + discards the temporary engine it built. + * - Standard-library ``sqlite3.Connection`` on a file + - ``dialect="sqlite"``, ``database`` = absolute file path, and ``attached`` = the file of every + database attached to it, read with ``PRAGMA database_list`` on that same connection (no + transaction is started). Only this connection form can see attached databases; for the others + ``attached`` is ``None``. + * - In-memory SQLite (``:memory:``, ``sqlite://``, ``sqlite:///:memory:``) + - ``None`` with a note: two unrelated in-memory databases must not share an identity. A raw + ``sqlite3`` in-memory connection with files attached keeps ``database=""`` and ``attached``, so + tables in the attached files can still be named. + * - Anything else (other DBAPI connections, mocks, and similar objects) + - ``None`` with a note naming the type. Data loading and saving are unaffected. + +Inspection never opens a connection, runs a write, commits, rolls back, or changes session settings, +and never raises: a failure during inspection becomes a ``notes`` entry naming the exception type +only. ``source`` holds scalars, never the connection object or the URL string, so it serializes +with ``json.dumps`` and cannot carry a username, password or URL query parameter. The ``query`` +field is the SQL text you supplied, as before: keep secrets out of literals or strip them +downstream. + +Schema precedence and unknown cases +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A table referenced by a statement is qualified from, in order: the qualification written in the +SQL (``sales.public.orders``), the ``schema`` argument given to the writer, then ``source["default_schema"]``. +For SQLite, a schema names a database file rather than a namespace inside one: ``main`` (or no +schema) is the connection's file, and an attached database's tables are named in that file's +namespace, resolved through ``source["attached"]``. When the attached file cannot be known (a URL +or SQLAlchemy connection), or the schema is ``temp``, the table is left out rather than +attributed to the main file. +The default schema is what SQLAlchemy read from the server, not an assumption that PostgreSQL uses +``public``; when the connection's ``search_path`` spans several schemas, qualify table names in +the SQL to remove the ambiguity. When no schema can be determined, the table is left out of +lineage and the reason is reported. Only ``SELECT``-style statements pandas can execute are in +scope; SQL run inside ordinary Python functions is not observed. + +Call the helper yourself +~~~~~~~~~~~~~~~~~~~~~~~~ + +Custom ``@dataloader`` / ``@datasaver`` functions can produce the same metadata: + +.. code-block:: python + + from hamilton.io import utils + + @dataloader() + def orders(sales_db: Engine) -> tuple[pd.DataFrame, dict]: + query = "SELECT * FROM sales.public.orders" + df = pd.read_sql(query, sales_db) + return df, utils.get_sql_metadata(query, df, db_connection=sales_db, operation="read") + +Pass ``operation="write"`` from a saver so the table name is never mistaken for a statement. +The original two-argument call ``get_sql_metadata(query_or_table, results)`` keeps working and +keeps its keys, their values and row-count semantics; it simply reports ``source=None`` with a note and +``operation=None``, so consumers diagnose it as incomplete rather than guessing. diff --git a/docs/reference/lifecycle-hooks/OpenLineageAdapter.rst b/docs/reference/lifecycle-hooks/OpenLineageAdapter.rst index ce1e5f487..d0211ad09 100644 --- a/docs/reference/lifecycle-hooks/OpenLineageAdapter.rst +++ b/docs/reference/lifecycle-hooks/OpenLineageAdapter.rst @@ -2,7 +2,127 @@ plugins.h_openlineage.OpenLineageAdapter ======================================== +Install with ``pip install "apache-hamilton[openlineage]"``. The extra brings ``openlineage-python`` +(the client) and, on Linux and macOS, ``openlineage-sql`` (the parser used to find the tables a +query reads). ``openlineage-sql`` publishes no Windows wheel, so the extra skips it there and SQL +queries, and written table names that are not plain identifiers, are reported without table +datasets on Windows. The parser is used only with +``sql_dataset_identity="datasource"``; the default identity needs neither it nor anything else +beyond the client. + .. autoclass:: hamilton.plugins.h_openlineage.OpenLineageAdapter :special-members: __init__ :members: :inherited-members: + +SQL datasets +------------ + +SQL loaders and savers (``@load_from.sql``, ``@save_to.sql`` and the pandas SQL materializers) record +the datasource they used (see :ref:`sql-metadata-and-lineage`). How the adapter names their datasets +is set by ``sql_dataset_identity``: + +- ``"legacy"``, the default: datasets are named as in earlier Hamilton releases, under the adapter's + *job* namespace with the bare ``table_name``. As before, a query read containing ``SELECT`` + produces a dataset with no name and the query in the job's ``sql`` facet, and other strings are + used as the dataset name. Leaving the option unset warns once per adapter (a + ``FutureWarning``) because the default will change; pass ``"legacy"`` explicitly to keep these names + without the warning. +- ``"datasource"``: datasets are named after the datasource, following the `OpenLineage naming + conventions `_, and every physical table a query reads is + reported. A report written by one job and read by another then resolves to the same dataset, and + two tables with the same name in different databases stay distinct. + +.. code-block:: python + + adapter = OpenLineageAdapter(client, "my_namespace", "my_job", sql_dataset_identity="datasource") + +The rest of this section describes the ``"datasource"`` identity: + +.. list-table:: + :header-rows: 1 + :widths: 15 35 50 + + * - Dialect + - Namespace + - Name + * - PostgreSQL + - ``postgres://{host}:{port}`` (port defaults to 5432) + - ``{database}.{schema}.{table}``; unquoted identifiers are folded to lower case, as the server does + * - SQLite + - ``sqlite://{absolute file path}`` + - ``{table}``. A table in an attached database (``reporting.orders`` in the SQL, or the writer's + ``schema``) is named in the attached file's namespace, which is known only for a + standard-library ``sqlite3`` connection; otherwise it is left out. + +Aliases and common table expressions are not reported as tables. Each dataset carries a +``dataSource`` facet with the namespace; the ``schema`` facet from ``dataframe_metadata`` is +attached only when the node maps to a single table. The job keeps the ``sql`` facet with the +statement's text; a node that read or wrote a table by name has no statement to report. The +exception is a table read by a name that the metadata files as a query (one containing +``SELECT``, or whose first word is ``select`` or ``with``, such as ``SELECT_LOG`` or +``select-log``): it can't be told apart from a statement, so the name is reported as the job's +SQL and no input dataset is emitted for it. Give such tables plain names, or read them with a +query. + +Whatever cannot be fully identified is left out and logged as a warning from the +``hamilton.plugins.h_openlineage`` logger, never guessed. The following cases are left out: + +- an unknown datasource (in-memory SQLite, an unsupported connection object, legacy two-argument metadata) +- an unsupported dialect +- a table whose schema cannot be determined +- a statement ``openlineage-sql`` cannot parse +- a missing ``openlineage-sql`` install + +A failure inside the conversion is logged with its exception type and the run event is still +emitted without datasets. The node itself has already succeeded and is never failed by lineage. + +.. _sql-dataset-identity-change: + +Migrating to the datasource identity +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The default stays ``"legacy"`` for now and will become ``"datasource"`` in a future major release. +Switching changes every SQL dataset's namespace and name (for example from ``my_namespace`` + +``daily_revenue`` to ``postgres://warehouse.example:5432`` + ``analytics.reporting.daily_revenue``). +A lineage backend shows the new names as new datasets: history recorded under the old names does +not connect to them, and nothing is rewritten automatically. Some datasets also stop appearing: + +- loaders and savers whose metadata has no ``source`` (custom functions using the two-argument + helper, in-memory databases) +- any of the unidentifiable cases above, including every SQL query and every written table name + that is not a plain identifier on Windows, where ``openlineage-sql`` is not installed + +To migrate: + +1. Install ``openlineage-sql`` where you run Hamilton. ``apache-hamilton[openlineage]`` includes it + everywhere except Windows. Pin it in locked environments. +2. Run the pipeline once with ``sql_dataset_identity="datasource"`` against a test backend, or read + the events with ``FileTransport``, and note the new namespace and name of each dataset. Check the + ``hamilton.plugins.h_openlineage`` warnings for anything left out. +3. In your lineage backend, move what is keyed to the old names (ownership, tags, alerts, policies, + saved queries) to the new ones, or link old and new datasets where the backend supports it. +4. Pass ``sql_dataset_identity="datasource"`` in production. + +To stay on the current names, pass ``sql_dataset_identity="legacy"``. It silences the warning. + +Reuse the conversion +-------------------- + +:func:`~hamilton.plugins.h_openlineage.sql_datasets` is the boundary another integration (an +orchestrator provider, a custom adapter) can call on metadata Hamilton produced, with no +``OpenLineageClient``, ``Driver`` or database connection involved: + +.. code-block:: python + + from hamilton.plugins.h_openlineage import sql_datasets + + lineage = sql_datasets(node_result_metadata) # the dict a SQL loader/saver returned + lineage.inputs # list[openlineage.client.event_v2.Dataset] + lineage.outputs + lineage.notes # why anything was left out + +.. autofunction:: hamilton.plugins.h_openlineage.sql_datasets + +.. autoclass:: hamilton.plugins.h_openlineage.SqlDatasets + :members: diff --git a/examples/openlineage/.gitignore b/examples/openlineage/.gitignore new file mode 100644 index 000000000..7a02c159d --- /dev/null +++ b/examples/openlineage/.gitignore @@ -0,0 +1,4 @@ +# generated by run.py +pipeline.json +sales.db +warehouse.db diff --git a/examples/openlineage/README.md b/examples/openlineage/README.md index 973f49c34..1b96a97e3 100644 --- a/examples/openlineage/README.md +++ b/examples/openlineage/README.md @@ -17,30 +17,79 @@ specific language governing permissions and limitations under the License. --> -# OpenLineage Adapter +# OpenLineage adapter -This is an example of how to use the OpenLineage adapter that can be used to send metadata to an OpenLineage server. +This example emits [OpenLineage](https://openlineage.io/) events from an Apache Hamilton dataflow +that reads two SQL tables, aggregates them in Python and writes a report table. It runs locally +against two SQLite files and writes the events to `pipeline.json`; no lineage server, Airflow or +external database is needed. -## Motivation -OpenLineage is an open standard for data lineage. -With Apache Hamilton you can read and write data, and with OpenLineage you can track the lineage of that data. +## The problem it shows -## Steps -1. Build your project with Apache Hamilton. -2. Use one of the [materialization approaches](https://hamilton.apache.org/concepts/materialization/) to surface metadata about what is loaded and saved. -3. Use the OpenLineage adapter to send metadata to an OpenLineage server. +A team reads `orders` and `customers` from a sales database with a query, computes daily revenue, +and writes `daily_revenue` into a warehouse. They want to answer "which tables fed this report?" +and "is that later task reading the same report?" from their lineage backend. -## To run this example: +Before this feature, the SQL metadata Hamilton produced carried the query text and, for writes, a bare table name, but +not the database or server. A query read produced a dataset with no table name at all, and the +OpenLineage adapter filed SQL datasets under the *job* namespace, so `daily_revenue` written by one +job and read by another were two unrelated datasets. Getting real lineage meant writing a custom +loader that duplicated connection details. -1. Install the requirements: -```bash -pip install -r requirements.txt +The built-in `@load_from.sql` / `@save_to.sql` (and `from_.sql` / `to.sql`) materializers record +the datasource they used, and the adapter, created with `sql_dataset_identity="datasource"`, names +the physical tables from it (the default still uses the earlier names; see the adapter reference +below). `pipeline.py` is ordinary Hamilton code: + +```python +@load_from.sql(query_or_table=value(REVENUE_QUERY), db_connection=source("sales_db")) +def order_lines(df: pd.DataFrame) -> pd.DataFrame: + return df + +def daily_revenue(order_lines: pd.DataFrame) -> pd.DataFrame: + return order_lines.groupby(["order_date", "country"], as_index=False)["amount"].sum() + +@save_to.sql(table_name=value("daily_revenue"), db_connection=source("warehouse_db"), ...) +def revenue_report(daily_revenue: pd.DataFrame) -> pd.DataFrame: + return daily_revenue ``` -2. Run the example: -```bash -python run.py + +and the events it emits identify the tables by datasource, not by job: + +```text +RUNNING input: sqlite:///.../examples/openlineage/sales.db customers +RUNNING input: sqlite:///.../examples/openlineage/sales.db orders +RUNNING output: sqlite:///.../examples/openlineage/warehouse.db daily_revenue ``` -Or run the example in a notebook: + +The query uses a common table expression and aliases; only the two physical tables are reported. +Run against PostgreSQL, the same module reports `postgres://{host}:{port}` namespaces and +`{database}.{schema}.{table}` names. The same table in two databases stays distinct, and a later +read of `daily_revenue` gets the identity it was written under. The module does not change between +a script, a notebook and an orchestrator; only the connections you pass in do. + +## Run it + ```bash -jupyter notebook +pip install -r requirements.txt # apache-hamilton[openlineage] and sqlalchemy +python run.py ``` + +`run.py` seeds `sales.db`, runs the dataflow with the adapter using a `FileTransport`, prints the +saver's `sql_metadata` and the datasets found in `pipeline.json`. To send events to a running +OpenLineage server such as Marquez instead, swap the client as shown in `run.py`. The notebook +does the same steps interactively. + +## Where the metadata comes from + +Each SQL loader/saver returns `sql_metadata` with the `source` the connection points at (dialect, +host, port, database, default schema), alongside the query or table name. Custom `@dataloader` / +`@datasaver` functions can produce it too by calling +`hamilton.io.utils.get_sql_metadata(query, df, db_connection=conn)`. The +[materialization guide](https://hamilton.apache.org/concepts/materialization/#sql-metadata-and-lineage) +documents the fields, supported connections and what happens when a datasource cannot be +identified (in-memory SQLite, unknown connection objects): the data still loads, and the adapter +logs why the dataset was left out. The +[adapter reference](https://hamilton.apache.org/reference/lifecycle-hooks/OpenLineageAdapter/) +covers dataset naming, how to migrate from the default identity, and `sql_datasets()`, which other +integrations can call on the same metadata without emitting events. diff --git a/examples/openlineage/data.csv b/examples/openlineage/data.csv deleted file mode 100644 index 62fac067e..000000000 --- a/examples/openlineage/data.csv +++ /dev/null @@ -1,201 +0,0 @@ -id,name,email,address,phone_number,date_of_birth -1,Janet Garcia,ssmith@example.org,"4029 Jose Ferry Suite 300 -South Kristi, MH 40576",+1-288-900-0512x78889,1943-03-12 -2,Jay White,lkemp@example.com,"64555 Dennis Key Suite 552 -East Josephberg, OH 98796",+1-662-554-4225x070,1934-10-09 -3,Erik Brady,michael13@example.net,"00298 Patton Inlet Suite 564 -Lake Scott, PA 59977",8696803708,1990-02-07 -4,Melissa Riley,david38@example.com,"4113 Lisa Ville -New Geraldland, CT 59935",+1-853-644-8994x09790,2018-07-30 -5,Todd Holt,xsullivan@example.net,"48742 Williams Lakes Suite 089 -Juliefurt, TN 67462",(954)298-6387,1964-03-10 -6,Amy Ellis,ccarr@example.net,"1345 Jennifer Rest -Sanchezberg, AZ 29821",791-667-1022x974,1942-08-26 -7,Joseph Frey,marquezmonica@example.org,"32745 Sean Walk Suite 342 -Villarrealchester, SD 28498",(587)681-3404x1983,1910-02-24 -8,Amy Walker,lawrence99@example.net,"41956 Bowman Greens -Jenniferbury, DE 81997",001-889-687-1105x2891,1945-12-27 -9,Ruth Terry,wolferebecca@example.net,"1957 Heidi Square -Gonzalezchester, KY 15612",4148930353,2022-05-12 -10,Jaime Williams,adrianaberry@example.org,"133 Gary Way Apt. 170 -West Rebeccaborough, ME 85226",(216)264-8990x65483,2022-01-12 -11,Brandon Anderson,jose19@example.org,"USNV Torres -FPO AE 80322",228-744-9852x86443,1941-12-19 -12,Jennifer Clark,gpaul@example.net,"1191 Samantha Fall -Sanderstown, AK 01154",(671)318-2174x4573,1948-11-07 -13,Alexis Mcconnell,tammiesnyder@example.org,"3666 Houston Road -Griffinmouth, NC 43228",001-392-616-0473,1908-12-28 -14,Jessica Odom,louiswilliams@example.org,"8053 Frazier Drives Suite 902 -West Jennifer, WA 47476",555.271.7305,1953-06-16 -15,Terry Chung,hayesamanda@example.org,"211 Christopher Oval Apt. 367 -Davilafort, MP 50991",5792549149,1940-10-18 -16,Matthew Jordan,washingtonandrew@example.com,"6549 David Via -North Thomastown, IL 19729",001-623-386-0035,1942-11-28 -17,Heather Vasquez,martinsarah@example.net,"15889 Alvarez Island Suite 018 -Lake Jennifer, NC 53538",423.517.0330x03756,1915-05-18 -18,Linda Snyder,solomonjennifer@example.org,"998 Oconnor Lane -South Lindsay, AR 42644",778.825.2995x220,1936-01-02 -19,Sandy Brown,ryan67@example.com,"469 Kayla Glen -South Ryan, UT 43171",+1-408-632-9163,1957-12-31 -20,Katherine Mcintosh,bryangarrison@example.com,"32900 Smith Shoals Suite 744 -Gonzalezfurt, AR 35830",3362328948,1939-06-17 -21,Laura Griffin,cwells@example.com,"7773 Velazquez Estates Apt. 959 -East Robert, SC 42146",001-564-589-3712x54867,1910-12-15 -22,Justin Mcdonald,kayla84@example.org,"86158 Wilson Ports -North Amy, OR 65115",703.839.9325x17495,1910-04-15 -23,Kayla Campbell,nealadam@example.org,"320 Bell Parkways Suite 712 -South Micheal, AS 31101",(339)317-9523x821,1980-03-23 -24,Jason Stephens,allenjohnathan@example.net,"36869 Danielle Port Apt. 712 -Jamesberg, ND 90121",809-268-5402,1997-11-26 -25,Nicole Bailey,ewilliams@example.com,"224 Susan Harbor -South Kathleen, SD 57734",001-936-815-1918x4751,1963-02-17 -26,Valerie Marquez,mhunter@example.net,"00028 Gregory Tunnel Suite 056 -New Jonathan, HI 93036",+1-299-247-3040x0931,1959-03-16 -27,Rick Williams,kevinscott@example.com,"54907 Christopher Harbor -Port Angela, MO 14988",+1-381-255-0437x31048,1960-08-25 -28,Jacqueline Long,santiagojay@example.org,"51603 Alison Cliff -Smithmouth, CO 37407",559-407-8380,1941-12-22 -29,Barbara Barron,lawrence40@example.com,"3952 Daniels Field Apt. 764 -Hendersonville, KY 54656",280.648.8277x523,1983-10-18 -30,Amber Hines,michaelcole@example.com,"940 Jackson Locks Apt. 164 -Allenhaven, MT 74945",285-497-8687,1952-03-01 -31,Alice Collins,glarson@example.com,"9694 Katelyn Crest Apt. 843 -West Chloe, NE 79813",(631)466-9476,1929-11-03 -32,Erin Lawrence,melissaflores@example.org,"86666 Long Greens Suite 850 -Johnhaven, CA 20338",001-538-302-9217x20110,1939-07-10 -33,Barbara Marshall,pray@example.org,"395 Beasley Loop -South Patrick, MH 88741",648-963-5897x33902,1994-03-01 -34,Christopher Salazar,jacksonroy@example.org,"7212 Theodore Centers Suite 031 -Brittanymouth, OH 51760",(996)339-1568,1918-12-29 -35,Leroy Osborne,pshaw@example.com,"18655 Shelby Square Suite 445 -Mcguireburgh, IN 91418",741-219-4528x761,1986-09-04 -36,Shannon Medina,linda04@example.com,"PSC 5253, Box 2196 -APO AE 64800",317-804-4121,2000-11-17 -37,Alejandro Lane,brussell@example.net,"9483 Kevin Curve Apt. 195 -Torresmouth, AS 30284",961-234-6247x052,1932-06-01 -38,Clayton Hunt,samantha79@example.com,"Unit 5001 Box 7065 -DPO AP 24233",688.871.7400,2017-01-30 -39,Elizabeth Beltran,robleseric@example.org,"91222 Linda Glen -Lake Billy, VI 38356",(837)286-5718x43909,1964-02-16 -40,Troy Howell,troy83@example.net,"6377 Gutierrez Station Apt. 354 -Port Alexa, IL 96237",+1-268-302-9205x8468,1929-12-14 -41,Derek Lee,mariafox@example.org,"4555 Katherine Underpass -North Mariaview, DE 92959",4476109080,1910-07-31 -42,Lawrence Johnson,kingmadeline@example.net,"Unit 2651 Box 6754 -DPO AA 16161",477.573.2994x7443,1983-08-24 -43,Jennifer Flynn,shari74@example.org,"3707 Clements Branch -New Maria, ME 70362",+1-929-348-3938x91388,2012-06-25 -44,Laura Welch,alexandrasmith@example.org,"97547 Zachary Hills -Tylerfort, OR 91719",+1-445-917-3034x15476,1936-03-15 -45,Jessica Sanford,karljordan@example.com,"6916 Shields Drives Suite 684 -Herringmouth, NV 79066",001-405-607-8627x46350,1990-11-14 -46,Vickie Matthews,james08@example.com,"0480 Evans Springs Suite 140 -North Pamview, WI 65451",488-733-6798x42433,1986-12-23 -47,Kevin Garrett,melissabradley@example.net,"9220 Beth Track -Sierraburgh, GA 33193",284.958.6775x17624,1969-10-16 -48,Katherine Thomas MD,rmacdonald@example.com,"6608 Vincent Circles Apt. 103 -Port Barbara, RI 34326",738-760-4858,2023-03-19 -49,Gail Freeman,patriciabates@example.net,"12116 Smith Mountain Suite 774 -Hernandezview, HI 82529",668-232-2075x47811,1964-08-08 -50,Jessica Davenport,tprice@example.org,"6274 Brian Mall Suite 818 -Hensonmouth, WV 06237",001-763-869-4892,2024-05-10 -51,Robert Leonard,marcofrancis@example.org,"294 Winters River -Anthonytown, NV 73764",+1-567-240-0624,1964-04-19 -52,Jennifer Brown,schmittjudy@example.net,"3720 Nicholas Mission Suite 851 -Lake Melaniefort, OK 30214",001-835-977-3852x07954,1919-04-21 -53,Edward Thomas,steven29@example.com,"Unit 4594 Box 1296 -DPO AP 84060",483.458.4693x57935,1944-02-03 -54,Kelsey Flores,mwilliams@example.net,"5656 Brandy Locks Suite 533 -South Susanport, MS 80770",001-736-802-9647x155,1946-05-12 -55,Chelsea Cooper,martinezmichael@example.org,"285 Melanie Mountain Suite 891 -Jonesview, ID 57575",001-996-371-2505x7895,2004-11-21 -56,Troy Sanders,chungstephanie@example.com,"286 Johnson Island -Joneschester, VT 80357",001-844-712-0678x86682,1959-10-01 -57,Thomas Watkins,alexander56@example.net,"315 Michael Stravenue Apt. 385 -Josephfurt, ID 69125",+1-931-862-3454,1980-07-23 -58,Marcus Smith,richardvalerie@example.com,"683 Donovan Port Suite 654 -Port Amanda, ME 35634",(679)693-9921,1948-07-16 -59,Sheila Richards,mflynn@example.com,"055 Valencia Mews -North Juliehaven, PA 37227",522.566.5004x7148,1984-06-01 -60,Ellen Gay,douglasjohnson@example.net,"8631 Lee Dale Apt. 802 -East Nicholas, CT 49134",479-445-0566,1996-02-06 -61,Zachary Swanson,leonallen@example.net,"2323 Ryan Plain -Donaldsonfort, KS 89155",+1-414-404-8055,1976-08-03 -62,Carol Carlson,christine83@example.org,"712 Mills Mission -Lake Jonathanmouth, WA 11783",+1-234-543-7774x069,1928-02-18 -63,Melissa Nguyen,mooreshane@example.org,"2226 Tyler Parkway Suite 612 -Meadowschester, OH 66452",(700)949-3450,1957-10-05 -64,Michelle Higgins,susan60@example.com,"3357 Gomez Walks -Lake Davidside, WV 23491",785.814.9808,2010-11-15 -65,Rebecca Pearson,alvarezcarlos@example.net,"6196 Geoffrey Crescent -North Johnhaven, TN 79391",+1-936-830-9792x1671,1942-10-16 -66,John Johnson,pedwards@example.net,"9467 Jones Brooks Apt. 003 -West Scottside, UT 06859",547.392.4356x021,1937-12-29 -67,Derek Krause,brandonmeyer@example.org,"8860 Emma Street -Micheleton, GA 14221",(720)576-2210x26858,1916-04-19 -68,Kyle Scott,brian84@example.com,"07865 Amy Brooks -Lake Kevin, KY 05743",530.599.7946x37274,2001-09-23 -69,David Manning,taylormichael@example.net,"9821 Turner Grove -South Alice, AZ 53101",001-804-324-6062x352,2024-08-21 -70,Barbara Holmes,pittsbrian@example.net,"20984 Joseph Branch Apt. 569 -West Markview, WY 47710",+1-616-596-5874x2870,1980-09-18 -71,Ebony Coleman,lawsondanielle@example.com,"312 Abbott Mountains -Hodgesburgh, MN 33503",367-951-4487x38940,2021-07-03 -72,Cynthia Young,deleoncharles@example.com,"3514 Jacob Bridge Suite 736 -Lake Jamesside, KY 10050",535.662.5244x873,1927-10-04 -73,Mark Martin,hayscollin@example.com,"698 Allen Fork Suite 364 -Hamptonhaven, GA 79936",919-346-4360x76443,1970-06-22 -74,Benjamin Harrison,deborah28@example.net,"5119 Benjamin Forks Apt. 025 -Wardland, IN 16201",001-535-832-3376x3987,1928-11-11 -75,Joshua Wright,reynoldssteven@example.net,"USNV Bates -FPO AA 87237",001-684-218-9565x409,1963-11-26 -76,Heather Friedman,littlerussell@example.org,"Unit 1671 Box 9168 -DPO AA 76526",6183823196,1912-10-17 -77,Jessica Morris,richardconley@example.com,"USCGC Olson -FPO AE 22594",001-661-829-0045x01121,1931-03-22 -78,Aaron White,sherry90@example.net,"48964 Nicole Square -West Joyce, SD 81713",(316)492-2152x3535,2010-08-17 -79,Patricia Underwood,keith35@example.org,"97666 Gomez Ramp -New Kelly, AR 05858",+1-623-451-4892x6217,1990-02-11 -80,Larry Herrera,andersonkeith@example.org,"2230 Sonya Expressway Suite 786 -Staffordchester, OH 31076",621-845-2260x281,2011-10-09 -81,Jennifer Bennett,delacruzmichael@example.com,"43270 Ashley Forges -South Tracey, CA 94416",(578)875-7289,1970-03-22 -82,Laurie Randolph,nhaley@example.org,"563 Novak Forest -Stevenfurt, MH 36185",+1-508-376-7174x8696,1950-05-15 -83,William Foster,srodriguez@example.org,"742 Johnson Green -West David, MS 26881",001-771-477-6195x49391,1956-05-14 -84,Sue Allen,cabrerajustin@example.net,"8901 Jason Skyway Apt. 350 -Lopezville, TN 70257",527-506-7108,1919-04-25 -85,Charles Smith,vsmith@example.net,"488 Little Street Suite 355 -West Andrew, KY 21281",410-251-9899x5364,1951-06-02 -86,Alice Johnson,jameslee@example.com,"PSC 1676, Box 0530 -APO AE 74230",+1-308-863-1123,2017-06-05 -87,Henry Cohen,michael34@example.com,"6457 Jason Ville -Youngview, SD 86938",755.941.3036x2047,1961-03-18 -88,David Gamble,jeremy03@example.org,"USS Francis -FPO AA 16709",431-951-6042,1991-03-29 -89,Edward Clarke,johnsontaylor@example.org,"41069 Brenda Ports -North Cherylburgh, MI 13881",001-528-917-0283x27035,2002-12-15 -90,John Kent,jennifermonroe@example.com,"19777 Michael Common -Whitehaven, NC 87637",689.635.2818,1914-06-04 -91,Margaret Callahan,norriskelly@example.com,"2023 Ricky Plains Apt. 297 -Romeroborough, WY 28401",001-871-381-0577x6285,2015-07-14 -92,Amanda Morgan,chelseysharp@example.net,"952 Anthony Station Suite 855 -Tiffanyfurt, CA 41975",(345)977-8778x430,2013-10-29 -93,Holly Burton,tracyadams@example.com,"882 Anna Cove Apt. 778 -Ashleybury, OH 55885",+1-684-960-0990x5979,1947-07-13 -94,Amanda Robinson,kennedylisa@example.net,"19589 Green Islands Suite 864 -Brendahaven, CA 12378",313-224-0264x44718,1974-07-04 -95,Tracy Thompson,nortiz@example.net,"22638 Mary Passage -Meredithberg, MH 58223",(665)449-9719x724,1968-11-08 -96,Kevin Hernandez Jr.,dianawilkinson@example.net,"187 Kevin Trafficway Apt. 248 -Pereztown, NY 42119",+1-306-914-4658x822,2020-06-23 -97,Shannon Moreno,xhenry@example.org,"Unit 4537 Box 3464 -DPO AA 97501",474-812-4305x344,1990-09-13 -98,Juan Rodriguez,david06@example.com,"8872 Melissa Course Suite 593 -Port Peggy, PR 25450",+1-208-455-4138x7284,2012-06-20 -99,John Klein,amanda30@example.net,"PSC 9088, Box 5002 -APO AP 33766",001-818-902-1981,1993-12-06 -100,Gregory Lee,evanskimberly@example.org,"2246 Tucker Shores -Port Tracychester, MS 73839",(461)669-1109x517,1992-01-15 diff --git a/examples/openlineage/fake_data.py b/examples/openlineage/fake_data.py deleted file mode 100644 index 85f67e687..000000000 --- a/examples/openlineage/fake_data.py +++ /dev/null @@ -1,72 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -""" -Module to help generate fake data for testing purposes. -""" - -import pandas as pd -from faker import Faker - -# Initialize Faker -fake = Faker() - -# Define the columns -columns = ["id", "name", "email", "address", "phone_number", "date_of_birth"] - -# Generate fake data -data = { - "id": [i + 1 for i in range(100)], - "name": [fake.name() for _ in range(100)], - "email": [fake.email() for _ in range(100)], - "address": [fake.address() for _ in range(100)], - "phone_number": [fake.phone_number() for _ in range(100)], - "date_of_birth": [fake.date_of_birth() for _ in range(100)], -} - -# Create a DataFrame -df = pd.DataFrame(data, columns=columns) - -# Save to CSV -df.to_csv("data.csv", index=False) - -# now create some fake purchase data -columns = ["id", "user_id", "product_id", "price", "purchase_date"] - -# Generate fake data -fake_price_data = {i + 1: fake.random_int(min=1, max=1000) for i in range(20)} - -product_ids = [fake.random_int(min=1, max=20) for _ in range(1000)] - -data = { - "id": [i + 1 for i in range(1000)], - "user_id": [fake.random_int(min=1, max=100) for _ in range(1000)], - "product_id": product_ids, - "price": [fake_price_data[product_id] for product_id in product_ids], - "purchase_date": [fake.date_this_year() for _ in range(1000)], -} - -# Create a DataFrame -df = pd.DataFrame(data, columns=columns) - -# Save to SQLLite DB -import sqlite3 - -conn = sqlite3.connect("purchase_data.db") -df.to_sql("purchase_data", conn, index=False) -df.to_csv("purchase_data.csv", index=False) -conn.close() diff --git a/examples/openlineage/graph.png b/examples/openlineage/graph.png deleted file mode 100644 index 1d996e736..000000000 Binary files a/examples/openlineage/graph.png and /dev/null differ diff --git a/examples/openlineage/notebook.ipynb b/examples/openlineage/notebook.ipynb index 4d6bb7db6..64ca5410e 100644 --- a/examples/openlineage/notebook.ipynb +++ b/examples/openlineage/notebook.ipynb @@ -16,7 +16,7 @@ "outputs": [], "source": [ "# Execute this cell to install dependencies\n", - "%pip install apache-hamilton[visualization]" + "%pip install apache-hamilton[openlineage,visualization] sqlalchemy" ] }, { @@ -26,9 +26,7 @@ "source": [ "# OpenLineage example pipeline [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/openlineage/notebook.ipynb) [![GitHub badge](https://img.shields.io/badge/github-view_source-2b3137?logo=github)](https://github.com/apache/hamilton/blob/main/examples/openlineage/notebook.ipynb)\n", "\n", - "\n", - "This is a simple example of a pipeline that reads data from a file and a database, joins them, fits a model, and saves the model to a file and the joined data to a database. The pipeline does not import open lineage, and doesn't need to know about it. The salient point is that metadata is exposed by the data loading and data\n", - "saving functions. This is what is used to populated OpenLineage events." + "This pipeline reads a query joining `orders` and `customers` from a sales database, aggregates daily revenue in Python, and writes `daily_revenue` to a warehouse database. The pipeline does not import OpenLineage and does not need to know about it: the built-in SQL materializers record the datasource they used, and the OpenLineage adapter names the physical tables from that metadata.\n" ] }, { @@ -43,7 +41,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "id": "2ccc7699", "metadata": { "ExecuteTime": { @@ -51,311 +49,51 @@ "start_time": "2024-09-06T17:30:38.182382Z" } }, - "outputs": [ - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "cluster__legend\n", - "\n", - "Legend\n", - "\n", - "\n", - "\n", - "purchase_dataset.loader\n", - "\n", - "\n", - "purchase_dataset.loader\n", - "purchase_dataset()\n", - "\n", - "\n", - "\n", - "purchase_dataset\n", - "\n", - "purchase_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "purchase_dataset.loader->purchase_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "saved_to_db\n", - "\n", - "\n", - "saved_to_db\n", - "saved_to_db()\n", - "\n", - "\n", - "\n", - "user_dataset.loader\n", - "\n", - "\n", - "user_dataset.loader\n", - "user_dataset()\n", - "\n", - "\n", - "\n", - "user_dataset\n", - "\n", - "user_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "user_dataset.loader->user_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "transformed_user_dataset\n", - "\n", - "transformed_user_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "user_dataset->transformed_user_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "transformed_purchase_dataset\n", - "\n", - "transformed_purchase_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "purchase_dataset->transformed_purchase_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "joined_dataset\n", - "\n", - "joined_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "transformed_purchase_dataset->joined_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "saved_file\n", - "\n", - "\n", - "saved_file\n", - "saved_file()\n", - "\n", - "\n", - "\n", - "fit_model\n", - "\n", - "fit_model\n", - "ModelObject\n", - "\n", - "\n", - "\n", - "fit_model->saved_file\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "transformed_user_dataset->joined_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "joined_dataset->saved_to_db\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "joined_dataset->fit_model\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_purchase_dataset.loader_inputs\n", - "\n", - "db_client\n", - "object\n", - "\n", - "\n", - "\n", - "_purchase_dataset.loader_inputs->purchase_dataset.loader\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_saved_to_db_inputs\n", - "\n", - "db_client\n", - "object\n", - "joined_table_name\n", - "str\n", - "\n", - "\n", - "\n", - "_saved_to_db_inputs->saved_to_db\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_user_dataset.loader_inputs\n", - "\n", - "file_ds_path\n", - "str\n", - "\n", - "\n", - "\n", - "_user_dataset.loader_inputs->user_dataset.loader\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_saved_file_inputs\n", - "\n", - "file_path\n", - "str\n", - "\n", - "\n", - "\n", - "_saved_file_inputs->saved_file\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "input\n", - "\n", - "input\n", - "\n", - "\n", - "\n", - "function\n", - "\n", - "function\n", - "\n", - "\n", - "\n", - "materializer\n", - "\n", - "\n", - "materializer\n", - "\n", - "\n", - "\n" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "outputs": [], "source": [ "%%cell_to_module pipeline --display\n", "\n", - "import pickle\n", - "from typing import Tuple\n", - "\n", - "import pandas as pd\n", - "\n", - "from hamilton.function_modifiers import dataloader, datasaver\n", - "from hamilton.io import utils\n", - "\n", - "@dataloader()\n", - "def user_dataset(file_ds_path: str) -> Tuple[pd.DataFrame, dict]:\n", - " df = pd.read_csv(file_ds_path)\n", - " return df, utils.get_file_and_dataframe_metadata(file_ds_path, df)\n", - "\n", - "\n", - "@dataloader()\n", - "def purchase_dataset(db_client: object) -> Tuple[pd.DataFrame, dict]:\n", - " query = \"SELECT * FROM purchase_data\"\n", - " df = pd.read_sql(query, con=db_client)\n", - " metadata = {\n", - " \"sql_metadata\": {\"query\": query, \"table_name\": \"purchase_data\", \"database\": \"sqlite\"}\n", - " }\n", - " metadata.update(utils.get_dataframe_metadata(df))\n", - " return df, metadata\n", + "\"\"\"Revenue reporting with built-in SQL materializers.\n", "\n", + "- order_lines reads a query joining orders and customers from the sales database.\n", + "- daily_revenue aggregates in Python.\n", + "- revenue_report writes daily_revenue into the warehouse database.\n", "\n", - "def transformed_user_dataset(user_dataset: pd.DataFrame) -> pd.DataFrame:\n", - " return user_dataset\n", - "\n", - "\n", - "def transformed_purchase_dataset(purchase_dataset: pd.DataFrame) -> pd.DataFrame:\n", - " return purchase_dataset\n", - "\n", - "\n", - "def joined_dataset(\n", - " transformed_user_dataset: pd.DataFrame, transformed_purchase_dataset: pd.DataFrame\n", - ") -> pd.DataFrame:\n", - " joined = pd.merge(transformed_user_dataset,\n", - " transformed_purchase_dataset,\n", - " left_on=\"id\", right_on=\"user_id\")\n", - " del joined[\"id_x\"]\n", - " del joined[\"id_y\"]\n", - " return joined\n", + "No custom loader and no hand-written lineage metadata: the SQL materializers record the\n", + "datasource they used, and the OpenLineage adapter names the physical tables from it.\n", + "\"\"\"\n", "\n", + "import pandas as pd\n", "\n", - "class ModelObject:\n", - " def __init__(self):\n", - " pass\n", + "from hamilton.function_modifiers import load_from, save_to, source, value\n", "\n", - " def predict(self, data):\n", - " return data + 1\n", + "REVENUE_QUERY = \"\"\"\n", + "-- paid order lines with the customer's country\n", + "WITH paid AS (SELECT * FROM orders WHERE status = 'paid')\n", + "SELECT p.order_date, c.country, p.amount\n", + "FROM paid p\n", + "JOIN customers c ON p.customer_id = c.id\n", + "\"\"\"\n", "\n", "\n", - "def fit_model(joined_dataset: pd.DataFrame) -> ModelObject:\n", - " # model = ...\n", - " return ModelObject()\n", + "@load_from.sql(query_or_table=value(REVENUE_QUERY), db_connection=source(\"sales_db\"))\n", + "def order_lines(df: pd.DataFrame) -> pd.DataFrame:\n", + " return df\n", "\n", "\n", - "@datasaver()\n", - "def saved_file(fit_model: ModelObject, file_path: str) -> dict:\n", - " with open(file_path, \"wb\") as f:\n", - " pickle.dump(fit_model, f)\n", - " return utils.get_file_metadata(file_path)\n", + "def daily_revenue(order_lines: pd.DataFrame) -> pd.DataFrame:\n", + " return order_lines.groupby([\"order_date\", \"country\"], as_index=False)[\"amount\"].sum()\n", "\n", "\n", - "@datasaver()\n", - "def saved_to_db(joined_dataset: pd.DataFrame, db_client: object, joined_table_name: str) -> dict:\n", - " joined_dataset.to_sql(joined_table_name, con=db_client, index=False, if_exists=\"replace\")\n", - " # raise ValueError(\"Hi\")\n", - " metadata = utils.get_sql_metadata(joined_table_name, joined_dataset)\n", - " metadata.update(utils.get_dataframe_metadata(joined_dataset))\n", - " return metadata" + "@save_to.sql(\n", + " table_name=value(\"daily_revenue\"),\n", + " db_connection=source(\"warehouse_db\"),\n", + " if_exists=value(\"replace\"),\n", + " index=value(False),\n", + " output_name_=\"saved_revenue\",\n", + ")\n", + "def revenue_report(daily_revenue: pd.DataFrame) -> pd.DataFrame:\n", + " return daily_revenue\n" ] }, { @@ -363,12 +101,12 @@ "id": "50a7f746", "metadata": {}, "source": [ - "# Create OpenLineage client" + "# Seed a small sales database and create the OpenLineage client" ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "id": "f7dcecc8", "metadata": { "ExecuteTime": { @@ -378,18 +116,34 @@ }, "outputs": [], "source": [ + "import sqlite3\n", + "from pathlib import Path\n", + "\n", + "import pandas as pd\n", "from openlineage.client import OpenLineageClient\n", "from openlineage.client.transport.file import FileConfig, FileTransport\n", - "\n", - "# if you don't have a running OpenLineage server, you can use the FileTransport\n", - "file_config = FileConfig(\n", - " log_file_path=\"pipeline.json\",\n", - " append=True,\n", + "from sqlalchemy import create_engine\n", + "\n", + "sales_db = sqlite3.connect(\"sales.db\")\n", + "pd.DataFrame(\n", + " {\n", + " \"customer_id\": [1, 1, 2, 2],\n", + " \"order_date\": [\"2026-09-01\", \"2026-09-01\", \"2026-09-01\", \"2026-09-02\"],\n", + " \"amount\": [10.0, 5.0, 7.5, 3.0],\n", + " \"status\": [\"paid\", \"paid\", \"paid\", \"open\"],\n", + " }\n", + ").to_sql(\"orders\", sales_db, index=False, if_exists=\"replace\")\n", + "pd.DataFrame({\"id\": [1, 2], \"country\": [\"NL\", \"DE\"]}).to_sql(\n", + " \"customers\", sales_db, index=False, if_exists=\"replace\"\n", ")\n", + "warehouse_db = create_engine(f\"sqlite:///{Path('warehouse.db').resolve()}\")\n", "\n", - "# if you have a running OpenLineage server, e.g. marquez, uncomment this line.\n", - "# client = OpenLineageClient(url=\"http://localhost:9000\")\n", - "client = OpenLineageClient(transport=FileTransport(file_config))" + "# without a running OpenLineage server, the FileTransport writes events to a file\n", + "Path(\"pipeline.json\").unlink(missing_ok=True)\n", + "client = OpenLineageClient(\n", + " transport=FileTransport(FileConfig(log_file_path=\"pipeline.json\", append=True))\n", + ")\n", + "# with a running server, e.g. marquez: client = OpenLineageClient(url=\"http://localhost:5000\")" ] }, { @@ -397,12 +151,12 @@ "id": "4a49f34a", "metadata": {}, "source": [ - "# Create Hamilton DAG with OpenLineage Adapter" + "# Run the dataflow with the OpenLineage adapter" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "id": "6db87906", "metadata": { "ExecuteTime": { @@ -410,255 +164,23 @@ "start_time": "2024-09-06T17:32:30.852466Z" } }, - "outputs": [ - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "cluster__legend\n", - "\n", - "Legend\n", - "\n", - "\n", - "\n", - "purchase_dataset.loader\n", - "\n", - "\n", - "purchase_dataset.loader\n", - "purchase_dataset()\n", - "\n", - "\n", - "\n", - "purchase_dataset\n", - "\n", - "purchase_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "purchase_dataset.loader->purchase_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "saved_to_db\n", - "\n", - "\n", - "saved_to_db\n", - "saved_to_db()\n", - "\n", - "\n", - "\n", - "user_dataset.loader\n", - "\n", - "\n", - "user_dataset.loader\n", - "user_dataset()\n", - "\n", - "\n", - "\n", - "user_dataset\n", - "\n", - "user_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "user_dataset.loader->user_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "transformed_user_dataset\n", - "\n", - "transformed_user_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "user_dataset->transformed_user_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "transformed_purchase_dataset\n", - "\n", - "transformed_purchase_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "purchase_dataset->transformed_purchase_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "joined_dataset\n", - "\n", - "joined_dataset\n", - "DataFrame\n", - "\n", - "\n", - "\n", - "transformed_purchase_dataset->joined_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "saved_file\n", - "\n", - "\n", - "saved_file\n", - "saved_file()\n", - "\n", - "\n", - "\n", - "fit_model\n", - "\n", - "fit_model\n", - "ModelObject\n", - "\n", - "\n", - "\n", - "fit_model->saved_file\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "transformed_user_dataset->joined_dataset\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "joined_dataset->saved_to_db\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "joined_dataset->fit_model\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_purchase_dataset.loader_inputs\n", - "\n", - "db_client\n", - "object\n", - "\n", - "\n", - "\n", - "_purchase_dataset.loader_inputs->purchase_dataset.loader\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_saved_to_db_inputs\n", - "\n", - "db_client\n", - "object\n", - "joined_table_name\n", - "str\n", - "\n", - "\n", - "\n", - "_saved_to_db_inputs->saved_to_db\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_user_dataset.loader_inputs\n", - "\n", - "file_ds_path\n", - "str\n", - "\n", - "\n", - "\n", - "_user_dataset.loader_inputs->user_dataset.loader\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "_saved_file_inputs\n", - "\n", - "file_path\n", - "str\n", - "\n", - "\n", - "\n", - "_saved_file_inputs->saved_file\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "input\n", - "\n", - "input\n", - "\n", - "\n", - "\n", - "function\n", - "\n", - "function\n", - "\n", - "\n", - "\n", - "materializer\n", - "\n", - "\n", - "materializer\n", - "\n", - "\n", - "\n" - ], - "text/plain": [ - "" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ - "import pipeline\n", - "\n", "from hamilton import driver\n", "from hamilton.plugins import h_openlineage\n", "\n", - "ola = h_openlineage.OpenLineageAdapter(client, \"demo_namespace\", \"my_hamilton_job\")\n", - "\n", - "# create the DAG\n", - "dr = driver.Builder().with_modules(pipeline).with_adapters(ola).build()\n", - "# display the graph\n", - "dr.display_all_functions()" + "adapter = h_openlineage.OpenLineageAdapter(\n", + " client, \"demo_namespace\", \"revenue_job\", sql_dataset_identity=\"datasource\"\n", + ")\n", + "dr = driver.Builder().with_modules(pipeline).with_adapters(adapter).build()\n", + "result = dr.execute([\"saved_revenue\"], inputs={\"sales_db\": sales_db, \"warehouse_db\": warehouse_db})\n", + "sales_db.close()\n", + "result[\"saved_revenue\"][\"sql_metadata\"]" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "id": "ec295725", "metadata": { "ExecuteTime": { @@ -668,32 +190,15 @@ }, "outputs": [], "source": [ - "# create inputs to run the DAG\n", - "import sqlite3\n", - "\n", - "db_client = sqlite3.connect(\"purchase_data.db\")\n", - "\n", - "# execute & emit lineage\n", - "result = dr.execute(\n", - " [\"saved_file\", \"saved_to_db\"],\n", - " inputs={\n", - " \"db_client\": db_client,\n", - " \"file_ds_path\": \"data.csv\",\n", - " \"file_path\": \"model.pkl\",\n", - " \"joined_table_name\": \"joined_data\",\n", - " },\n", - ")\n", - "# close the DB\n", - "db_client.close()" + "# The datasets are named after the datasource (the SQLite files), not the job namespace\n", + "import json\n", + "\n", + "for line in Path(\"pipeline.json\").read_text().splitlines():\n", + " event = json.loads(line)\n", + " for kind in (\"inputs\", \"outputs\"):\n", + " for dataset in event.get(kind) or []:\n", + " print(f\"{event['eventType']} {kind[:-1]}: {dataset['namespace']} {dataset['name']}\")" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7b4ad693", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/examples/openlineage/pipeline.py b/examples/openlineage/pipeline.py index 03d1482ad..13c586dc3 100644 --- a/examples/openlineage/pipeline.py +++ b/examples/openlineage/pipeline.py @@ -15,121 +15,44 @@ # specific language governing permissions and limitations # under the License. -import pickle +"""Revenue reporting with built-in SQL materializers. -import pandas as pd - -from hamilton.function_modifiers import dataloader, datasaver -from hamilton.io import utils +- order_lines reads a query joining orders and customers from the sales database. +- daily_revenue aggregates in Python. +- revenue_report writes daily_revenue into the warehouse database. +No custom loader and no hand-written lineage metadata: the SQL materializers record the +datasource they used, and the OpenLineage adapter names the physical tables from it. """ -Narrative: - - this is a pipeline that will be used to show open lineage integration - - one function loads from file - - another loads from a database - - one save to a file - - another saves to a database - - there are transform functions in between -""" - - -@dataloader() -def user_dataset(file_ds_path: str) -> tuple[pd.DataFrame, dict]: - df = pd.read_csv(file_ds_path) - return df, utils.get_file_and_dataframe_metadata(file_ds_path, df) - - -@dataloader() -def purchase_dataset(db_client: object) -> tuple[pd.DataFrame, dict]: - query = "SELECT * FROM purchase_data" - df = pd.read_sql(query, con=db_client) - metadata = { - "sql_metadata": {"query": query, "table_name": "purchase_data", "database": "sqlite"} - } - metadata.update(utils.get_dataframe_metadata(df)) - return df, metadata - - -def transformed_user_dataset(user_dataset: pd.DataFrame) -> pd.DataFrame: - return user_dataset - - -def transformed_purchase_dataset(purchase_dataset: pd.DataFrame) -> pd.DataFrame: - return purchase_dataset - - -def joined_dataset( - transformed_user_dataset: pd.DataFrame, transformed_purchase_dataset: pd.DataFrame -) -> pd.DataFrame: - joined = pd.merge( - transformed_user_dataset, transformed_purchase_dataset, left_on="id", right_on="user_id" - ) - del joined["id_x"] - del joined["id_y"] - return joined +import pandas as pd -class ModelObject: - def __init__(self): - pass - - def predict(self, data): - return data + 1 - - -def fit_model(joined_dataset: pd.DataFrame) -> ModelObject: - # model = ... - return ModelObject() - - -@datasaver() -def saved_file(fit_model: ModelObject, file_path: str) -> dict: - with open(file_path, "wb") as f: - pickle.dump(fit_model, f) - return utils.get_file_metadata(file_path) - - -@datasaver() -def saved_to_db(joined_dataset: pd.DataFrame, db_client: object, joined_table_name: str) -> dict: - joined_dataset.to_sql(joined_table_name, con=db_client, index=False, if_exists="replace") - # raise ValueError("Hi") - metadata = utils.get_sql_metadata(joined_table_name, joined_dataset) - metadata.update(utils.get_dataframe_metadata(joined_dataset)) - return metadata - - -if __name__ == "__main__": - import sqlite3 - - from openlineage.client import OpenLineageClient - from openlineage.client.transport.file import FileConfig, FileTransport +from hamilton.function_modifiers import load_from, save_to, source, value - import __main__ as pipeline - from hamilton import driver - from hamilton.plugins import h_openlineage +REVENUE_QUERY = """ +-- paid order lines with the customer's country +WITH paid AS (SELECT * FROM orders WHERE status = 'paid') +SELECT p.order_date, c.country, p.amount +FROM paid p +JOIN customers c ON p.customer_id = c.id +""" - file_config = FileConfig( - log_file_path="pipeline.json", - append=True, - ) - # client = OpenLineageClient(url="http://localhost:9000") - client = OpenLineageClient(transport=FileTransport(file_config)) +@load_from.sql(query_or_table=value(REVENUE_QUERY), db_connection=source("sales_db")) +def order_lines(df: pd.DataFrame) -> pd.DataFrame: + return df - ola = h_openlineage.OpenLineageAdapter(client, "demo_namespace", "hamilton_job") - db_client = sqlite3.connect("purchase_data.db") +def daily_revenue(order_lines: pd.DataFrame) -> pd.DataFrame: + return order_lines.groupby(["order_date", "country"], as_index=False)["amount"].sum() - dr = driver.Builder().with_modules(pipeline).with_adapters(ola).build() - dr.display_all_functions("graph.png") - result = dr.execute( - ["saved_file", "saved_to_db"], - inputs={ - "db_client": db_client, - "file_ds_path": "data.csv", - "file_path": "model.pkl", - "joined_table_name": "joined_data", - }, - ) - db_client.close() +@save_to.sql( + table_name=value("daily_revenue"), + db_connection=source("warehouse_db"), + if_exists=value("replace"), + index=value(False), + output_name_="saved_revenue", +) +def revenue_report(daily_revenue: pd.DataFrame) -> pd.DataFrame: + return daily_revenue diff --git a/examples/openlineage/purchase_data.csv b/examples/openlineage/purchase_data.csv deleted file mode 100644 index 17f5757c2..000000000 --- a/examples/openlineage/purchase_data.csv +++ /dev/null @@ -1,1001 +0,0 @@ -id,user_id,product_id,price,purchase_date -1,4,7,554,2024-05-10 -2,11,15,878,2024-02-18 -3,7,8,796,2024-02-08 -4,18,16,119,2024-05-14 -5,68,13,269,2024-04-08 -6,72,13,269,2024-04-13 -7,42,20,805,2024-06-20 -8,33,9,832,2024-01-16 -9,45,13,269,2024-04-22 -10,28,3,927,2024-05-11 -11,22,6,609,2024-05-27 -12,48,2,228,2024-07-19 -13,75,18,926,2024-02-24 -14,4,18,926,2024-03-08 -15,43,11,174,2024-08-29 -16,62,16,119,2024-08-26 -17,12,7,554,2024-03-05 -18,26,2,228,2024-05-31 -19,60,16,119,2024-02-20 -20,49,17,525,2024-03-04 -21,12,13,269,2024-07-31 -22,29,20,805,2024-01-31 -23,12,1,982,2024-04-18 -24,51,8,796,2024-08-03 -25,73,17,525,2024-03-27 -26,83,9,832,2024-07-01 -27,21,3,927,2024-02-14 -28,77,13,269,2024-02-10 -29,15,19,636,2024-02-02 -30,96,15,878,2024-04-23 -31,33,18,926,2024-01-28 -32,77,17,525,2024-08-04 -33,18,18,926,2024-04-13 -34,15,3,927,2024-06-24 -35,17,20,805,2024-05-30 -36,25,2,228,2024-06-01 -37,91,7,554,2024-08-11 -38,11,11,174,2024-07-19 -39,100,15,878,2024-08-30 -40,20,4,938,2024-08-23 -41,53,11,174,2024-03-27 -42,83,14,994,2024-05-18 -43,71,11,174,2024-08-18 -44,8,20,805,2024-06-01 -45,16,7,554,2024-07-30 -46,40,14,994,2024-08-08 -47,59,6,609,2024-06-16 -48,26,16,119,2024-08-12 -49,96,16,119,2024-07-10 -50,43,9,832,2024-06-05 -51,100,19,636,2024-08-14 -52,98,19,636,2024-01-07 -53,30,18,926,2024-04-13 -54,21,4,938,2024-03-10 -55,21,4,938,2024-07-12 -56,38,7,554,2024-02-01 -57,64,7,554,2024-02-06 -58,9,10,98,2024-04-14 -59,24,16,119,2024-07-07 -60,54,18,926,2024-01-17 -61,34,10,98,2024-01-11 -62,86,18,926,2024-06-25 -63,89,7,554,2024-04-07 -64,93,3,927,2024-07-12 -65,57,10,98,2024-02-22 -66,60,17,525,2024-02-09 -67,28,19,636,2024-05-14 -68,98,11,174,2024-02-28 -69,15,6,609,2024-08-27 -70,60,10,98,2024-05-12 -71,74,2,228,2024-05-13 -72,3,15,878,2024-08-18 -73,13,12,394,2024-01-08 -74,70,1,982,2024-03-02 -75,22,11,174,2024-07-30 -76,52,5,118,2024-07-21 -77,74,14,994,2024-08-24 -78,36,3,927,2024-04-02 -79,59,1,982,2024-05-31 -80,23,11,174,2024-02-19 -81,12,4,938,2024-04-03 -82,2,20,805,2024-02-05 -83,71,19,636,2024-08-08 -84,9,2,228,2024-06-16 -85,5,11,174,2024-06-04 -86,18,5,118,2024-08-20 -87,34,2,228,2024-04-11 -88,79,9,832,2024-04-14 -89,74,19,636,2024-06-14 -90,5,8,796,2024-08-26 -91,35,19,636,2024-04-22 -92,74,16,119,2024-07-17 -93,10,7,554,2024-05-20 -94,9,2,228,2024-07-19 -95,1,17,525,2024-05-08 -96,33,13,269,2024-06-28 -97,97,9,832,2024-08-26 -98,100,2,228,2024-05-25 -99,96,16,119,2024-01-26 -100,49,2,228,2024-02-11 -101,29,14,994,2024-02-03 -102,70,14,994,2024-08-07 -103,65,18,926,2024-04-09 -104,37,2,228,2024-04-22 -105,75,13,269,2024-03-30 -106,44,18,926,2024-07-20 -107,60,17,525,2024-02-07 -108,42,3,927,2024-02-07 -109,11,9,832,2024-04-06 -110,57,6,609,2024-02-16 -111,97,10,98,2024-03-29 -112,45,18,926,2024-08-17 -113,43,3,927,2024-04-27 -114,96,6,609,2024-07-07 -115,87,19,636,2024-04-15 -116,63,17,525,2024-02-26 -117,19,19,636,2024-01-06 -118,71,18,926,2024-05-10 -119,11,12,394,2024-07-11 -120,46,8,796,2024-03-16 -121,42,14,994,2024-04-03 -122,91,11,174,2024-04-14 -123,51,5,118,2024-01-20 -124,98,11,174,2024-07-17 -125,65,20,805,2024-01-05 -126,48,1,982,2024-06-23 -127,47,16,119,2024-04-17 -128,16,3,927,2024-07-13 -129,53,5,118,2024-04-05 -130,16,4,938,2024-04-02 -131,60,7,554,2024-01-25 -132,74,17,525,2024-05-25 -133,24,3,927,2024-03-17 -134,70,16,119,2024-05-23 -135,41,20,805,2024-08-21 -136,19,2,228,2024-06-27 -137,55,3,927,2024-03-15 -138,7,8,796,2024-03-08 -139,42,15,878,2024-02-11 -140,80,10,98,2024-07-26 -141,95,20,805,2024-05-17 -142,25,12,394,2024-04-05 -143,83,16,119,2024-06-23 -144,8,10,98,2024-02-12 -145,58,14,994,2024-04-01 -146,42,20,805,2024-04-23 -147,96,3,927,2024-05-15 -148,35,5,118,2024-05-06 -149,19,5,118,2024-05-31 -150,64,1,982,2024-04-17 -151,13,15,878,2024-03-15 -152,37,14,994,2024-08-06 -153,73,3,927,2024-02-28 -154,11,14,994,2024-07-30 -155,25,9,832,2024-03-02 -156,79,4,938,2024-05-13 -157,78,6,609,2024-08-27 -158,7,1,982,2024-06-10 -159,11,20,805,2024-02-29 -160,17,17,525,2024-05-19 -161,94,10,98,2024-06-04 -162,51,7,554,2024-01-15 -163,98,20,805,2024-02-04 -164,17,1,982,2024-04-20 -165,24,17,525,2024-06-18 -166,38,10,98,2024-07-20 -167,72,9,832,2024-02-17 -168,29,20,805,2024-03-06 -169,36,10,98,2024-04-18 -170,12,20,805,2024-04-26 -171,7,10,98,2024-08-28 -172,29,17,525,2024-02-29 -173,59,19,636,2024-07-18 -174,80,7,554,2024-04-08 -175,4,16,119,2024-05-13 -176,79,3,927,2024-06-09 -177,27,4,938,2024-05-19 -178,63,6,609,2024-07-17 -179,75,20,805,2024-06-20 -180,22,1,982,2024-08-24 -181,73,12,394,2024-08-01 -182,31,14,994,2024-05-11 -183,71,7,554,2024-03-17 -184,10,11,174,2024-08-08 -185,34,5,118,2024-01-30 -186,81,8,796,2024-01-05 -187,61,6,609,2024-02-25 -188,9,11,174,2024-08-25 -189,17,4,938,2024-02-11 -190,67,19,636,2024-04-01 -191,59,13,269,2024-04-14 -192,51,13,269,2024-02-27 -193,69,7,554,2024-04-30 -194,13,5,118,2024-05-31 -195,29,16,119,2024-01-05 -196,68,8,796,2024-05-15 -197,73,17,525,2024-02-04 -198,66,6,609,2024-08-16 -199,48,12,394,2024-05-23 -200,63,10,98,2024-01-17 -201,79,10,98,2024-01-02 -202,52,4,938,2024-04-23 -203,77,16,119,2024-07-25 -204,43,8,796,2024-01-08 -205,66,9,832,2024-08-04 -206,85,5,118,2024-04-26 -207,16,7,554,2024-07-17 -208,93,5,118,2024-03-13 -209,8,11,174,2024-04-25 -210,55,11,174,2024-01-19 -211,39,6,609,2024-02-25 -212,33,14,994,2024-07-08 -213,69,9,832,2024-06-23 -214,84,3,927,2024-03-20 -215,1,18,926,2024-01-23 -216,41,1,982,2024-07-01 -217,26,2,228,2024-03-25 -218,57,16,119,2024-02-06 -219,84,7,554,2024-03-12 -220,71,19,636,2024-03-30 -221,59,2,228,2024-02-05 -222,45,19,636,2024-08-29 -223,61,19,636,2024-02-15 -224,81,15,878,2024-01-29 -225,34,11,174,2024-04-25 -226,28,13,269,2024-06-29 -227,65,5,118,2024-07-13 -228,97,17,525,2024-06-19 -229,77,2,228,2024-02-22 -230,42,13,269,2024-08-08 -231,13,4,938,2024-04-04 -232,90,14,994,2024-01-03 -233,49,14,994,2024-08-09 -234,75,3,927,2024-06-12 -235,17,1,982,2024-03-23 -236,73,13,269,2024-02-23 -237,1,16,119,2024-08-11 -238,87,6,609,2024-03-31 -239,51,3,927,2024-01-09 -240,12,17,525,2024-05-03 -241,76,16,119,2024-06-14 -242,93,3,927,2024-06-29 -243,73,2,228,2024-06-04 -244,82,10,98,2024-05-09 -245,36,6,609,2024-06-22 -246,36,19,636,2024-04-02 -247,6,11,174,2024-05-02 -248,31,1,982,2024-05-03 -249,75,15,878,2024-03-23 -250,39,16,119,2024-04-17 -251,36,5,118,2024-08-01 -252,84,13,269,2024-04-19 -253,31,9,832,2024-02-20 -254,61,1,982,2024-05-10 -255,34,3,927,2024-07-23 -256,57,13,269,2024-07-15 -257,38,17,525,2024-07-18 -258,59,12,394,2024-05-03 -259,48,13,269,2024-01-07 -260,76,10,98,2024-06-23 -261,36,7,554,2024-02-23 -262,17,8,796,2024-02-02 -263,71,17,525,2024-07-30 -264,93,2,228,2024-04-16 -265,84,20,805,2024-04-05 -266,23,7,554,2024-07-29 -267,53,15,878,2024-01-20 -268,43,5,118,2024-05-04 -269,26,18,926,2024-01-29 -270,8,5,118,2024-03-11 -271,21,16,119,2024-04-20 -272,16,10,98,2024-01-05 -273,81,11,174,2024-02-03 -274,44,5,118,2024-08-02 -275,44,19,636,2024-06-07 -276,48,16,119,2024-04-06 -277,91,3,927,2024-02-18 -278,16,4,938,2024-05-18 -279,63,16,119,2024-01-05 -280,63,10,98,2024-01-23 -281,83,12,394,2024-03-14 -282,88,14,994,2024-02-20 -283,73,8,796,2024-02-29 -284,2,12,394,2024-06-29 -285,9,6,609,2024-07-03 -286,30,15,878,2024-01-10 -287,30,10,98,2024-08-08 -288,79,14,994,2024-01-30 -289,74,17,525,2024-04-29 -290,63,3,927,2024-05-31 -291,42,9,832,2024-07-07 -292,20,19,636,2024-01-22 -293,51,7,554,2024-04-25 -294,13,10,98,2024-01-14 -295,37,12,394,2024-08-12 -296,78,13,269,2024-01-24 -297,72,1,982,2024-06-18 -298,64,4,938,2024-08-06 -299,88,8,796,2024-07-04 -300,45,17,525,2024-01-11 -301,42,8,796,2024-04-18 -302,75,6,609,2024-04-30 -303,9,3,927,2024-07-13 -304,58,9,832,2024-07-09 -305,82,8,796,2024-08-10 -306,26,2,228,2024-06-09 -307,91,17,525,2024-02-24 -308,54,17,525,2024-05-10 -309,32,16,119,2024-03-27 -310,61,11,174,2024-04-28 -311,4,17,525,2024-04-22 -312,16,10,98,2024-05-17 -313,96,14,994,2024-05-04 -314,91,1,982,2024-05-31 -315,70,13,269,2024-08-17 -316,45,2,228,2024-08-07 -317,63,8,796,2024-01-27 -318,29,19,636,2024-03-21 -319,5,6,609,2024-02-24 -320,25,10,98,2024-03-21 -321,35,4,938,2024-06-03 -322,30,15,878,2024-01-17 -323,4,19,636,2024-05-09 -324,45,15,878,2024-04-26 -325,89,17,525,2024-06-26 -326,70,11,174,2024-01-25 -327,55,3,927,2024-06-12 -328,81,20,805,2024-08-23 -329,23,1,982,2024-01-05 -330,23,9,832,2024-02-04 -331,64,5,118,2024-06-01 -332,35,2,228,2024-07-17 -333,61,15,878,2024-05-13 -334,8,13,269,2024-07-04 -335,64,15,878,2024-03-23 -336,74,12,394,2024-03-17 -337,71,4,938,2024-02-06 -338,36,19,636,2024-06-16 -339,11,20,805,2024-03-06 -340,66,13,269,2024-05-22 -341,57,2,228,2024-02-04 -342,60,12,394,2024-03-02 -343,23,2,228,2024-04-05 -344,51,17,525,2024-07-14 -345,18,1,982,2024-04-16 -346,8,10,98,2024-03-28 -347,40,1,982,2024-02-07 -348,76,11,174,2024-08-18 -349,62,3,927,2024-06-21 -350,88,13,269,2024-03-05 -351,85,13,269,2024-03-19 -352,27,14,994,2024-01-06 -353,52,17,525,2024-03-14 -354,79,9,832,2024-07-07 -355,82,5,118,2024-06-30 -356,15,4,938,2024-05-10 -357,33,7,554,2024-04-05 -358,84,4,938,2024-07-16 -359,94,6,609,2024-01-02 -360,99,13,269,2024-02-12 -361,54,10,98,2024-04-03 -362,22,8,796,2024-01-29 -363,93,18,926,2024-03-13 -364,26,19,636,2024-05-07 -365,6,1,982,2024-01-28 -366,43,7,554,2024-03-26 -367,12,13,269,2024-05-02 -368,63,8,796,2024-03-04 -369,81,15,878,2024-03-05 -370,21,4,938,2024-01-08 -371,69,8,796,2024-03-16 -372,71,11,174,2024-02-29 -373,26,13,269,2024-05-07 -374,64,1,982,2024-02-01 -375,68,18,926,2024-02-17 -376,83,11,174,2024-05-19 -377,30,2,228,2024-01-21 -378,62,7,554,2024-05-26 -379,27,7,554,2024-05-22 -380,53,15,878,2024-08-09 -381,98,8,796,2024-06-15 -382,53,15,878,2024-02-04 -383,67,19,636,2024-06-14 -384,44,4,938,2024-05-26 -385,28,3,927,2024-02-11 -386,19,17,525,2024-01-20 -387,69,9,832,2024-02-07 -388,44,5,118,2024-04-25 -389,88,17,525,2024-07-20 -390,56,12,394,2024-08-21 -391,3,20,805,2024-05-05 -392,32,6,609,2024-07-21 -393,75,15,878,2024-01-16 -394,92,7,554,2024-01-06 -395,52,1,982,2024-03-03 -396,18,14,994,2024-03-20 -397,84,20,805,2024-02-10 -398,4,18,926,2024-03-02 -399,5,9,832,2024-04-14 -400,58,3,927,2024-08-16 -401,94,12,394,2024-01-19 -402,46,11,174,2024-03-06 -403,40,20,805,2024-05-03 -404,99,14,994,2024-03-12 -405,16,18,926,2024-08-21 -406,88,15,878,2024-04-20 -407,95,20,805,2024-06-29 -408,85,16,119,2024-01-04 -409,95,8,796,2024-06-02 -410,32,15,878,2024-02-28 -411,16,16,119,2024-02-04 -412,17,6,609,2024-02-24 -413,86,11,174,2024-05-19 -414,3,19,636,2024-06-18 -415,58,16,119,2024-03-16 -416,67,12,394,2024-04-23 -417,63,11,174,2024-05-12 -418,53,3,927,2024-02-07 -419,8,4,938,2024-03-02 -420,58,11,174,2024-07-18 -421,31,16,119,2024-04-07 -422,74,20,805,2024-03-09 -423,4,19,636,2024-01-09 -424,41,5,118,2024-03-13 -425,72,18,926,2024-03-03 -426,51,14,994,2024-02-18 -427,42,4,938,2024-08-08 -428,30,15,878,2024-02-17 -429,9,4,938,2024-05-14 -430,17,1,982,2024-01-16 -431,18,10,98,2024-02-26 -432,48,17,525,2024-04-29 -433,8,6,609,2024-07-18 -434,49,20,805,2024-05-28 -435,66,11,174,2024-08-23 -436,83,7,554,2024-01-15 -437,37,12,394,2024-04-21 -438,6,15,878,2024-03-18 -439,1,17,525,2024-05-11 -440,58,3,927,2024-08-05 -441,33,16,119,2024-02-01 -442,17,8,796,2024-06-10 -443,14,4,938,2024-07-16 -444,59,3,927,2024-06-06 -445,66,5,118,2024-06-21 -446,57,10,98,2024-01-23 -447,92,19,636,2024-07-08 -448,15,19,636,2024-05-07 -449,29,17,525,2024-02-28 -450,34,13,269,2024-05-24 -451,4,15,878,2024-02-28 -452,82,20,805,2024-07-25 -453,60,7,554,2024-08-29 -454,5,18,926,2024-06-20 -455,55,5,118,2024-05-29 -456,91,9,832,2024-05-20 -457,67,20,805,2024-01-10 -458,57,17,525,2024-07-01 -459,60,5,118,2024-03-11 -460,9,9,832,2024-04-07 -461,42,12,394,2024-03-29 -462,5,17,525,2024-04-08 -463,16,12,394,2024-04-26 -464,34,7,554,2024-01-09 -465,89,1,982,2024-02-19 -466,95,7,554,2024-06-22 -467,41,2,228,2024-05-20 -468,28,13,269,2024-08-27 -469,76,1,982,2024-03-22 -470,97,17,525,2024-04-07 -471,100,4,938,2024-08-06 -472,61,11,174,2024-01-06 -473,38,5,118,2024-02-19 -474,45,14,994,2024-03-16 -475,2,20,805,2024-08-15 -476,28,15,878,2024-05-15 -477,11,9,832,2024-06-04 -478,96,6,609,2024-01-16 -479,81,8,796,2024-06-29 -480,81,5,118,2024-04-26 -481,16,11,174,2024-04-14 -482,95,5,118,2024-06-25 -483,48,19,636,2024-04-27 -484,50,17,525,2024-03-19 -485,99,10,98,2024-01-23 -486,80,6,609,2024-04-26 -487,33,20,805,2024-07-19 -488,75,15,878,2024-07-08 -489,12,4,938,2024-03-07 -490,100,4,938,2024-06-26 -491,64,17,525,2024-05-17 -492,83,8,796,2024-05-20 -493,64,5,118,2024-07-25 -494,3,2,228,2024-07-31 -495,89,6,609,2024-08-23 -496,7,5,118,2024-06-22 -497,80,14,994,2024-05-08 -498,90,2,228,2024-04-30 -499,66,9,832,2024-08-14 -500,59,20,805,2024-06-09 -501,2,12,394,2024-06-30 -502,2,17,525,2024-02-08 -503,60,8,796,2024-05-01 -504,87,13,269,2024-03-09 -505,90,4,938,2024-06-15 -506,36,20,805,2024-01-11 -507,65,15,878,2024-04-22 -508,57,15,878,2024-02-02 -509,19,19,636,2024-05-16 -510,31,20,805,2024-02-19 -511,12,14,994,2024-01-23 -512,11,9,832,2024-06-06 -513,46,14,994,2024-01-02 -514,11,13,269,2024-07-20 -515,92,4,938,2024-05-19 -516,1,20,805,2024-03-18 -517,64,16,119,2024-06-03 -518,91,9,832,2024-07-23 -519,81,5,118,2024-05-30 -520,90,2,228,2024-02-07 -521,82,2,228,2024-02-01 -522,12,5,118,2024-07-12 -523,23,15,878,2024-02-15 -524,4,5,118,2024-04-09 -525,94,11,174,2024-04-19 -526,79,10,98,2024-05-15 -527,69,15,878,2024-07-10 -528,13,11,174,2024-07-31 -529,6,14,994,2024-07-22 -530,25,18,926,2024-08-18 -531,15,16,119,2024-06-28 -532,45,5,118,2024-02-06 -533,78,16,119,2024-07-17 -534,26,15,878,2024-03-02 -535,45,18,926,2024-08-03 -536,72,14,994,2024-07-12 -537,60,10,98,2024-03-23 -538,7,6,609,2024-02-25 -539,67,8,796,2024-08-25 -540,28,4,938,2024-04-06 -541,78,4,938,2024-06-01 -542,38,1,982,2024-02-04 -543,69,16,119,2024-07-05 -544,21,17,525,2024-05-07 -545,32,6,609,2024-03-01 -546,58,2,228,2024-01-22 -547,44,8,796,2024-01-14 -548,49,1,982,2024-05-19 -549,19,9,832,2024-07-14 -550,4,14,994,2024-08-07 -551,75,2,228,2024-08-26 -552,23,17,525,2024-01-30 -553,98,17,525,2024-04-19 -554,98,20,805,2024-05-12 -555,90,7,554,2024-01-04 -556,46,1,982,2024-08-21 -557,33,19,636,2024-02-25 -558,87,20,805,2024-04-26 -559,90,19,636,2024-07-23 -560,74,4,938,2024-07-28 -561,22,5,118,2024-01-12 -562,68,2,228,2024-07-21 -563,68,13,269,2024-01-05 -564,81,11,174,2024-08-28 -565,93,12,394,2024-03-16 -566,52,5,118,2024-07-23 -567,49,8,796,2024-04-06 -568,87,19,636,2024-01-25 -569,33,4,938,2024-06-26 -570,7,2,228,2024-08-08 -571,45,15,878,2024-05-07 -572,76,1,982,2024-01-10 -573,16,11,174,2024-04-18 -574,74,7,554,2024-04-18 -575,32,6,609,2024-06-23 -576,39,15,878,2024-06-02 -577,98,18,926,2024-06-12 -578,98,12,394,2024-02-02 -579,85,15,878,2024-03-04 -580,63,19,636,2024-05-06 -581,7,18,926,2024-03-13 -582,79,5,118,2024-05-27 -583,27,18,926,2024-07-05 -584,98,3,927,2024-05-25 -585,27,13,269,2024-07-25 -586,16,4,938,2024-04-17 -587,69,4,938,2024-08-23 -588,54,14,994,2024-04-18 -589,30,8,796,2024-05-22 -590,50,10,98,2024-05-26 -591,39,15,878,2024-01-14 -592,16,7,554,2024-01-08 -593,46,17,525,2024-06-26 -594,8,15,878,2024-04-12 -595,24,7,554,2024-04-11 -596,40,9,832,2024-05-24 -597,47,15,878,2024-06-29 -598,67,8,796,2024-06-20 -599,57,15,878,2024-08-11 -600,18,18,926,2024-04-25 -601,30,5,118,2024-01-18 -602,14,16,119,2024-08-28 -603,10,14,994,2024-02-07 -604,37,16,119,2024-07-08 -605,95,19,636,2024-04-20 -606,26,8,796,2024-04-11 -607,87,3,927,2024-04-08 -608,81,10,98,2024-05-22 -609,57,7,554,2024-08-03 -610,15,10,98,2024-03-19 -611,25,13,269,2024-03-29 -612,6,13,269,2024-01-05 -613,64,11,174,2024-04-04 -614,48,2,228,2024-04-29 -615,1,7,554,2024-05-17 -616,25,13,269,2024-07-19 -617,40,17,525,2024-08-12 -618,64,14,994,2024-01-22 -619,56,17,525,2024-07-05 -620,20,10,98,2024-07-15 -621,68,14,994,2024-03-16 -622,74,7,554,2024-05-12 -623,94,1,982,2024-05-20 -624,63,1,982,2024-07-27 -625,37,2,228,2024-08-30 -626,88,17,525,2024-06-09 -627,23,9,832,2024-05-08 -628,92,17,525,2024-02-12 -629,50,17,525,2024-02-27 -630,65,8,796,2024-03-16 -631,45,4,938,2024-05-23 -632,14,4,938,2024-05-21 -633,41,6,609,2024-08-14 -634,52,12,394,2024-07-08 -635,3,3,927,2024-06-29 -636,4,12,394,2024-03-24 -637,42,7,554,2024-06-11 -638,6,12,394,2024-08-08 -639,45,7,554,2024-06-12 -640,12,4,938,2024-03-07 -641,56,19,636,2024-04-28 -642,26,4,938,2024-04-04 -643,7,8,796,2024-03-01 -644,38,14,994,2024-07-15 -645,47,10,98,2024-06-01 -646,18,17,525,2024-08-10 -647,85,2,228,2024-06-30 -648,53,3,927,2024-03-12 -649,17,3,927,2024-08-05 -650,82,18,926,2024-01-10 -651,18,19,636,2024-03-14 -652,30,13,269,2024-07-27 -653,26,3,927,2024-05-04 -654,66,16,119,2024-04-28 -655,20,17,525,2024-01-13 -656,67,14,994,2024-04-11 -657,16,10,98,2024-02-02 -658,62,20,805,2024-05-30 -659,9,8,796,2024-05-17 -660,51,12,394,2024-02-08 -661,3,12,394,2024-07-11 -662,48,5,118,2024-03-31 -663,59,14,994,2024-05-19 -664,82,4,938,2024-06-12 -665,15,10,98,2024-03-14 -666,93,9,832,2024-02-25 -667,57,5,118,2024-04-08 -668,56,10,98,2024-06-04 -669,68,19,636,2024-02-05 -670,90,6,609,2024-08-28 -671,37,11,174,2024-08-22 -672,79,10,98,2024-07-06 -673,12,12,394,2024-04-11 -674,20,18,926,2024-06-08 -675,5,20,805,2024-01-14 -676,24,6,609,2024-04-30 -677,25,16,119,2024-05-31 -678,11,9,832,2024-04-29 -679,41,3,927,2024-07-30 -680,38,2,228,2024-05-01 -681,35,20,805,2024-05-19 -682,69,4,938,2024-01-05 -683,85,4,938,2024-05-30 -684,44,18,926,2024-05-10 -685,34,12,394,2024-06-08 -686,50,20,805,2024-01-21 -687,97,4,938,2024-08-09 -688,20,14,994,2024-02-03 -689,45,11,174,2024-07-13 -690,80,14,994,2024-03-06 -691,60,5,118,2024-04-21 -692,47,13,269,2024-05-20 -693,38,14,994,2024-05-25 -694,55,3,927,2024-01-10 -695,10,12,394,2024-05-17 -696,11,8,796,2024-03-02 -697,40,11,174,2024-06-30 -698,73,2,228,2024-07-26 -699,4,20,805,2024-05-19 -700,51,19,636,2024-02-07 -701,89,9,832,2024-01-26 -702,68,10,98,2024-06-07 -703,91,20,805,2024-07-04 -704,31,2,228,2024-02-04 -705,49,17,525,2024-03-02 -706,59,6,609,2024-04-30 -707,84,15,878,2024-06-30 -708,95,6,609,2024-01-07 -709,96,19,636,2024-01-11 -710,98,18,926,2024-07-16 -711,60,5,118,2024-06-19 -712,84,7,554,2024-06-20 -713,88,19,636,2024-05-11 -714,19,19,636,2024-08-13 -715,37,4,938,2024-02-18 -716,75,10,98,2024-02-26 -717,30,12,394,2024-08-04 -718,30,18,926,2024-05-28 -719,87,3,927,2024-06-26 -720,26,16,119,2024-02-12 -721,62,3,927,2024-07-22 -722,95,9,832,2024-02-04 -723,42,5,118,2024-04-09 -724,36,5,118,2024-01-10 -725,38,16,119,2024-06-06 -726,85,19,636,2024-08-16 -727,77,3,927,2024-05-17 -728,63,5,118,2024-02-17 -729,36,17,525,2024-02-10 -730,67,1,982,2024-05-01 -731,36,12,394,2024-01-11 -732,40,16,119,2024-06-12 -733,42,20,805,2024-03-20 -734,74,11,174,2024-01-28 -735,47,1,982,2024-08-01 -736,28,15,878,2024-06-10 -737,40,19,636,2024-01-15 -738,68,12,394,2024-06-30 -739,25,8,796,2024-07-22 -740,98,7,554,2024-03-26 -741,62,7,554,2024-08-11 -742,11,20,805,2024-08-09 -743,30,13,269,2024-04-21 -744,25,11,174,2024-07-11 -745,82,4,938,2024-01-04 -746,91,4,938,2024-05-25 -747,92,18,926,2024-02-28 -748,51,17,525,2024-01-14 -749,100,20,805,2024-03-14 -750,56,20,805,2024-06-29 -751,41,4,938,2024-01-29 -752,97,5,118,2024-04-29 -753,89,4,938,2024-04-07 -754,21,2,228,2024-04-30 -755,63,12,394,2024-06-29 -756,26,9,832,2024-01-06 -757,7,12,394,2024-04-25 -758,27,18,926,2024-08-26 -759,55,11,174,2024-04-01 -760,76,17,525,2024-02-11 -761,49,14,994,2024-05-04 -762,23,5,118,2024-04-23 -763,15,3,927,2024-06-08 -764,54,13,269,2024-08-13 -765,62,14,994,2024-06-03 -766,98,8,796,2024-03-01 -767,16,10,98,2024-05-02 -768,72,1,982,2024-01-09 -769,8,13,269,2024-03-06 -770,32,2,228,2024-04-10 -771,90,19,636,2024-06-25 -772,56,6,609,2024-05-20 -773,54,14,994,2024-01-18 -774,15,18,926,2024-08-19 -775,38,15,878,2024-07-02 -776,15,8,796,2024-06-08 -777,36,4,938,2024-08-10 -778,61,15,878,2024-07-10 -779,86,19,636,2024-06-19 -780,7,5,118,2024-08-23 -781,33,10,98,2024-01-20 -782,84,17,525,2024-06-09 -783,66,4,938,2024-07-30 -784,82,5,118,2024-07-18 -785,29,9,832,2024-02-01 -786,2,8,796,2024-02-20 -787,23,20,805,2024-01-22 -788,9,2,228,2024-05-21 -789,54,8,796,2024-05-20 -790,61,17,525,2024-06-28 -791,27,2,228,2024-01-12 -792,85,4,938,2024-04-30 -793,33,5,118,2024-04-07 -794,82,12,394,2024-07-18 -795,68,15,878,2024-08-29 -796,55,15,878,2024-06-23 -797,21,3,927,2024-03-05 -798,86,2,228,2024-05-16 -799,15,12,394,2024-07-22 -800,19,19,636,2024-03-01 -801,67,11,174,2024-01-03 -802,57,10,98,2024-08-02 -803,55,3,927,2024-08-09 -804,27,11,174,2024-03-23 -805,47,12,394,2024-02-24 -806,1,8,796,2024-08-11 -807,8,16,119,2024-04-21 -808,93,3,927,2024-04-21 -809,21,8,796,2024-06-06 -810,18,4,938,2024-02-20 -811,88,14,994,2024-08-30 -812,94,5,118,2024-06-14 -813,45,11,174,2024-03-06 -814,71,6,609,2024-04-09 -815,36,17,525,2024-06-16 -816,91,14,994,2024-08-30 -817,18,3,927,2024-06-08 -818,72,20,805,2024-04-16 -819,10,15,878,2024-08-29 -820,26,18,926,2024-05-26 -821,69,3,927,2024-01-05 -822,43,1,982,2024-08-06 -823,98,6,609,2024-07-01 -824,40,7,554,2024-07-19 -825,8,17,525,2024-04-23 -826,69,3,927,2024-06-19 -827,51,11,174,2024-03-29 -828,49,19,636,2024-02-15 -829,20,6,609,2024-03-05 -830,13,19,636,2024-07-29 -831,3,1,982,2024-01-23 -832,92,18,926,2024-04-27 -833,15,3,927,2024-01-29 -834,61,15,878,2024-04-22 -835,46,15,878,2024-05-04 -836,50,10,98,2024-03-08 -837,62,18,926,2024-03-17 -838,69,18,926,2024-04-26 -839,91,10,98,2024-02-21 -840,32,10,98,2024-04-28 -841,66,4,938,2024-08-03 -842,41,13,269,2024-04-22 -843,97,10,98,2024-04-24 -844,95,7,554,2024-06-01 -845,78,8,796,2024-03-04 -846,90,1,982,2024-04-25 -847,89,1,982,2024-03-06 -848,93,18,926,2024-06-10 -849,64,8,796,2024-06-08 -850,38,2,228,2024-08-05 -851,72,15,878,2024-07-04 -852,61,2,228,2024-03-14 -853,61,11,174,2024-08-21 -854,97,4,938,2024-02-18 -855,29,1,982,2024-06-12 -856,27,2,228,2024-05-07 -857,14,3,927,2024-04-02 -858,100,16,119,2024-01-21 -859,100,6,609,2024-04-03 -860,66,4,938,2024-05-29 -861,33,9,832,2024-03-21 -862,14,13,269,2024-08-10 -863,40,8,796,2024-03-01 -864,59,16,119,2024-03-30 -865,46,19,636,2024-05-05 -866,48,4,938,2024-08-04 -867,39,10,98,2024-01-16 -868,6,2,228,2024-07-12 -869,1,13,269,2024-04-14 -870,35,3,927,2024-08-18 -871,60,15,878,2024-04-30 -872,23,3,927,2024-06-14 -873,2,4,938,2024-06-02 -874,96,20,805,2024-03-03 -875,37,20,805,2024-05-19 -876,7,1,982,2024-02-14 -877,95,4,938,2024-06-26 -878,91,16,119,2024-02-25 -879,7,17,525,2024-03-15 -880,94,5,118,2024-03-11 -881,2,19,636,2024-02-11 -882,100,18,926,2024-08-15 -883,16,10,98,2024-06-29 -884,70,17,525,2024-02-03 -885,81,16,119,2024-08-13 -886,32,11,174,2024-02-26 -887,48,3,927,2024-03-07 -888,15,17,525,2024-05-08 -889,48,1,982,2024-02-11 -890,73,19,636,2024-06-08 -891,54,7,554,2024-04-14 -892,66,8,796,2024-07-04 -893,45,11,174,2024-03-15 -894,34,12,394,2024-06-16 -895,51,9,832,2024-01-22 -896,2,17,525,2024-06-29 -897,5,13,269,2024-06-09 -898,40,3,927,2024-08-10 -899,47,10,98,2024-04-07 -900,61,14,994,2024-02-14 -901,98,17,525,2024-02-25 -902,77,6,609,2024-07-14 -903,94,3,927,2024-03-24 -904,86,7,554,2024-01-24 -905,85,5,118,2024-08-27 -906,39,4,938,2024-01-10 -907,92,3,927,2024-07-24 -908,75,12,394,2024-08-06 -909,97,6,609,2024-03-17 -910,24,9,832,2024-01-26 -911,29,5,118,2024-01-10 -912,82,16,119,2024-01-06 -913,73,4,938,2024-07-17 -914,60,6,609,2024-03-07 -915,64,13,269,2024-08-08 -916,35,5,118,2024-06-18 -917,15,3,927,2024-04-21 -918,37,3,927,2024-02-23 -919,5,4,938,2024-04-27 -920,93,10,98,2024-07-30 -921,87,13,269,2024-04-21 -922,85,11,174,2024-03-02 -923,34,6,609,2024-06-15 -924,64,18,926,2024-05-30 -925,15,16,119,2024-01-21 -926,22,12,394,2024-07-16 -927,38,5,118,2024-02-22 -928,37,17,525,2024-07-13 -929,54,10,98,2024-08-24 -930,85,18,926,2024-06-09 -931,6,14,994,2024-05-30 -932,26,17,525,2024-01-07 -933,53,12,394,2024-07-17 -934,93,6,609,2024-06-21 -935,8,16,119,2024-06-18 -936,22,10,98,2024-02-18 -937,76,13,269,2024-04-09 -938,18,12,394,2024-03-02 -939,46,20,805,2024-05-03 -940,67,3,927,2024-06-07 -941,2,13,269,2024-04-08 -942,75,1,982,2024-02-24 -943,41,11,174,2024-03-10 -944,90,12,394,2024-05-24 -945,86,10,98,2024-06-09 -946,38,9,832,2024-06-11 -947,25,19,636,2024-05-25 -948,91,1,982,2024-05-21 -949,93,9,832,2024-07-23 -950,57,2,228,2024-04-07 -951,4,20,805,2024-01-05 -952,59,6,609,2024-01-05 -953,93,7,554,2024-05-06 -954,79,14,994,2024-08-13 -955,46,19,636,2024-01-30 -956,81,10,98,2024-01-23 -957,6,2,228,2024-08-22 -958,49,5,118,2024-04-30 -959,7,18,926,2024-07-09 -960,34,14,994,2024-03-10 -961,73,20,805,2024-08-15 -962,35,7,554,2024-05-12 -963,21,11,174,2024-05-25 -964,14,16,119,2024-04-30 -965,86,2,228,2024-05-16 -966,45,14,994,2024-08-24 -967,15,3,927,2024-03-10 -968,72,3,927,2024-03-30 -969,71,3,927,2024-03-16 -970,95,13,269,2024-02-04 -971,88,6,609,2024-06-17 -972,58,10,98,2024-05-27 -973,8,12,394,2024-03-24 -974,26,20,805,2024-02-04 -975,17,3,927,2024-04-09 -976,10,2,228,2024-05-04 -977,15,17,525,2024-06-30 -978,75,3,927,2024-07-04 -979,11,9,832,2024-06-06 -980,35,12,394,2024-06-28 -981,89,19,636,2024-02-17 -982,20,3,927,2024-02-08 -983,81,4,938,2024-01-05 -984,43,7,554,2024-08-20 -985,98,15,878,2024-04-03 -986,55,20,805,2024-04-05 -987,83,2,228,2024-01-20 -988,20,9,832,2024-02-12 -989,25,1,982,2024-05-15 -990,12,13,269,2024-01-12 -991,18,7,554,2024-03-17 -992,65,10,98,2024-02-27 -993,13,5,118,2024-03-06 -994,76,9,832,2024-03-11 -995,15,16,119,2024-07-30 -996,34,13,269,2024-01-22 -997,95,7,554,2024-07-27 -998,72,2,228,2024-02-18 -999,71,8,796,2024-04-15 -1000,55,19,636,2024-07-10 diff --git a/examples/openlineage/purchase_data.db b/examples/openlineage/purchase_data.db deleted file mode 100644 index 2eecc0ce7..000000000 Binary files a/examples/openlineage/purchase_data.db and /dev/null differ diff --git a/examples/openlineage/requirements.txt b/examples/openlineage/requirements.txt index ae9f2e653..909e75669 100644 --- a/examples/openlineage/requirements.txt +++ b/examples/openlineage/requirements.txt @@ -1,2 +1,2 @@ -apache-hamilton -openlineage-python +apache-hamilton[openlineage] +sqlalchemy diff --git a/examples/openlineage/run.py b/examples/openlineage/run.py index deda4b147..92318d1a4 100644 --- a/examples/openlineage/run.py +++ b/examples/openlineage/run.py @@ -15,42 +15,71 @@ # specific language governing permissions and limitations # under the License. +import json import sqlite3 +from pathlib import Path +import pandas as pd import pipeline from openlineage.client import OpenLineageClient from openlineage.client.transport.file import FileConfig, FileTransport +from sqlalchemy import create_engine from hamilton import driver from hamilton.plugins import h_openlineage -# if you don't have a running OpenLineage server, you can use the FileTransport -file_config = FileConfig( - log_file_path="pipeline.json", - append=True, -) - -# if you have a running OpenLineage server, e.g. marquez, uncomment this line. -# client = OpenLineageClient(url="http://localhost:9000") -client = OpenLineageClient(transport=FileTransport(file_config)) - -ola = h_openlineage.OpenLineageAdapter(client, "demo_namespace", "my_hamilton_job") - -# create inputs to run the DAG -db_client = sqlite3.connect("purchase_data.db") -# create the DAG -dr = driver.Builder().with_modules(pipeline).with_adapters(ola).build() -# display the graph -dr.display_all_functions("graph.png") -# execute & emit lineage -result = dr.execute( - ["saved_file", "saved_to_db"], - inputs={ - "db_client": db_client, - "file_ds_path": "data.csv", - "file_path": "model.pkl", - "joined_table_name": "joined_data", - }, -) -# close the DB -db_client.close() +HERE = Path(__file__).parent + + +def seed_sales_db(path: Path) -> sqlite3.Connection: + """A small sales database: two tables the report reads.""" + conn = sqlite3.connect(path) + orders = pd.DataFrame( + { + "customer_id": [1, 1, 2, 2], + "order_date": ["2026-09-01", "2026-09-01", "2026-09-01", "2026-09-02"], + "amount": [10.0, 5.0, 7.5, 3.0], + "status": ["paid", "paid", "paid", "open"], + } + ) + customers = pd.DataFrame({"id": [1, 2], "country": ["NL", "DE"]}) + orders.to_sql("orders", conn, index=False, if_exists="replace") + customers.to_sql("customers", conn, index=False, if_exists="replace") + return conn + + +if __name__ == "__main__": + events_file = HERE / "pipeline.json" + events_file.unlink(missing_ok=True) + # if you don't have a running OpenLineage server, the FileTransport writes events to a file + client = OpenLineageClient( + transport=FileTransport(FileConfig(log_file_path=str(events_file), append=True)) + ) + # if you have a running OpenLineage server, e.g. marquez, use this instead: + # client = OpenLineageClient(url="http://localhost:5000") + adapter = h_openlineage.OpenLineageAdapter( + client, "demo_namespace", "revenue_job", sql_dataset_identity="datasource" + ) + + sales_db = seed_sales_db(HERE / "sales.db") # a raw sqlite3 connection ... + warehouse_db = create_engine( + f"sqlite:///{HERE / 'warehouse.db'}" + ) # ... and a SQLAlchemy engine + + dr = driver.Builder().with_modules(pipeline).with_adapters(adapter).build() + result = dr.execute( + ["saved_revenue"], inputs={"sales_db": sales_db, "warehouse_db": warehouse_db} + ) + sales_db.close() + + print("saver metadata:", json.dumps(result["saved_revenue"]["sql_metadata"], indent=2)) + print( + "report:", pd.read_sql("SELECT * FROM daily_revenue", warehouse_db).to_string(index=False) + ) + for line in events_file.read_text().splitlines(): + event = json.loads(line) + for kind in ("inputs", "outputs"): + for dataset in event.get(kind) or []: + print( + f"{event['eventType']} {kind[:-1]}: {dataset['namespace']} {dataset['name']}" + ) diff --git a/hamilton/io/utils.py b/hamilton/io/utils.py index 0953eb9bd..0eed6790c 100644 --- a/hamilton/io/utils.py +++ b/hamilton/io/utils.py @@ -16,11 +16,13 @@ # under the License. import os +import re +import sqlite3 import time from datetime import datetime from os import PathLike from pathlib import Path -from typing import Any +from typing import Any, Literal from urllib import parse import pandas as pd @@ -29,6 +31,40 @@ SQL_METADATA = "sql_metadata" FILE_METADATA = "file_metadata" +SqlOperation = Literal["read", "write"] + +_LEADING_WORD = re.compile(r"\w+") + + +def _leading_sql_word(text: str) -> str | None: + """The first word of ``text`` after any SQL comments, lower-cased; ``None`` if there is none.""" + rest = text.lstrip() + while rest.startswith(("--", "/*")): + line_comment = rest.startswith("--") + end = rest.find("\n" if line_comment else "*/", 2) + if end < 0: + return None + rest = rest[end + (1 if line_comment else 2) :].lstrip() + word = _LEADING_WORD.match(rest) + return word.group().lower() if word else None + + +def _starts_with_select(text: str) -> bool: + """Whether ``text`` begins, after any comments, with ``select`` or ``with`` in any case.""" + return _leading_sql_word(text) in ("select", "with") + + +def _is_query(query_or_table: str, operation: SqlOperation | None, legacy: bool) -> bool: + """Whether ``query_or_table`` goes under ``query`` (else ``table_name``) in the metadata. + + Metadata 1.0.0 counted anything containing ``SELECT`` as a query. That rule is kept for the + two-argument form and for writes (``operation`` says a write names a table); a read also counts + a statement that starts with a lower-case ``select``/``with``, which 1.0.0 misfiled as a table. + """ + if "SELECT" in query_or_table: + return True + return not legacy and operation != "write" and _starts_with_select(query_or_table) + def get_file_metadata(path: str | Path | PathLike) -> dict[str, Any]: """Gives metadata from loading a file. @@ -127,7 +163,94 @@ def get_file_and_dataframe_metadata(path: str, df: pd.DataFrame) -> dict[str, An return {**get_file_metadata(path), **get_dataframe_metadata(df)} -def get_sql_metadata(query_or_table: str, results: int | pd.DataFrame) -> dict[str, Any]: +def get_sql_source(db_connection: Any) -> tuple[dict[str, Any] | None, str]: + """Describes the database a connection points at, without credentials or live objects. + + Supported forms: SQLAlchemy URL strings, ``Engine`` and ``Connection`` objects, and + standard-library ``sqlite3`` connections. Inspection is read-only: it reads URL fields + that are already in memory and, for a raw sqlite3 connection, runs ``PRAGMA database_list`` + on that same connection (no transaction is started). + + :return: ``(source, notes)``. ``source`` is ``None`` when the connection cannot be + identified, and ``notes`` then says why. When present, ``source`` holds + ``dialect`` (SQLAlchemy backend name, e.g. ``postgresql``, ``sqlite``), ``host``, + ``port``, ``database`` (the absolute file path for SQLite) and ``default_schema`` + (the schema the connection resolves unqualified names against, when SQLAlchemy has + already fetched it; ``None`` otherwise). SQLite sources also hold ``attached``: a + mapping of attached database name to absolute file path for a raw sqlite3 + connection, or ``None`` when the attached databases cannot be known without opening + a connection. Usernames, passwords and URL query parameters are never included. + """ + kind = type(db_connection).__name__ + try: + source = _inspect_sql_source(db_connection) + except Exception as e: # inspection must never break the data operation + return None, f"Could not inspect {kind} connection: {type(e).__name__}" + if source is None: + return None, f"Unsupported connection type for SQL metadata: {kind}" + if source["dialect"] == "sqlite" and not source["database"] and not source.get("attached"): + return None, "In-memory SQLite database has no stable identity" + return source, "" + + +def _inspect_sql_source(db_connection: Any) -> dict[str, Any] | None: + if isinstance(db_connection, sqlite3.Connection): + # (seq, name, file) per database; file is "" for :memory: and temp + databases = db_connection.execute("PRAGMA database_list").fetchall() + path = next(file for _, name, file in databases if name == "main") + source = _sql_source("sqlite", None, None, path, None) + source["attached"] = { + name: os.path.abspath(file) + for _, name, file in databases + if name not in ("main", "temp") and file + } + return source + if isinstance(db_connection, str): + from sqlalchemy.engine import make_url + + url = make_url(db_connection) + default_schema = None + else: + engine = getattr(db_connection, "engine", db_connection) # a Connection knows its Engine + url = getattr(engine, "url", None) + if url is None: + return None + default_schema = getattr( + getattr(db_connection, "dialect", None), "default_schema_name", None + ) + return _sql_source(url.get_backend_name(), url.host, url.port, url.database, default_schema) + + +def _sql_source( + dialect: str, + host: str | None, + port: int | None, + database: str | None, + default_schema: str | None, +) -> dict[str, Any]: + if dialect == "sqlite": + if database == ":memory:": + database = "" # "sqlite:///:memory:" is in-memory too; leave it unidentified + elif database: + database = os.path.abspath(database) + return { + "dialect": dialect, + "host": host, + "port": port, + "database": database, + "default_schema": default_schema, + **({"attached": None} if dialect == "sqlite" else {}), + } + + +def get_sql_metadata( + query_or_table: str, + results: int | pd.DataFrame | None, + *, + db_connection: Any = None, + schema: str | None = None, + operation: SqlOperation | None = None, +) -> dict[str, Any]: """Gives metadata from reading a SQL table or writing to SQL db. Note: we reserve the right to change this schema. So if you're using this come @@ -138,21 +261,48 @@ def get_sql_metadata(query_or_table: str, results: int | pd.DataFrame) -> dict[s - the sql query (e.g., "SELECT foo FROM bar") - the table name (e.g., "bar") - the current time + - with ``db_connection``: the datasource (see :func:`get_sql_source`) and the + ``operation`` (``read`` for a query, ``write`` for a table target, ``None`` when + the legacy two-argument form is used), so lineage consumers can qualify tables. + + :param query_or_table: the SQL executed, or the bare table name read or written. + :param results: the resulting DataFrame, or the row count returned by the write (``None`` + when the write reports no count, e.g. with a custom pandas ``method``). + :param db_connection: the connection the operation ran on. Optional; when omitted the + datasource is unknown and ``notes`` says so. + :param schema: the schema the table was explicitly written to, when the writer had one. + :param operation: ``"write"`` when ``query_or_table`` is what was written to (a table name, + or a statement for custom savers), so lineage consumers know which key holds the target; + ``"read"`` when it was read with, + e.g., ``pandas.read_sql``. Inferred from ``results`` when omitted and ``db_connection`` given. """ - query = query_or_table if "SELECT" in query_or_table else None - table_name = query_or_table if "SELECT" not in query_or_table else None + is_query = _is_query( + query_or_table, operation, legacy=operation is None and db_connection is None + ) if isinstance(results, int): rows = results elif isinstance(results, pd.DataFrame): rows = len(results) else: rows = None + if db_connection is None: + source, notes = None, "No connection supplied; SQL datasource is unknown" + else: + source, notes = get_sql_source(db_connection) + if operation is None: + # writers return a row count (or None); readers return a DataFrame or an iterator + wrote = results is None or isinstance(results, int) + operation = "write" if wrote and not is_query else "read" return { SQL_METADATA: { "rows": rows, - "query": query, - "table_name": table_name, + "query": query_or_table if is_query else None, + "table_name": None if is_query else query_or_table, + "schema": schema, + "operation": operation, + "source": source, + "notes": notes, "timestamp": datetime.now().utcnow().timestamp(), - "__version__": "1.0.0", + "__version__": "1.1.0", } } diff --git a/hamilton/plugins/h_openlineage.py b/hamilton/plugins/h_openlineage.py index 526dbd344..d3065650b 100644 --- a/hamilton/plugins/h_openlineage.py +++ b/hamilton/plugins/h_openlineage.py @@ -15,19 +15,29 @@ # specific language governing permissions and limitations # under the License. +import dataclasses import json +import logging +import re import sys import traceback +import warnings from datetime import datetime, timezone -from typing import Any +from typing import Any, Literal, NamedTuple, get_args import attr from openlineage.client import OpenLineageClient, event_v2, facet_v2 from hamilton import graph as h_graph from hamilton import graph_types, node +from hamilton.io.utils import SQL_METADATA, SqlOperation from hamilton.lifecycle import base +logger = logging.getLogger(__name__) + +SqlDatasetIdentity = Literal["legacy", "datasource"] +_SQL_DATASET_IDENTITIES = get_args(SqlDatasetIdentity) + @attr.s class HamiltonFacet(facet_v2.RunFacet): @@ -71,8 +81,247 @@ def extract_schema_facet(metadata): return None -def create_input_dataset(namespace: str, metadata: dict, node_) -> list[event_v2.InputDataset]: - """Creates the open lineage input dataset.""" +@dataclasses.dataclass +class SqlDatasets: + """Datasets resolved from Hamilton SQL metadata. ``notes`` explains anything left out. + + ``query`` is the SQL statement the metadata recorded, or ``None`` when it named a table (a + written name may be filed under the metadata's ``query``) or cannot be told apart from one. A + table read by a name the metadata files as a query (one containing ``SELECT``, or whose first + word after comments is ``select`` or ``with`` in any case) cannot be told apart from a + statement, so its name is reported here and no input dataset is emitted for it. + """ + + inputs: list[event_v2.Dataset] + outputs: list[event_v2.Dataset] + notes: list[str] + query: str | None = None + + +class _Dialect(NamedTuple): + parser: str # openlineage-sql dialect name + scheme: str # OpenLineage namespace scheme + folds_unquoted: bool # server lower-cases unquoted identifiers + default_port: int | None = None + + +# an unquoted, optionally qualified identifier +_PLAIN_NAME = re.compile(r"[\w$]+(\.[\w$]+)*") + +# keyed by SQLAlchemy backend name +_DIALECTS = { + "postgresql": _Dialect("postgres", "postgres", True, 5432), + "sqlite": _Dialect("sqlite", "sqlite", False), +} + + +def sql_datasets( + sql_metadata: dict[str, Any], operation: SqlOperation | None = None +) -> SqlDatasets: + """Converts Hamilton SQL metadata into OpenLineage datasets. Emits nothing, opens nothing. + + This is the reusable boundary for other integrations (e.g. an orchestrator provider): + feed it the ``sql_metadata`` produced by :func:`hamilton.io.utils.get_sql_metadata` (either + the whole metadata dict or its ``sql_metadata`` entry) and get datasets named per the + `OpenLineage naming conventions `_: + + - PostgreSQL: namespace ``postgres://{host}:{port}``, name ``{database}.{schema}.{table}``. + Unquoted identifiers are folded to lower case, as the server does. + - SQLite: namespace ``sqlite://{absolute file path}``, name ``{table}``. A table in an + attached database (``{schema}.{table}`` in the SQL, or the writer's ``schema``) is named + in the attached file's namespace; it is left out when that file cannot be known. + + Queries are parsed with ``openlineage-sql``; every physical table read appears in ``inputs`` + and every table written in ``outputs`` (aliases and common table expressions are not tables). + A bare table name is placed by ``operation`` (``read``/``write``), taken from the metadata + unless given here. What a write names is its table when it is a plain identifier; otherwise it + is parsed, and is the table written unless it parses as a statement that names tables (then the + tables it writes are used, if any). Without a parse it is left out. A read string that is not a + plain name is parsed as SQL rather than used as one. Schema precedence: explicit in the SQL, + then the writer's ``schema``, then the connection's default schema. Anything that cannot be + fully identified is left out and explained in ``notes`` rather than guessed — an unknown datasource, an unsupported + dialect, a missing schema, a parse error or a missing ``openlineage-sql`` install. + """ + md = sql_metadata.get(SQL_METADATA, sql_metadata) + result = SqlDatasets([], [], []) + source = md.get("source") + if not source: + result.notes.append(md.get("notes") or "SQL datasource is unknown; no dataset emitted") + return result + if source["dialect"] not in _DIALECTS: + result.notes.append(f"No OpenLineage dataset naming for dialect {source['dialect']!r}") + return result + dialect = _DIALECTS[source["dialect"]] + query, table_name = md.get("query"), md.get("table_name") + operation = operation or md.get("operation") + # what a write names may be under either key (1.0.0 files anything containing "SELECT" under + # ``query``): a plain name is the table; anything else is parsed, and is the table written + # unless it parses as a statement that names tables + write_target = (query or table_name) if operation == "write" else None + if write_target and _PLAIN_NAME.fullmatch(write_target): + query, table_name = None, write_target + elif write_target: + query, table_name = write_target, None + elif operation == "read" and table_name and not _PLAIN_NAME.fullmatch(table_name): + # a read string that is not a plain name is SQL, whichever key it was filed under + query, table_name = table_name, None + + def to_datasets( + tables: list[tuple[str | None, str | None, str, bool]], + ) -> list[event_v2.Dataset]: + datasets = [] + for table in tables: + dataset, note = _dataset(source, md.get("schema"), *table) + if dataset: + datasets.append(dataset) + else: + result.notes.append(note) + return datasets + + if query: + try: + import openlineage_sql + except ImportError: + openlineage_sql = None + note = "openlineage-sql is not installed; install apache-hamilton[openlineage] to resolve tables from SQL" + parsed = None + if openlineage_sql: + try: + parsed = openlineage_sql.parse([query], dialect=dialect.parser) + except Exception as e: + note = f"SQL parsing failed: {type(e).__name__}" + # a statement, not a table name: filed as a read query, or parsed as naming tables + recorded_as_query = not write_target and md.get("query") == query + if recorded_as_query or (parsed and (parsed.out_tables or parsed.in_tables)): + result.query = query + if write_target and parsed and not parsed.out_tables: + if parsed.in_tables: + result.notes.append("Write metadata holds a statement that writes no table") + return result + # names no table at all, so it is not a statement: it is the name of the table written + result.outputs = to_datasets([(None, None, write_target, True)]) + return result + if parsed is None: # without a parse a name and a statement can't be told apart + result.notes.append(note) + return result + result.notes.extend(f"SQL parsing error: {err.message}" for err in parsed.errors) + result.inputs = to_datasets(_parsed_tables(parsed.in_tables, dialect.folds_unquoted)) + result.outputs = to_datasets(_parsed_tables(parsed.out_tables, dialect.folds_unquoted)) + elif table_name: + # a name pandas passed straight to the database: taken as written, no folding + datasets = to_datasets([(None, None, table_name, True)]) + if operation == "read": + result.inputs = datasets + elif operation == "write": + result.outputs = datasets + else: + result.notes.append( + f"Operation for table {table_name!r} is unknown (legacy metadata); no dataset emitted" + ) + else: + result.notes.append("SQL metadata has neither a query nor a table name") + return result + + +def _parsed_tables( + tables: list[Any], folds: bool +) -> list[tuple[str | None, str | None, str, bool]]: + """(database, schema, name, name_quoted) per parsed table, folding unquoted parts if asked.""" + + def part(value, style, key): + quoted = getattr(style, key, None) is not None + return value.lower() if value and folds and not quoted else value + + return [ + ( + part(t.database, t.quote_style, "database"), + part(t.schema, t.quote_style, "schema"), + t.name, + getattr(t.quote_style, "name", None) is not None, + ) + for t in tables + ] + + +def _dataset( + source: dict[str, Any], + explicit_schema: str | None, + database: str | None, + schema: str | None, + name: str, + quoted: bool, +) -> tuple[event_v2.Dataset | None, str]: + """Names one table; returns (dataset, "") or (None, why not).""" + dialect = _DIALECTS[source["dialect"]] + if dialect.folds_unquoted and not quoted: + name = name.lower() + schema = schema or explicit_schema # written in the SQL, else the writer's schema= + if source["dialect"] == "sqlite": + # a SQLite schema names a database file; the dataset lives in that file's namespace + path, note = _sqlite_file(source, schema, name) + if not path: + return None, note + namespace = f"{dialect.scheme}://{path}" + full_name = name + else: + if not source["host"]: + return None, f"{source['dialect']} host is unknown; cannot name {name!r}" + namespace = f"{dialect.scheme}://{source['host']}:{source['port'] or dialect.default_port}" + database = database or source["database"] + schema = schema or source["default_schema"] + if not database or not schema: + return None, ( + f"Table {name!r} cannot be fully qualified (database={database!r}, schema={schema!r}); " + "qualify it in the SQL, pass schema= to the writer, or use a SQLAlchemy Engine/Connection" + ) + full_name = f"{database}.{schema}.{name}" + facets = { + "dataSource": facet_v2.datasource_dataset.DatasourceDatasetFacet( + name=namespace, uri=namespace + ) + } + return event_v2.Dataset(namespace, full_name, facets=facets), "" + + +def _sqlite_file(source: dict[str, Any], schema: str | None, name: str) -> tuple[str | None, str]: + """The file a SQLite table lives in; returns (path, "") or (None, why not).""" + if not schema or schema.lower() == "main": + if not source["database"]: + return None, f"Table {name!r} is in an in-memory database, which has no stable identity" + return source["database"], "" + if schema.lower() == "temp": + return None, f"Table {name!r} is in the temporary database, which has no stable identity" + attached = source.get("attached") + if attached is None: + return None, ( + f"Cannot tell which file SQLite database {schema!r} is for table {name!r}; " + "pass the sqlite3 connection the database was attached on" + ) + # SQLite database names are case-insensitive + path = {k.lower(): v for k, v in attached.items()}.get(schema.lower()) + if not path: + return None, f"SQLite database {schema!r} for table {name!r} is not attached to a file" + return path, "" + + +def _legacy_sql_fields(sql_metadata: dict[str, Any]) -> tuple[str | None, str | None]: + """``(query, table_name)`` as metadata 1.0.0 filed them, which the legacy identity names from. + + 1.1.0 files a read starting with a lower-case ``select``/``with`` under ``query``, where 1.0.0 + filed it (and so named the legacy dataset) under ``table_name``; only that case is moved back. + Metadata without a newer ``__version__`` (e.g. built by hand) is used exactly as given. + """ + query, table_name = sql_metadata.get("query"), sql_metadata.get("table_name") + refiled = sql_metadata.get("__version__", "1.0.0") != "1.0.0" + if refiled and query and table_name is None and "SELECT" not in query: + return None, query + return query, table_name + + +def create_input_dataset( + namespace: str, metadata: dict, node_ +) -> tuple[list[event_v2.InputDataset], facet_v2.sql_job.SQLJobFacet | None]: + """Creates the open lineage input dataset, in the job namespace (legacy SQL identity).""" datasource_facet = None storage_facet = None sql_facet = None @@ -91,9 +340,9 @@ def create_input_dataset(namespace: str, metadata: dict, node_) -> list[event_v2 uri=path, ) elif "sql_metadata" in metadata: - name = metadata["sql_metadata"]["table_name"] + query, name = _legacy_sql_fields(metadata["sql_metadata"]) sql_facet = facet_v2.sql_job.SQLJobFacet( - query=metadata["sql_metadata"]["query"], + query=query, ) else: name = "--UNKNOWN--" @@ -111,8 +360,22 @@ def create_input_dataset(namespace: str, metadata: dict, node_) -> list[event_v2 return inputs, sql_facet +def _sql_lineage( + metadata: dict[str, Any], operation: SqlOperation, node_: node.Node +) -> tuple[list[event_v2.Dataset], str | None]: + """Datasets and SQL statement for a SQL loader/saver node, named after the datasource.""" + lineage = sql_datasets(metadata, operation) + for note in lineage.notes: + logger.warning("OpenLineage SQL lineage for node %s is incomplete: %s", node_.name, note) + datasets = lineage.inputs if operation == "read" else lineage.outputs + schema_facet = extract_schema_facet(metadata) if len(datasets) == 1 else None + if schema_facet: + datasets[0].facets["schema"] = schema_facet + return datasets, lineage.query + + def create_output_dataset(namespace: str, metadata: dict, node_) -> list[event_v2.OutputDataset]: - """Creates the open lineage output dataset.""" + """Creates the open lineage output dataset, in the job namespace (legacy SQL identity).""" datasource_facet = None storage_facet = None if "file_metadata" in metadata: @@ -127,7 +390,7 @@ def create_output_dataset(namespace: str, metadata: dict, node_) -> list[event_v uri=name, ) elif "sql_metadata" in metadata: - name = metadata["sql_metadata"]["table_name"] + _, name = _legacy_sql_fields(metadata["sql_metadata"]) else: name = "--UNKNOWN--" schema_facet = extract_schema_facet(metadata) @@ -186,19 +449,59 @@ class OpenLineageAdapter( on the driver object. """ - def __init__(self, client: OpenLineageClient, namespace: str, job_name: str): + def __init__( + self, + client: OpenLineageClient, + namespace: str, + job_name: str, + sql_dataset_identity: SqlDatasetIdentity | None = None, + ): """Constructor. You pass in the OLClient. :param self: :param client: :param namespace: :param job_name: + :param sql_dataset_identity: how SQL loader/saver datasets are identified. + ``"legacy"`` (the current default) names them ``namespace`` + bare table name, as + earlier releases did. ``"datasource"`` names them after the database they live in, + per the OpenLineage naming conventions (see :func:`sql_datasets`), so lineage + connects across jobs. Leaving it unset uses ``"legacy"`` and warns once, since the + default will change to ``"datasource"`` in a future major release. :return: """ + if sql_dataset_identity not in (None, *_SQL_DATASET_IDENTITIES): + raise ValueError( + f"sql_dataset_identity must be one of {_SQL_DATASET_IDENTITIES}, " + f"got {sql_dataset_identity!r}" + ) # self.transport = transport self.client = client self.namespace = namespace self.job_name = job_name + self.sql_dataset_identity: SqlDatasetIdentity = sql_dataset_identity or "legacy" + self._warn_sql_identity_default = sql_dataset_identity is None + + def _warn_legacy_sql_identity(self): + """Warns once per adapter that SQL datasets use the default legacy identity.""" + if not self._warn_sql_identity_default: + return + self._warn_sql_identity_default = False + message = ( + "OpenLineageAdapter is naming SQL datasets with the legacy identity (job namespace + " + "bare table name). A future major release will default to " + "sql_dataset_identity='datasource', which names datasets after their database " + "(e.g. postgres://host:5432 + db.schema.table) so lineage connects across jobs. " + "Existing lineage history will not connect to the new names automatically. To keep " + "the current names, pass sql_dataset_identity='legacy'. To migrate, pass " + "sql_dataset_identity='datasource' and move anything keyed to the old names " + "(ownership, tags, alerts, policies) in your lineage backend. See " + "https://hamilton.apache.org/reference/lifecycle-hooks/OpenLineageAdapter/" + ) + try: + warnings.warn(message, FutureWarning, stacklevel=2) + except Exception: # warnings filtered to errors must not fail the node or drop lineage + logger.warning(message) def pre_graph_execute( self, @@ -320,10 +623,41 @@ def post_node_execute( inputs = [] outputs = [] sql_facet = None - if saved_or_loaded == "loaded": - inputs, sql_facet = create_input_dataset(self.namespace, metadata, node_) - else: - outputs = create_output_dataset(self.namespace, metadata, node_) + try: + if "sql_metadata" in metadata and self.sql_dataset_identity == "datasource": + # SQL datasets are named after their datasource, not the job namespace + operation: SqlOperation = "read" if saved_or_loaded == "loaded" else "write" + sql_datasets_, query = _sql_lineage(metadata, operation, node_) + datasets = [ + (event_v2.InputDataset if operation == "read" else event_v2.OutputDataset)( + d.namespace, d.name, facets=d.facets + ) + for d in sql_datasets_ + ] + if operation == "read": + inputs = datasets + else: + outputs = datasets + if query: + sql_facet = facet_v2.sql_job.SQLJobFacet(query=query) + else: + if "sql_metadata" in metadata: + self._warn_legacy_sql_identity() + if saved_or_loaded == "loaded": + inputs, sql_facet = create_input_dataset(self.namespace, metadata, node_) + else: + outputs = create_output_dataset(self.namespace, metadata, node_) + except Exception as e: # lineage must never fail a node that already succeeded + # only the exception type at WARNING: messages may quote connection details + logger.warning( + "OpenLineage dataset conversion failed for node %s in run %s (%s); emitting the " + "run event without datasets", + node_.name, + run_id, + type(e).__name__, + ) + logger.debug("OpenLineage dataset conversion failure", exc_info=True) + inputs, outputs, sql_facet = [], [], None run = event_v2.Run( runId=run_id, diff --git a/hamilton/plugins/pandas_extensions.py b/hamilton/plugins/pandas_extensions.py index 72c569c14..32c9923cb 100644 --- a/hamilton/plugins/pandas_extensions.py +++ b/hamilton/plugins/pandas_extensions.py @@ -42,7 +42,6 @@ except ImportError: FILESYSTEM_TYPE = type | None -from sqlite3 import Connection from pandas._typing import NpDtype from pandas.core.dtypes.dtypes import ExtensionDtype @@ -708,7 +707,7 @@ class PandasSqlReader(DataLoader): """ query_or_table: str - db_connection: str | Connection # can pass in SQLAlchemy engine/connection + db_connection: Any # SQLAlchemy URL string, Engine or Connection, or a DBAPI connection # kwarg chunksize: int | None = None coerce_float: bool = True @@ -745,7 +744,9 @@ def _get_loading_kwargs(self) -> dict[str, Any]: def load_data(self, type_: type) -> tuple[DATAFRAME_TYPE, dict[str, Any]]: df = pd.read_sql(self.query_or_table, self.db_connection, **self._get_loading_kwargs()) - sql_metadata = utils.get_sql_metadata(self.query_or_table, df) + sql_metadata = utils.get_sql_metadata( + self.query_or_table, df, db_connection=self.db_connection, operation="read" + ) df_metadata = utils.get_dataframe_metadata(df) return df, {**sql_metadata, **df_metadata} @@ -801,7 +802,13 @@ def _get_saving_kwargs(self) -> dict[str, Any]: def save_data(self, data: DATAFRAME_TYPE) -> dict[str, Any]: results = data.to_sql(self.table_name, self.db_connection, **self._get_saving_kwargs()) - sql_metadata = utils.get_sql_metadata(self.table_name, results) + sql_metadata = utils.get_sql_metadata( + self.table_name, + results, + db_connection=self.db_connection, + schema=self.schema, + operation="write", + ) df_metadata = utils.get_dataframe_metadata(data) return {**sql_metadata, **df_metadata} diff --git a/pyproject.toml b/pyproject.toml index e635e13c5..b29c53987 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -68,7 +68,11 @@ experiments = [ "uvicorn", ] lsp = ["apache-hamilton-lsp"] -openlineage = ["openlineage-python"] +openlineage = [ + "openlineage-python", + # SQL table resolution; no Windows wheel is published, the adapter degrades without it + "openlineage-sql; sys_platform != 'win32'", +] pandera = ["pandera"] performance = [ # Internal performance boosters go here diff --git a/tests/conftest.py b/tests/conftest.py index 1d52afb5f..7fb13b8c9 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -15,7 +15,11 @@ # specific language governing permissions and limitations # under the License. +import os import sys +import uuid + +import pytest # Skip tests that require packages not yet available on Python 3.14 collect_ignore = [] @@ -48,3 +52,26 @@ def pytest_sessionfinish(session, exitstatus): if exitstatus == 5: # pytest.ExitCode.NO_TESTS_COLLECTED if sys.version_info >= (3, 14): session.exitstatus = 0 + + +@pytest.fixture +def postgres_schema(): + """A throwaway schema on the PostgreSQL server named by ``HAMILTON_TEST_POSTGRES_URL``. + + Yields ``(engine, schema_name)``; skips when the variable is unset. Only that schema is + created and it is dropped afterwards, so any server the variable points at stays clean. + """ + url = os.environ.get("HAMILTON_TEST_POSTGRES_URL") + if not url: + pytest.skip("HAMILTON_TEST_POSTGRES_URL not set") + sqlalchemy = pytest.importorskip("sqlalchemy") + engine = sqlalchemy.create_engine(url) + schema = f"lineage_{uuid.uuid4().hex[:8]}" + with engine.begin() as conn: + conn.execute(sqlalchemy.text(f"CREATE SCHEMA {schema}")) + try: + yield engine, schema + finally: + with engine.begin() as conn: + conn.execute(sqlalchemy.text(f"DROP SCHEMA {schema} CASCADE")) + engine.dispose() diff --git a/tests/io/test_utils.py b/tests/io/test_utils.py index 45288c814..e7a086a4a 100644 --- a/tests/io/test_utils.py +++ b/tests/io/test_utils.py @@ -15,11 +15,18 @@ # specific language governing permissions and limitations # under the License. +import json import pathlib +import sqlite3 +import subprocess +import sys +import time import pandas as pd +import pytest +from sqlalchemy import create_engine -from hamilton.io.utils import SQL_METADATA, get_file_metadata, get_sql_metadata +from hamilton.io.utils import SQL_METADATA, get_file_metadata, get_sql_metadata, get_sql_source def test_get_sql_metadata(): @@ -35,6 +42,163 @@ def test_get_sql_metadata(): assert metadata2["query"] == query assert metadata2["rows"] == 5 assert metadata3["rows"] is None + # legacy two-argument form: datasource is unknown, never guessed + for metadata in (metadata1, metadata2, metadata3): + assert metadata["source"] is None + assert metadata["operation"] is None + assert metadata["notes"] + + +@pytest.mark.parametrize( + "query_or_table", + ["foo", "daily revenue", "SELECT foo FROM bar", "select foo from bar", "SELECTIONS"], +) +def test_get_sql_metadata_legacy_form_classifies_as_version_1_0_0(query_or_table): + """The two-argument form keeps the 1.0.0 rule: a query is anything containing SELECT.""" + expected = { + "foo": (None, "foo"), + "daily revenue": (None, "daily revenue"), + "SELECT foo FROM bar": ("SELECT foo FROM bar", None), + "select foo from bar": (None, "select foo from bar"), + "SELECTIONS": ("SELECTIONS", None), + }[query_or_table] + metadata = get_sql_metadata(query_or_table, 1)[SQL_METADATA] + assert (metadata["query"], metadata["table_name"]) == expected + + +@pytest.mark.parametrize( + ("query_or_table", "operation", "query", "table_name"), + [ + # writes keep the 1.0.0 filing; operation="write" is what marks the name as a table + ("daily revenue", "write", None, "daily revenue"), + ("SELECTED_ROWS", "write", "SELECTED_ROWS", None), + ("select_rows", "write", None, "select_rows"), + # reads: select/with statements are recognised in any case, other strings keep 1.0.0 filing + ("select foo from bar", "read", "select foo from bar", None), + ( + "-- note\n with x as (select 1) select * from x", + "read", + "-- note\n with x as (select 1) select * from x", + None, + ), + ("/* hint */ /* two */ select 1", "read", "/* hint */ /* two */ select 1", None), + ("daily revenue", "read", None, "daily revenue"), + ("selections", "read", None, "selections"), + ("(select a from t)", "read", None, "(select a from t)"), + ("/* unterminated select", "read", None, "/* unterminated select"), + ("/*/ select 1 */ orders", "read", None, "/*/ select 1 */ orders"), # one comment + ], +) +def test_get_sql_metadata_explicit_operation( + tmp_path, query_or_table, operation, query, table_name +): + conn = sqlite3.connect(tmp_path / "a.db") + metadata = get_sql_metadata(query_or_table, 1, db_connection=conn, operation=operation) + metadata = metadata[SQL_METADATA] + assert (metadata["query"], metadata["table_name"]) == (query, table_name) + assert metadata["operation"] == operation + + +def test_get_sql_metadata_sqlite_connections(tmp_path: pathlib.Path): + path = tmp_path / "sales.db" + conn = sqlite3.connect(path) + metadata = get_sql_metadata("orders", 3, db_connection=conn)[SQL_METADATA] + assert metadata["operation"] == "write" + assert metadata["source"] == { + "dialect": "sqlite", + "host": None, + "port": None, + "database": str(path.resolve()), + "default_schema": None, + "attached": {}, + } + assert metadata["notes"] == "" + assert conn.in_transaction is False # inspection did not start a transaction + + # attached databases are mapped to their own files; a SQLAlchemy URL cannot know them + other = tmp_path / "other.db" + conn.execute(f"ATTACH DATABASE '{other}' AS reporting") + conn.execute("ATTACH DATABASE ':memory:' AS scratch") + source = get_sql_metadata("orders", 3, db_connection=conn)[SQL_METADATA]["source"] + assert source["attached"] == {"reporting": str(other.resolve())} + memory = sqlite3.connect(":memory:") + memory.execute(f"ATTACH DATABASE '{other}' AS reporting") + in_memory, notes = get_sql_source(memory) + assert (in_memory["database"], in_memory["attached"], notes) == ( + "", + {"reporting": str(other.resolve())}, + "", + ) + assert source["database"] == str(path.resolve()) + assert get_sql_source(f"sqlite:///{path}")[0]["attached"] is None + + engine = create_engine(f"sqlite:///{path}") + with engine.connect() as sa_conn: + read = get_sql_metadata("SELECT * FROM orders", pd.DataFrame(), db_connection=sa_conn) + assert read[SQL_METADATA]["operation"] == "read" + assert read[SQL_METADATA]["source"]["database"] == str(path.resolve()) + assert ( + get_sql_metadata("orders", 3, db_connection=engine)[SQL_METADATA]["source"]["dialect"] + == "sqlite" + ) + + for memory in ( + sqlite3.connect(":memory:"), + create_engine("sqlite://"), + create_engine("sqlite:///:memory:"), + "sqlite:///:memory:", + ): + metadata = get_sql_metadata("orders", 3, db_connection=memory)[SQL_METADATA] + assert metadata["source"] is None, memory + assert "In-memory" in metadata["notes"] + + +def test_get_sql_metadata_url_string_has_no_credentials(): + url = "postgresql+psycopg2://alice:s3cret-pw@db.example:6543/sales?sslmode=require&application_name=x" + metadata = get_sql_metadata("daily_revenue", 10, db_connection=url, schema="reporting") + assert metadata[SQL_METADATA]["source"] == { + "dialect": "postgresql", + "host": "db.example", + "port": 6543, + "database": "sales", + "default_schema": None, + } + assert metadata[SQL_METADATA]["schema"] == "reporting" + serialized = json.dumps(metadata) + for secret in ("alice", "s3cret-pw", "sslmode", "application_name", url): + assert secret not in serialized + + +class _Broken: + @property + def url(self): + raise RuntimeError("postgresql://alice:s3cret-pw@db.example/sales") + + +@pytest.mark.parametrize( + ("connection", "note"), + [ + (object(), "Unsupported connection type for SQL metadata: object"), + (_Broken(), "Could not inspect _Broken connection: RuntimeError"), + ], +) +def test_get_sql_metadata_unknown_or_failing_connection(connection, note): + metadata = get_sql_metadata("orders", 3, db_connection=connection)[SQL_METADATA] + assert metadata["source"] is None + assert metadata["notes"] == note # exception text (which may hold a URL) is not copied + assert metadata["rows"] == 3 and metadata["table_name"] == "orders" + assert get_sql_source(connection) == (None, note) + + +def test_get_sql_metadata_is_serializable_without_openlineage(tmp_path: pathlib.Path): + script = f""" +import json, sqlite3, sys +from hamilton.io import utils +md = utils.get_sql_metadata("SELECT * FROM orders", 1, db_connection=sqlite3.connect({str(tmp_path / "a.db")!r})) +json.dumps(md) +assert not any(m.startswith("openlineage") for m in sys.modules), "optional stack was imported" +""" + subprocess.run([sys.executable, "-c", script], check=True) def test_get_file_metadata(tmp_path: pathlib.Path): @@ -52,3 +216,12 @@ def test_get_file_metadata_url_schema(): metadata = get_file_metadata(url) assert metadata["file_metadata"]["path"] == url assert metadata["file_metadata"]["scheme"] == "s3" + + +def test_get_sql_metadata_statement_detection_is_linear(tmp_path): + conn = sqlite3.connect(tmp_path / "a.db") + text = "/* c */ " * 5000 + "pragma table_info(t)" + start = time.perf_counter() + metadata = get_sql_metadata(text, 1, db_connection=conn, operation="read")[SQL_METADATA] + assert time.perf_counter() - start < 1 + assert metadata["table_name"] == text diff --git a/tests/plugins/test_h_openlineage.py b/tests/plugins/test_h_openlineage.py new file mode 100644 index 000000000..5ee3e2eec --- /dev/null +++ b/tests/plugins/test_h_openlineage.py @@ -0,0 +1,801 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import json +import logging +import sqlite3 +import warnings + +import pandas as pd +import pytest +from sqlalchemy import create_engine, text + +from hamilton import ad_hoc_utils, driver +from hamilton.function_modifiers import load_from, save_to, source, value +from hamilton.io import utils + +pytest.importorskip("openlineage.client") + +from openlineage.client import OpenLineageClient # noqa: E402 +from openlineage.client.transport.file import FileConfig, FileTransport # noqa: E402 + +from hamilton.plugins import h_openlineage # noqa: E402 + +REVENUE_QUERY = """ +-- daily revenue per customer country +WITH paid AS (SELECT * FROM orders WHERE status = 'paid') +select p.order_date, c.country, p.amount +FROM paid p JOIN "customers" c ON p.customer_id = c.id +""" + + +def sqlite_metadata(path, query_or_table, results=1, schema=None): + return utils.get_sql_metadata( + query_or_table, results, db_connection=sqlite3.connect(path), schema=schema + ) + + +def identities(datasets): + return sorted((d.namespace, d.name) for d in datasets) + + +@pytest.mark.parametrize( + ("query", "tables"), + [ + ("SELECT * FROM orders", ["orders"]), + ("select o.id from orders o join customers c on o.cid = c.id", ["customers", "orders"]), + (REVENUE_QUERY, ["customers", "orders"]), # CTE `paid` and aliases are not tables + ('/* leading */ SeLeCt id FrOm "Orders"', ["Orders"]), + ("SELECT * FROM (SELECT * FROM orders) sub", ["orders"]), + ], +) +def test_sql_datasets_sqlite_queries(tmp_path, query, tables): + path = tmp_path / "sales.db" + result = h_openlineage.sql_datasets(sqlite_metadata(path, query)) + assert result.notes == [] + assert result.outputs == [] + assert identities(result.inputs) == [(f"sqlite://{path.resolve()}", t) for t in tables] + assert all(d.facets["dataSource"].uri == d.namespace for d in result.inputs) + + +POSTGRES_SOURCE = { + "dialect": "postgresql", + "host": "source.example", + "port": None, + "database": "sales", + "default_schema": "public", +} + + +@pytest.mark.parametrize( + ("query", "default_schema", "names", "note"), + [ + ( + 'SELECT * FROM Orders o JOIN "Customers" c ON o.cid = c.id JOIN sales.Reporting."Daily" d ON 1=1', + "public", + ["sales.public.Customers", "sales.public.orders", "sales.reporting.Daily"], + None, + ), + ("SELECT * FROM orders", None, [], "cannot be fully qualified"), + ("SELECT * FROM archive.orders", None, ["sales.archive.orders"], None), + ], +) +def test_sql_datasets_postgres_naming_without_a_server(query, default_schema, names, note): + """Folding, quoting and schema precedence are pure functions of the metadata.""" + metadata = { + "sql_metadata": { + "query": query, + "source": {**POSTGRES_SOURCE, "default_schema": default_schema}, + } + } + result = h_openlineage.sql_datasets(metadata) + assert identities(result.inputs) == [("postgres://source.example:5432", n) for n in names] + assert (note is None and result.notes == []) or any(note in n for n in result.notes) + + +def test_sql_datasets_writer_schema_applies_to_every_dialect(tmp_path): + postgres = { + "sql_metadata": { + "table_name": "daily_revenue", + "schema": "reporting", + "operation": "write", + "source": POSTGRES_SOURCE, + } + } + assert identities(h_openlineage.sql_datasets(postgres).outputs) == [ + ("postgres://source.example:5432", "sales.reporting.daily_revenue") + ] + conn = sqlite3.connect(tmp_path / "a.db") + conn.execute(f"ATTACH DATABASE '{tmp_path / 'reporting.db'}' AS reporting") + sqlite = utils.get_sql_metadata( + "daily_revenue", 1, db_connection=conn, schema="reporting", operation="write" + ) + assert identities(h_openlineage.sql_datasets(sqlite).outputs) == [ + (f"sqlite://{(tmp_path / 'reporting.db').resolve()}", "daily_revenue") + ] + + +def test_sql_datasets_sqlite_attached_databases(tmp_path): + main_path, other_path = tmp_path / "main.db", tmp_path / "other.db" + conn = sqlite3.connect(main_path) + conn.execute(f"ATTACH DATABASE '{other_path}' AS Reporting") + query = "SELECT * FROM reporting.orders JOIN main.customers ON 1=1 JOIN regions ON 1=1" + metadata = utils.get_sql_metadata(query, pd.DataFrame(), db_connection=conn) + result = h_openlineage.sql_datasets(metadata) + main_ns, other_ns = f"sqlite://{main_path.resolve()}", f"sqlite://{other_path.resolve()}" + assert identities(result.inputs) == [ + (main_ns, "customers"), + (main_ns, "regions"), + (other_ns, "orders"), + ] + assert result.notes == [] + + # a URL names only the main file: attached tables are left out, never put in main's namespace + via_url = utils.get_sql_metadata(query, pd.DataFrame(), db_connection=f"sqlite:///{main_path}") + result = h_openlineage.sql_datasets(via_url) + assert identities(result.inputs) == [(main_ns, "customers"), (main_ns, "regions")] + assert any("Cannot tell which file SQLite database 'reporting'" in n for n in result.notes) + + upper = utils.get_sql_metadata( + "SELECT * FROM MAIN.customers", pd.DataFrame(), db_connection=conn + ) + assert identities(h_openlineage.sql_datasets(upper).inputs) == [(main_ns, "customers")] + + memory = sqlite3.connect(":memory:") + memory.execute(f"ATTACH DATABASE '{other_path}' AS reporting") + query = "SELECT * FROM reporting.orders JOIN scratch ON 1=1" + result = h_openlineage.sql_datasets( + utils.get_sql_metadata(query, pd.DataFrame(), db_connection=memory) + ) + assert identities(result.inputs) == [(other_ns, "orders")] + assert any("in-memory database" in n for n in result.notes) + + for query, note in [ + ("SELECT * FROM temp.scratch", "temporary database"), + ("SELECT * FROM missing.orders", "'missing' for table 'orders' is not attached"), + ]: + metadata = utils.get_sql_metadata(query, pd.DataFrame(), db_connection=conn) + result = h_openlineage.sql_datasets(metadata) + assert result.inputs == [] and any(note in n for n in result.notes), result.notes + + +def test_sql_datasets_identity_agreement_and_distinction(tmp_path): + a, b = tmp_path / "a.db", tmp_path / "b.db" + written = h_openlineage.sql_datasets(sqlite_metadata(a, "daily_revenue", results=5)) + read = h_openlineage.sql_datasets(sqlite_metadata(a, "SELECT * FROM daily_revenue")) + assert written.inputs == [] and read.outputs == [] + assert identities(written.outputs) == identities(read.inputs) + other_db = h_openlineage.sql_datasets(sqlite_metadata(b, "daily_revenue", results=5)) + assert identities(other_db.outputs) != identities(written.outputs) + + +def test_sql_datasets_accepts_inner_dict_and_operation_override(tmp_path): + metadata = sqlite_metadata(tmp_path / "a.db", "orders", results=pd.DataFrame({"x": [1]})) + assert metadata["sql_metadata"]["operation"] == "read" + inner = metadata["sql_metadata"] + assert h_openlineage.sql_datasets(inner).inputs + assert h_openlineage.sql_datasets(inner, operation="write").outputs + + +@pytest.mark.parametrize( + ("metadata", "note"), + [ + (utils.get_sql_metadata("orders", 5), "No connection supplied"), + (utils.get_sql_metadata("SELECT * FROM orders", 5), "No connection supplied"), + ({"sql_metadata": {"query": "SELECT * FROM orders", "table_name": None}}, "unknown"), + ( + {"sql_metadata": {"rows": 1, "query": None, "table_name": None, "source": None}}, + "unknown", + ), + ( + { + "sql_metadata": { + "table_name": "orders", + "source": {"dialect": "sqlite", "database": "/x.db"}, + } + }, + "Operation for table 'orders' is unknown", + ), + ( + { + "sql_metadata": { + "table_name": "orders", + "operation": "write", + "source": {"dialect": "mysql"}, + } + }, + "No OpenLineage dataset naming for dialect 'mysql'", + ), + ( + utils.get_sql_metadata( + "daily_revenue", 5, db_connection="postgresql://u:p@h/analytics" + ), + "cannot be fully qualified", + ), + ( + utils.get_sql_metadata("SELECT * FROM orders", 5, db_connection="postgresql:///sales"), + "host is unknown", + ), + ( + { + "sql_metadata": { + "query": "orders where", + "source": {"dialect": "sqlite", "database": "/x.db"}, + } + }, + "SQL parsing error", + ), + ], +) +def test_sql_datasets_incomplete_metadata_is_diagnosed_not_guessed(metadata, note): + result = h_openlineage.sql_datasets(metadata) + assert result.inputs == [] and result.outputs == [] + assert any(note in n for n in result.notes), result.notes + + +def test_sql_datasets_parser_unavailable_or_failing(tmp_path, monkeypatch): + metadata = sqlite_metadata(tmp_path / "a.db", "SELECT * FROM orders") + monkeypatch.setitem(__import__("sys").modules, "openlineage_sql", None) + missing = h_openlineage.sql_datasets(metadata) + assert missing.inputs == [] and "openlineage-sql is not installed" in missing.notes[0] + assert missing.query == "SELECT * FROM orders" # recorded as a read query: still the job's SQL + monkeypatch.undo() + + import openlineage_sql + + def boom(*args, **kwargs): + raise RuntimeError("parser exploded") + + monkeypatch.setattr(openlineage_sql, "parse", boom) + failed = h_openlineage.sql_datasets(metadata) + assert failed.inputs == [] and failed.notes == ["SQL parsing failed: RuntimeError"] + assert failed.query == "SELECT * FROM orders" + + +def test_sql_datasets_recorded_read_query_naming_no_table_keeps_its_statement(tmp_path): + result = h_openlineage.sql_datasets(sqlite_metadata(tmp_path / "a.db", "SELECT 1")) + assert result.inputs == [] and result.query == "SELECT 1" + + +def revenue_module(query=REVENUE_QUERY): + @load_from.sql(query_or_table=value(query), db_connection=source("sales_db")) + def order_lines(df: pd.DataFrame) -> pd.DataFrame: + return df + + def daily_revenue(order_lines: pd.DataFrame) -> pd.DataFrame: + return order_lines.groupby(["order_date", "country"], as_index=False)["amount"].sum() + + @save_to.sql( + table_name=source("revenue_table"), + db_connection=source("warehouse_db"), + schema=source("revenue_schema"), + if_exists=value("replace"), + index=value(False), + output_name_="saved_revenue", + ) + def revenue_report(daily_revenue: pd.DataFrame) -> pd.DataFrame: + return daily_revenue + + return ad_hoc_utils.create_temporary_module(order_lines, daily_revenue, revenue_report) + + +def seed_sales(connection, schema=None): + prefix = f"{schema}." if schema else "" + pd.DataFrame( + { + "customer_id": [1, 1, 2], + "order_date": ["d1", "d1", "d2"], + "amount": [10.0, 5.0, 7.0], + "status": ["paid", "paid", "open"], + } + ).to_sql("orders", connection, schema=schema, index=False, if_exists="replace") + pd.DataFrame({"id": [1, 2], "country": ["NL", "DE"]}).to_sql( + "customers", connection, schema=schema, index=False, if_exists="replace" + ) + return prefix + + +def run_with_lineage( + tmp_path, namespace, inputs, module=None, final_vars=("saved_revenue",), **adapter_kwargs +): + events_path = tmp_path / f"events-{namespace}.json" + client = OpenLineageClient( + transport=FileTransport(FileConfig(log_file_path=str(events_path), append=True)) + ) + adapter = h_openlineage.OpenLineageAdapter(client, namespace, "revenue_job", **adapter_kwargs) + dr = driver.Builder().with_modules(module or revenue_module()).with_adapters(adapter).build() + result = dr.execute(list(final_vars), inputs=inputs) + events = [json.loads(line) for line in events_path.read_text().splitlines() if line.strip()] + return result, events + + +def dataset_ids(events, key): + return sorted((d["namespace"], d["name"]) for e in events for d in e.get(key) or []) + + +def test_revenue_flow_sqlite_emits_datasource_lineage(tmp_path): + sales_path, warehouse_path = tmp_path / "sales.db", tmp_path / "warehouse.db" + sales = sqlite3.connect(sales_path) + seed_sales(sales) + warehouse = create_engine(f"sqlite:///{warehouse_path}") + inputs = { + "sales_db": sales, + "warehouse_db": warehouse, + "revenue_table": "daily_revenue", + "revenue_schema": None, + } + result, events = run_with_lineage( + tmp_path, "demo_namespace", inputs, sql_dataset_identity="datasource" + ) + + written = pd.read_sql("SELECT * FROM daily_revenue", warehouse) + assert written["amount"].tolist() == [15.0] + assert result["saved_revenue"]["sql_metadata"]["rows"] == 1 + + sales_ns = f"sqlite://{sales_path.resolve()}" + assert dataset_ids(events, "inputs") == [(sales_ns, "customers"), (sales_ns, "orders")] + assert dataset_ids(events, "outputs") == [ + (f"sqlite://{warehouse_path.resolve()}", "daily_revenue") + ] + assert {e["job"]["namespace"] for e in events} == {"demo_namespace"} + assert {e["job"]["name"] for e in events} == {"revenue_job"} + assert [e["eventType"] for e in events] == ["START", "RUNNING", "RUNNING", "COMPLETE"] + running = [e for e in events if e["eventType"] == "RUNNING"] + assert running[0]["job"]["facets"]["sql"]["query"] == REVENUE_QUERY + output = running[1]["outputs"][0] + assert output["facets"]["schema"]["fields"][0]["name"] == "order_date" + assert output["facets"]["dataSource"]["uri"] == output["namespace"] + + # dataset identity comes from the datasource, not the job namespace + _, other_events = run_with_lineage( + tmp_path, "another_namespace", inputs, sql_dataset_identity="datasource" + ) + assert dataset_ids(other_events, "inputs") == dataset_ids(events, "inputs") + assert dataset_ids(other_events, "outputs") == dataset_ids(events, "outputs") + assert {e["job"]["namespace"] for e in other_events} == {"another_namespace"} + sales.close() + + +def test_lineage_failures_do_not_interrupt_data_work(tmp_path, monkeypatch, caplog): + sales = sqlite3.connect(tmp_path / "sales.db") + seed_sales(sales) + warehouse = create_engine(f"sqlite:///{tmp_path / 'warehouse.db'}") + inputs = { + "sales_db": sales, + "warehouse_db": warehouse, + "revenue_table": "r", + "revenue_schema": None, + } + + def boom(*args, **kwargs): + raise RuntimeError("conversion exploded postgresql://alice:s3cret@db/x") + + monkeypatch.setattr(h_openlineage, "sql_datasets", boom) + with caplog.at_level(logging.WARNING, logger="hamilton.plugins.h_openlineage"): + result, events = run_with_lineage(tmp_path, "ns", inputs, sql_dataset_identity="datasource") + assert result["saved_revenue"]["sql_metadata"]["rows"] == 1 + assert pd.read_sql("SELECT * FROM r", warehouse)["amount"].tolist() == [15.0] + assert [e["eventType"] for e in events] == ["START", "RUNNING", "RUNNING", "COMPLETE"] + assert dataset_ids(events, "inputs") == [] and dataset_ids(events, "outputs") == [] + assert "conversion failed" in caplog.text and "RuntimeError" in caplog.text + monkeypatch.undo() + + # producer boundary: connection inspection fails, data still flows, notes are logged + monkeypatch.setattr(utils, "_inspect_sql_source", boom) + with caplog.at_level(logging.WARNING, logger="hamilton.plugins.h_openlineage"): + result, events = run_with_lineage( + tmp_path, "ns2", inputs, sql_dataset_identity="datasource" + ) + assert result["saved_revenue"]["sql_metadata"]["source"] is None + assert "Could not inspect" in result["saved_revenue"]["sql_metadata"]["notes"] + assert "s3cret" not in json.dumps(result["saved_revenue"]) + caplog.text + assert dataset_ids(events, "outputs") == [] + monkeypatch.undo() + + # an actual SQL failure still fails the graph + with pytest.raises(Exception, match="no such table"): + run_with_lineage(tmp_path, "ns3", {**inputs, "sales_db": sqlite3.connect(":memory:")}) + sales.close() + + +def test_revenue_flow_postgres_names_server_database_schema(tmp_path, postgres_schema): + engine, schema = postgres_schema + seed_sales(engine, schema) + with engine.begin() as conn: + conn.execute(text(f"CREATE SCHEMA {schema}_reporting")) + try: + query = REVENUE_QUERY.replace("FROM orders", f"FROM {schema}.orders").replace( + '"customers"', f'{schema}."customers"' + ) + module = revenue_module(query) + events_path = tmp_path / "events.json" + client = OpenLineageClient( + transport=FileTransport(FileConfig(log_file_path=str(events_path), append=True)) + ) + adapter = h_openlineage.OpenLineageAdapter( + client, "job_ns", "revenue_job", sql_dataset_identity="datasource" + ) + dr = driver.Builder().with_modules(module).with_adapters(adapter).build() + with engine.connect() as warehouse_conn: + dr.execute( + ["saved_revenue"], + inputs={ + "sales_db": engine, + "warehouse_db": warehouse_conn, + "revenue_table": "daily_revenue", + "revenue_schema": f"{schema}_reporting", + }, + ) + warehouse_conn.commit() + events_text = events_path.read_text() + events = [json.loads(line) for line in events_text.splitlines() if line.strip()] + namespace = f"postgres://{engine.url.host}:{engine.url.port or 5432}" + db = engine.url.database + assert dataset_ids(events, "inputs") == [ + (namespace, f"{db}.{schema}.customers"), + (namespace, f"{db}.{schema}.orders"), + ] + assert dataset_ids(events, "outputs") == [ + (namespace, f"{db}.{schema}_reporting.daily_revenue") + ] + # the password may be an ordinary word found in event tags, so look for it as a credential + assert f":{engine.url.password}@" not in events_text + assert engine.url.render_as_string(hide_password=False) not in events_text + assert "job_ns" not in json.dumps(dataset_ids(events, "inputs")) + + # a later read of the report resolves to the identity it was written under + read_back = utils.get_sql_metadata( + f"SELECT * FROM {schema}_reporting.daily_revenue", pd.DataFrame(), db_connection=engine + ) + assert identities(h_openlineage.sql_datasets(read_back).inputs) == dataset_ids( + events, "outputs" + ) + # unqualified names use the connection's verified default schema; folding matches the server + default = utils.get_sql_metadata( + "SELECT * FROM Daily_Revenue", pd.DataFrame(), db_connection=engine + ) + with engine.connect() as conn: + current = conn.execute(text("select current_schema()")).scalar() + assert identities(h_openlineage.sql_datasets(default).inputs) == [ + (namespace, f"{db}.{current}.daily_revenue") + ] + finally: + with engine.begin() as conn: + conn.execute(text(f"DROP SCHEMA {schema}_reporting CASCADE")) + + +def legacy_inputs(tmp_path): + sales = sqlite3.connect(tmp_path / "sales.db") + seed_sales(sales) + return { + "sales_db": sales, + "warehouse_db": create_engine(f"sqlite:///{tmp_path / 'warehouse.db'}"), + "revenue_table": "daily_revenue", + "revenue_schema": None, + } + + +# string columns are "str" on pandas 3 and "object" before it; the facet records what pandas reports +STRING_DTYPE = str(pd.Series(["x"]).dtype) +REVENUE_FIELDS = [ + {"fields": [], "name": "order_date", "type": STRING_DTYPE}, + {"fields": [], "name": "country", "type": STRING_DTYPE}, + {"fields": [], "name": "amount", "type": "float64"}, +] + + +def strip_private(value): + """Drops the client-generated ``_producer``/``_schemaURL`` keys from an event fragment.""" + if isinstance(value, dict): + return {k: strip_private(v) for k, v in value.items() if not k.startswith("_")} + if isinstance(value, list): + return [strip_private(v) for v in value] + return value + + +def test_default_adapter_keeps_legacy_sql_identity(tmp_path, monkeypatch): + """Pinned against the apache/main adapter: job namespace, bare table name, SQL job facet.""" + monkeypatch.setitem(__import__("sys").modules, "openlineage_sql", None) # not needed + with pytest.warns(FutureWarning, match="sql_dataset_identity='datasource'"): + _, events = run_with_lineage(tmp_path, "demo_namespace", legacy_inputs(tmp_path)) + read, write = (strip_private(e) for e in events if e["eventType"] == "RUNNING") + # a query read had no table name in metadata 1.0.0, so the dataset has none + assert read["inputs"] == [ + { + "namespace": "demo_namespace", + "facets": {"schema": {"fields": REVENUE_FIELDS}}, + "inputFacets": {}, + } + ] + assert read["job"]["facets"] == {"sql": {"query": REVENUE_QUERY}} + assert write["outputs"] == [ + { + "namespace": "demo_namespace", + "name": "daily_revenue", + "facets": {"schema": {"fields": REVENUE_FIELDS}}, + "outputFacets": {}, + } + ] + assert write["job"]["facets"] == {} + + +def test_default_adapter_keeps_legacy_names_for_unusual_sql_strings(tmp_path): + """Pinned against the apache/main adapter, which filed names by ``"SELECT" in text``.""" + + @load_from.sql(query_or_table=value("select * from orders"), db_connection=source("sales_db")) + def lower_query(df: pd.DataFrame) -> pd.DataFrame: + return df + + @save_to.sql( + table_name=value("USER_SELECTIONS"), + db_connection=source("warehouse_db"), + if_exists=value("replace"), + index=value(False), + output_name_="saved_selections", + ) + def selections(lower_query: pd.DataFrame) -> pd.DataFrame: + return lower_query + + module = ad_hoc_utils.create_temporary_module(lower_query, selections) + with pytest.warns(FutureWarning): + _, events = run_with_lineage( + tmp_path, + "demo_namespace", + legacy_inputs(tmp_path), + module=module, + final_vars=["saved_selections"], + ) + read, write = (strip_private(e) for e in events if e["eventType"] == "RUNNING") + assert [(d["namespace"], d.get("name")) for d in read["inputs"]] == [ + ("demo_namespace", "select * from orders") + ] + assert read["job"]["facets"] == {"sql": {}} + # 1.0.0 filed a name containing SELECT as a query, so the legacy dataset has no name + assert [(d["namespace"], d.get("name")) for d in write["outputs"]] == [("demo_namespace", None)] + + +@pytest.mark.parametrize( + ("sql_metadata", "name", "query"), + [ + # hand-built by custom loaders (as in earlier versions of examples/openlineage) + ( + {"query": "SELECT * FROM orders", "table_name": "orders"}, + "orders", + "SELECT * FROM orders", + ), + ({"query": None, "table_name": "SELECT_LOG"}, "SELECT_LOG", None), + ({"query": "SELECT * FROM orders", "table_name": None}, None, "SELECT * FROM orders"), + ({"query": "select * from orders", "table_name": None}, None, "select * from orders"), + # 1.1.0 filing of a lower-case read; 1.0.0 filed (and named) it as a table + ( + {"query": "select * from orders", "table_name": None, "__version__": "1.1.0"}, + "select * from orders", + None, + ), + ], +) +def test_legacy_dataset_helpers_match_1_0_0(sql_metadata, name, query): + """Pinned against apache/main's create_input_dataset / create_output_dataset.""" + metadata = {"sql_metadata": sql_metadata} + inputs, sql_facet = h_openlineage.create_input_dataset("ns", metadata, None) + (output,) = h_openlineage.create_output_dataset("ns", metadata, None) + assert [(d.namespace, d.name) for d in inputs] == [("ns", name)] + assert (output.namespace, output.name) == ("ns", name) + assert sql_facet.query == query + + +def test_default_adapter_legacy_table_read(tmp_path): + @load_from.sql(query_or_table=value("customers"), db_connection=source("sales_url")) + def customers(df: pd.DataFrame) -> pd.DataFrame: + return df + + seed_sales(sqlite3.connect(tmp_path / "sales.db")) + module = ad_hoc_utils.create_temporary_module(customers) + events_path = tmp_path / "events.json" + client = OpenLineageClient( + transport=FileTransport(FileConfig(log_file_path=str(events_path), append=True)) + ) + adapter = h_openlineage.OpenLineageAdapter(client, "demo_namespace", "job") + dr = driver.Builder().with_modules(module).with_adapters(adapter).build() + with pytest.warns(FutureWarning): + dr.execute(["customers"], inputs={"sales_url": f"sqlite:///{tmp_path / 'sales.db'}"}) + events = [json.loads(line) for line in events_path.read_text().splitlines()] + (read,) = (strip_private(e) for e in events if e["eventType"] == "RUNNING") + assert [(d["namespace"], d["name"]) for d in read["inputs"]] == [ + ("demo_namespace", "customers") + ] + assert read["job"]["facets"] == {"sql": {}} # the 1.0.0 facet for a table read has no query + + +def test_legacy_identity_warning_fires_once_and_only_by_default(tmp_path, recwarn): + run_with_lineage(tmp_path, "ns_default", legacy_inputs(tmp_path)) + identity_warnings = [w for w in recwarn.list if "sql_dataset_identity" in str(w.message)] + assert [w.category for w in identity_warnings] == [FutureWarning] # two SQL nodes, one warning + message = str(identity_warnings[0].message) + assert "sql_dataset_identity='legacy'" in message and "OpenLineageAdapter/" in message + recwarn.clear() + for identity in ("legacy", "datasource"): + run_with_lineage( + tmp_path, f"ns_{identity}", legacy_inputs(tmp_path), sql_dataset_identity=identity + ) + assert not [w for w in recwarn.list if "sql_dataset_identity" in str(w.message)] + + +def test_legacy_identity_warning_as_error_does_not_fail_the_node(tmp_path, caplog): + with warnings.catch_warnings(): + warnings.simplefilter("error", FutureWarning) + with caplog.at_level(logging.WARNING, logger="hamilton.plugins.h_openlineage"): + result, events = run_with_lineage(tmp_path, "ns", legacy_inputs(tmp_path)) + assert result["saved_revenue"]["sql_metadata"]["rows"] == 1 + assert dataset_ids(events, "outputs") == [("ns", "daily_revenue")] + assert "legacy identity" in caplog.text + + +def test_legacy_identity_conversion_failure_does_not_fail_the_node(tmp_path, monkeypatch, caplog): + def boom(*args, **kwargs): + raise RuntimeError("legacy conversion exploded") + + monkeypatch.setattr(h_openlineage, "create_output_dataset", boom) + with caplog.at_level(logging.WARNING, logger="hamilton.plugins.h_openlineage"): + result, events = run_with_lineage( + tmp_path, "ns", legacy_inputs(tmp_path), sql_dataset_identity="legacy" + ) + assert result["saved_revenue"]["sql_metadata"]["rows"] == 1 + assert dataset_ids(events, "outputs") == [] + assert "conversion failed" in caplog.text + + +def test_invalid_sql_dataset_identity_fails_at_construction(): + with pytest.raises(ValueError, match="sql_dataset_identity must be one of"): + h_openlineage.OpenLineageAdapter(None, "ns", "job", sql_dataset_identity="qualified") + + +@pytest.mark.parametrize( + ("table", "query", "table_name"), + [ + ("daily revenue", None, "daily revenue"), + ("SELECTED_ROWS", "SELECTED_ROWS", None), + ("SELECT results", "SELECT results", None), + ], +) +def test_written_table_names_are_emitted_as_tables(tmp_path, table, query, table_name): + inputs = {**legacy_inputs(tmp_path), "revenue_table": table} + result, events = run_with_lineage(tmp_path, "ns", inputs, sql_dataset_identity="datasource") + metadata = result["saved_revenue"]["sql_metadata"] + # the 1.0.0 query/table_name filing is kept; operation marks the name as the table written + assert (metadata["query"], metadata["table_name"], metadata["operation"]) == ( + query, + table_name, + "write", + ) + assert dataset_ids(events, "outputs") == [ + (f"sqlite://{(tmp_path / 'warehouse.db').resolve()}", table) + ] + (write,) = (e for e in events if e["eventType"] == "RUNNING" and e.get("outputs")) + assert "sql" not in write["job"]["facets"] # a table name is not the job's SQL + + +def test_read_by_a_non_plain_table_name_reports_no_statement(tmp_path): + engine = create_engine(f"sqlite:///{tmp_path / 'a.db'}") + pd.DataFrame({"x": [1]}).to_sql("daily revenue", engine, index=False) + metadata = utils.get_sql_metadata( + "daily revenue", pd.DataFrame(), db_connection=engine, operation="read" + ) + result = h_openlineage.sql_datasets(metadata) + assert result.inputs == [] and result.notes # not guessed + assert result.query is None # a table name is not the job's SQL + + +@pytest.mark.parametrize("parser", ["missing", "failing"]) +def test_sql_datasets_written_name_without_parser(tmp_path, monkeypatch, parser): + if parser == "missing": + monkeypatch.setitem(__import__("sys").modules, "openlineage_sql", None) + else: + import openlineage_sql + + def boom(*args, **kwargs): + raise RuntimeError("parser exploded") + + monkeypatch.setattr(openlineage_sql, "parse", boom) + conn = sqlite3.connect(tmp_path / "a.db") + + def written(name): + metadata = utils.get_sql_metadata(name, 3, db_connection=conn, operation="write") + return h_openlineage.sql_datasets(metadata) + + assert identities(written("USER_SELECTIONS").outputs) == [ + (f"sqlite://{(tmp_path / 'a.db').resolve()}", "USER_SELECTIONS") + ] + # a table name and a statement can't be told apart without a parse: left out, explained + for ambiguous in ( + "SELECT results", + "daily revenue", + "INSERT INTO t SELECT * FROM s", + "TRUNCATE t; INSERT INTO t SELECT * FROM s", + "(SELECT * FROM s)", + ): + result = written(ambiguous) + assert result.outputs == [] and result.notes, ambiguous + + +@pytest.mark.parametrize( + ("statement", "inputs", "outputs"), + [ + ("INSERT INTO t SELECT * FROM s", ["s"], ["t"]), + ("CREATE TABLE t AS SELECT a FROM s", ["s"], ["t"]), + ("TRUNCATE t; INSERT INTO t SELECT * FROM s", ["s"], ["t"]), + ("insert into t select * from s", ["s"], ["t"]), # filed under table_name + ], +) +def test_sql_datasets_write_statements_are_parsed(tmp_path, statement, inputs, outputs): + """A custom saver recording the statement it ran gets the tables, not the SQL as a name.""" + metadata = sqlite_metadata(tmp_path / "a.db", statement, results=3) + ns = f"sqlite://{(tmp_path / 'a.db').resolve()}" + result = h_openlineage.sql_datasets(metadata, operation="write") + assert identities(result.outputs) == [(ns, t) for t in outputs] + assert identities(result.inputs) == [(ns, t) for t in inputs] + assert result.query == statement + + +def test_sql_datasets_write_of_a_pure_query_is_left_out(tmp_path): + metadata = sqlite_metadata(tmp_path / "a.db", "SELECT * FROM s", results=3) + result = h_openlineage.sql_datasets(metadata, operation="write") + assert result.outputs == [] and result.inputs == [] + assert result.notes == ["Write metadata holds a statement that writes no table"] + + +@pytest.mark.parametrize( + ("query", "tables"), + [("(select * from orders)", ["orders"]), ("pragma table_info(orders)", [])], +) +def test_read_strings_that_are_not_names_are_parsed_not_named(tmp_path, query, tables): + """1.0.0 files these under table_name; they must not become a dataset named after the SQL.""" + metadata = sqlite_metadata(tmp_path / "a.db", query, results=pd.DataFrame()) + assert metadata["sql_metadata"]["table_name"] == query + result = h_openlineage.sql_datasets(metadata) + # reported as the job's SQL only when the parse shows it is a statement naming tables + assert result.query == (query if tables else None) + assert identities(result.inputs) == [ + (f"sqlite://{(tmp_path / 'a.db').resolve()}", t) for t in tables + ] + assert tables or result.notes + + +def test_attached_sqlite_database_is_attributed_to_its_own_file(tmp_path): + main_path, other_path = tmp_path / "main.db", tmp_path / "other.db" + seed_sales(sqlite3.connect(other_path)) + conn = sqlite3.connect(main_path) + conn.execute(f"ATTACH DATABASE '{other_path}' AS reporting") + + @load_from.sql( + query_or_table=value("SELECT * FROM reporting.orders"), db_connection=source("db") + ) + def orders(df: pd.DataFrame) -> pd.DataFrame: + return df + + events_path = tmp_path / "events.json" + client = OpenLineageClient( + transport=FileTransport(FileConfig(log_file_path=str(events_path), append=True)) + ) + adapter = h_openlineage.OpenLineageAdapter( + client, "ns", "job", sql_dataset_identity="datasource" + ) + module = ad_hoc_utils.create_temporary_module(orders) + dr = driver.Builder().with_modules(module).with_adapters(adapter).build() + dr.execute(["orders"], inputs={"db": conn}) + events = [json.loads(line) for line in events_path.read_text().splitlines()] + assert dataset_ids(events, "inputs") == [(f"sqlite://{other_path.resolve()}", "orders")] diff --git a/tests/plugins/test_pandas_extensions.py b/tests/plugins/test_pandas_extensions.py index e4380ce78..2ce8081d9 100644 --- a/tests/plugins/test_pandas_extensions.py +++ b/tests/plugins/test_pandas_extensions.py @@ -106,13 +106,17 @@ def test_pandas_json(df: pd.DataFrame, tmp_path: pathlib.Path) -> None: @pytest.mark.parametrize( - "conn", + ("connect", "identified"), [ - sqlite3.connect(":memory:"), - create_engine("sqlite://"), + (lambda _: sqlite3.connect(":memory:"), False), + (lambda _: create_engine("sqlite://"), False), + (lambda path: sqlite3.connect(path), True), + (lambda path: create_engine(f"sqlite:///{path}"), True), ], ) -def test_pandas_sql(df: pd.DataFrame, conn: str | sqlite3.Connection) -> None: +def test_pandas_sql(df: pd.DataFrame, connect, identified: bool, tmp_path) -> None: + path = tmp_path / "test.db" + conn = connect(path) writer = PandasSqlWriter(table_name="bar", db_connection=conn) kwargs1 = writer._get_saving_kwargs() metadata1 = writer.save_data(df) @@ -130,11 +134,98 @@ def test_pandas_sql(df: pd.DataFrame, conn: str | sqlite3.Connection) -> None: assert metadata1["sql_metadata"]["rows"] == 1 assert metadata2["sql_metadata"]["rows"] == 1 assert metadata1["dataframe_metadata"]["datatypes"] == [str(df["foo"].dtype)] + assert metadata1["sql_metadata"]["operation"] == "write" + assert metadata2["sql_metadata"]["operation"] == "read" + assert metadata1["sql_metadata"]["table_name"] == "bar" + assert metadata2["sql_metadata"]["query"] == "SELECT foo FROM bar" + if identified: + expected = {"dialect": "sqlite", "database": str(path.resolve())} + assert expected.items() <= metadata1["sql_metadata"]["source"].items() + assert metadata1["sql_metadata"]["source"] == metadata2["sql_metadata"]["source"] + else: + assert metadata1["sql_metadata"]["source"] is None + assert metadata2["sql_metadata"]["source"] is None + assert metadata1["sql_metadata"]["notes"] if hasattr(conn, "close"): conn.close() +def test_pandas_sql_decorators_capture_source(tmp_path) -> None: + """@load_from.sql / @save_to.sql carry datasource context with no custom metadata code.""" + from hamilton import ad_hoc_utils, driver + from hamilton.function_modifiers import load_from, save_to, source, value + + sales = sqlite3.connect(tmp_path / "sales.db") + pd.DataFrame({"id": [1, 2], "amount": [10.0, 5.0]}).to_sql("orders", sales, index=False) + warehouse = create_engine(f"sqlite:///{tmp_path / 'warehouse.db'}") + + @load_from.sql( + query_or_table=value("select id, amount from orders"), db_connection=source("sales") + ) + def orders(df: pd.DataFrame) -> pd.DataFrame: + return df + + @save_to.sql( + table_name=value("revenue"), + db_connection=source("warehouse"), + if_exists=value("replace"), + index=value(False), + output_name_="saved", + ) + def revenue(orders: pd.DataFrame) -> pd.DataFrame: + return orders.assign(revenue=orders["amount"] * 2) + + module = ad_hoc_utils.create_temporary_module(orders, revenue) + dr = driver.Builder().with_modules(module).build() + result = dr.execute( + ["orders.load_data.df", "saved"], inputs={"sales": sales, "warehouse": warehouse} + ) + + loaded_df, loaded_metadata = result["orders.load_data.df"] + assert loaded_metadata["sql_metadata"]["source"]["database"] == str( + (tmp_path / "sales.db").resolve() + ) + assert loaded_metadata["sql_metadata"]["operation"] == "read" + saved_metadata = result["saved"] + assert saved_metadata["sql_metadata"]["source"]["database"] == str( + (tmp_path / "warehouse.db").resolve() + ) + assert saved_metadata["sql_metadata"]["table_name"] == "revenue" + assert pd.read_sql("select * from revenue", warehouse)["revenue"].tolist() == [20.0, 10.0] + sales.close() + + +def test_pandas_sql_postgres_forms(postgres_schema) -> None: + """URL string, Engine and Connection forms all identify server, database and schema.""" + engine, schema = postgres_schema + url = engine.url.render_as_string(hide_password=False) + df = pd.DataFrame({"foo": ["bar"]}) + written = PandasSqlWriter( + table_name="orders", db_connection=engine, schema=schema, index=False + ).save_data(df) + source = written["sql_metadata"]["source"] + assert source["dialect"] == "postgresql" + assert source["host"] == engine.url.host + assert source["port"] == engine.url.port + assert source["database"] == engine.url.database + assert source["default_schema"] # verified by SQLAlchemy at connect time, not assumed + assert written["sql_metadata"]["schema"] == schema + + query = f"SELECT * FROM {schema}.orders" + with engine.connect() as conn: + df_conn, via_conn = PandasSqlReader(query_or_table=query, db_connection=conn).load_data( + pd.DataFrame + ) + df_url, via_url = PandasSqlReader(query_or_table=query, db_connection=url).load_data( + pd.DataFrame + ) + assert df.equals(df_conn) and df.equals(df_url) + assert via_conn["sql_metadata"]["source"] == source + assert via_url["sql_metadata"]["source"] == {**source, "default_schema": None} + assert engine.url.password not in str(via_url) + + def test_pandas_xml_reader(tmp_path: pathlib.Path) -> None: path_to_test = "tests/resources/data/test_load_from_data.xml" reader = PandasXmlReader(path_or_buffer=path_to_test) diff --git a/writeups/adr/2609-01-staged-sql-dataset-identity.md b/writeups/adr/2609-01-staged-sql-dataset-identity.md new file mode 100644 index 000000000..82e754b6c --- /dev/null +++ b/writeups/adr/2609-01-staged-sql-dataset-identity.md @@ -0,0 +1,69 @@ + + +# 2609-01. Staged, opt-in datasource identity for OpenLineage SQL datasets + +**Status:** Accepted (2026-09, apache/hamilton#1720) + +## Context + +`OpenLineageAdapter` has always named SQL loader/saver datasets under the adapter's *job* +namespace with the bare `table_name`. Metadata version 1.1.0 records the datasource a SQL node +used, which allows naming datasets the OpenLineage way (`postgres://host:port` + +`db.schema.table`, `sqlite://{file}` + `table`). That lets a report written by one job connect +to the job that reads it. + +Switching names changes the identity of every existing SQL dataset. Lineage history stops +connecting, and anything keyed to the old names in a lineage backend (ownership, tags, alerts, +policies) silently stops matching. Datasource naming also needs `openlineage-sql`, which has no +Windows wheel, and it leaves out datasets it cannot identify instead of guessing. So an upgrade +alone would make some datasets disappear. + +## Decision + +- `OpenLineageAdapter(..., sql_dataset_identity="legacy" | "datasource")`. The default stays + `"legacy"`, which emits exactly what earlier releases emitted (namespace, name and facets). It + is pinned in tests against the previous adapter's output. The legacy path never imports + `openlineage-sql`. +- Leaving the option unset emits one `FutureWarning` per adapter, the first time a SQL node runs. + It names both values, says the default will change in a future **major** release, and links the + migration steps. Passing either value explicitly silences it. +- The warning and every lineage conversion run inside the adapter's never-fail boundary. A + warnings filter set to `error` is caught and logged, so lineage can never fail a node whose SQL + operation already succeeded. There is no strict mode. +- `create_input_dataset` / `create_output_dataset` keep their SQL handling and return shapes, + because they are public module functions. +- An invalid value raises `ValueError` at construction, never during a run. + +## Consequences + +- Upgrading changes no dataset identity. Users migrate on purpose: install `openlineage-sql`, + dry-run with `FileTransport`, move backend metadata, then opt in. This is documented in + `docs/reference/lifecycle-hooks/OpenLineageAdapter.rst`. +- Two naming paths exist until the default flips. The flip is a breaking change reserved for a + major release. + +## Alternatives considered + +- **Switch the default immediately** (the PR's first version). Rejected: it silently breaks every + existing lineage consumer, which is the maintainer's review objection. +- **Warn on every node, or raise on the legacy default.** Rejected: noisy, and raising turns + lineage into a way to fail successful data work. +- **Keep only legacy and publish datasource naming via `sql_datasets()` alone.** Rejected: the + adapter is where users get lineage, and a staged path gives them a migration route. diff --git a/writeups/adr/2609-02-sql-metadata-keeps-1-0-0-filing.md b/writeups/adr/2609-02-sql-metadata-keeps-1-0-0-filing.md new file mode 100644 index 000000000..02fe5a14b --- /dev/null +++ b/writeups/adr/2609-02-sql-metadata-keeps-1-0-0-filing.md @@ -0,0 +1,63 @@ + + +# 2609-02. SQL metadata keeps the 1.0.0 query/table_name filing; intent travels in `operation` + +**Status:** Accepted (2026-09, apache/hamilton#1720) + +## Context + +`get_sql_metadata(query_or_table, results)` files its string under `query` if it contains +`SELECT`, and under `table_name` otherwise. The legacy OpenLineage names are built from those +keys, so changing the filing changes dataset names. One such change is a lower-case `select` +read moving from `table_name` to `query`: on the previous adapter it produced a dataset named +after the query text. + +The filing is also wrong for some inputs. A written table named `USER_SELECTIONS` is filed as a +query. The first version of this change replaced the rule with "contains whitespace", which +misfiled `PandasSqlWriter(table_name="daily revenue")` as a read query and dropped its output +dataset. + +## Decision + +- Keep the 1.0.0 rule for the two-argument form and for every write. The only new filing is on + reads: a string whose first word after SQL comments is `select` or `with` (any case) is a query. + Detection is a linear scan, because a regex over repeated comments backtracked exponentially. +- Add keyword-only `operation="read" | "write"`. `PandasSqlReader`/`PandasSqlWriter` pass it, so + consumers know that a write's string is the target, whichever key holds it. +- The legacy lineage builders re-derive the 1.0.0 filing. They move only the one case 1.1.0 + changed (a query-only lower-case read) back to `table_name`, and only for metadata whose + `__version__` is newer than 1.0.0. Hand-built metadata, such as the `{query, table_name}` dicts + older examples built, is used exactly as given. +- New keys (`schema`, `operation`, `source`, `notes`) are additive, and `__version__` is `1.1.0`. + +## Consequences + +- Existing consumers of `sql_metadata` see the same `query`/`table_name` values for writes and + for two-argument calls. Default-mode lineage names are unchanged for every string tested. +- `query` does not always hold SQL. A written name containing `SELECT` sits there, so lineage code + must use `operation` and not the key alone (see ADR 0003). + +## Alternatives considered + +- **Whitespace heuristic** (first version). Rejected: it misclassifies valid table names. +- **Always file a write's string as `table_name`.** Rejected: it changes existing metadata values + for names containing `SELECT`, contrary to the compatibility goal. +- **Refile every query-only lower-case string in the legacy builders.** Rejected: it renamed + datasets for hand-built metadata, hence the version gate. diff --git a/writeups/adr/2609-03-resolving-sql-writes-and-statements.md b/writeups/adr/2609-03-resolving-sql-writes-and-statements.md new file mode 100644 index 000000000..1baef8090 --- /dev/null +++ b/writeups/adr/2609-03-resolving-sql-writes-and-statements.md @@ -0,0 +1,72 @@ + + +# 2609-03. How datasource-mode lineage resolves what a SQL write names + +**Status:** Accepted (2026-09, apache/hamilton#1720) + +## Context + +Under ADR 0002, a write's string may sit under `query` or `table_name`. It may be a table name +(`daily revenue`, `SELECT results`, `USER_SELECTIONS`) or, for custom savers, a statement +(`INSERT INTO t SELECT ...`). `sql_datasets()` must name the tables written, and it must never +invent a dataset named after SQL text or report a table name as the job's SQL. + +Several intermediate rules failed review: + +- A keyword list for spotting statements without a parser leaked. `DROP`, `TRUNCATE`, `BEGIN` and + `(` all slipped through. +- Treating every write string as a name turned statements into datasets. +- Building the job's `sql` facet from the raw metadata `query` reported `USER_SELECTIONS` as the + job's SQL. + +## Decision + +For a write (`operation="write"`), take the string from either key: + +1. A plain identifier (optionally qualified) is the table. No parser is needed. +2. Anything else is parsed with `openlineage-sql`: + - a statement that writes tables contributes those tables, plus the tables it reads; + - a statement that only reads tables is left out, with a note; + - a string that names no table at all is the table name. +3. Without a parse (parser missing or raising), a non-plain string is left out with a note. A name + and a statement cannot be told apart then. + +For a read, a `table_name` that is not a plain identifier is parsed as SQL rather than used as a +name. + +The job's `sql` facet comes from `SqlDatasets.query`, the statement actually resolved. It is set +only for a string the metadata recorded as a read query, or one whose parse names tables. + +## Consequences + +- No dataset is ever named after SQL text, and table names are not reported as the job's SQL. +- On Windows (no `openlineage-sql`), a written name that is not a plain identifier is left out of + lineage with a note. This is documented, and it only affects datasource mode. +- Known limitation: a table *read* by a name the metadata files as a query (`SELECT_LOG`, + `select-log`) cannot be told apart from a statement. The name is reported as the job's SQL and + no input dataset is emitted. For upper-case `SELECT` names, the previous adapter did the same. + +## Alternatives considered + +- **Keyword list to spot statements without a parser.** Rejected: it leaks by construction. +- **Repeat pandas' `has_table` check to settle table-versus-query reads.** Rejected: a second + database round-trip per read for a rare naming case; lineage stays side-effect free instead. +- **Always trust a write's string as the table name.** Rejected: it names datasets after + statements. diff --git a/writeups/adr/2609-04-sqlite-attached-databases.md b/writeups/adr/2609-04-sqlite-attached-databases.md new file mode 100644 index 000000000..530393649 --- /dev/null +++ b/writeups/adr/2609-04-sqlite-attached-databases.md @@ -0,0 +1,54 @@ + + +# 2609-04. SQLite attached databases are named by their own file + +**Status:** Accepted (2026-09, apache/hamilton#1720) + +## Context + +In SQLite a schema names a database *file*: `main`, `temp`, or anything attached with `ATTACH`. +Datasource naming used the main file's namespace for every table. So `reporting.orders`, stored +in an attached `other.db`, was attributed to `main.db`: the wrong physical datasource. + +## Decision + +- For a standard-library `sqlite3` connection, metadata inspection reads `PRAGMA database_list` + on that same connection and records `source["attached"]` as a schema-name→absolute-path map. It + opens no connection and starts no transaction. For URL strings and SQLAlchemy objects the + mapping cannot be known without opening a connection, so `attached` is `None`. +- A table qualified by an attached schema gets the attached file's namespace, and the name + `{table}`. `main` or no schema uses the main file. Names are compared case-insensitively, as + SQLite does. +- Left out with a note, never attributed to the main file: an unknown mapping (`attached` is + `None`), an unattached schema, `temp`, and unqualified tables of an in-memory main database. An + in-memory main connection with files attached keeps its mapping. + +## Consequences + +- A write to `orders` and a read of `main.orders` resolve to the same dataset. +- Attached-database lineage needs a raw `sqlite3` connection. Other connection forms leave those + tables out and explain why. + +## Alternatives considered + +- **Keep `{schema}.{table}` under the main file's namespace** (first version). Rejected: + misattributes the physical datasource. +- **Run the PRAGMA through a SQLAlchemy connection.** Rejected: SQLAlchemy 2's autobegin would + start a transaction, which breaks the read-only inspection guarantee. diff --git a/writeups/adr/README.md b/writeups/adr/README.md new file mode 100644 index 000000000..1a110e529 --- /dev/null +++ b/writeups/adr/README.md @@ -0,0 +1,36 @@ + + +# Architecture Decision Records + +Each file here records one design decision: the context that forced it, what was decided, what +it costs, and the alternatives that were rejected. The records outlive the commits and review +threads that produced them, so a later change can tell a deliberate choice from an accident. + +- File name: `YYMM-NN-short-title.md`, numbered in order. Numbers are never reused. +- Status: `Accepted`, `Superseded by YYMM-NN`, or `Deprecated`. An accepted record is not edited to + change its decision; write a new record that supersedes it. +- Sections: Status, Context, Decision, Consequences, Alternatives considered. + +| ADR | Title | +| ----- | ------- | +| [2609-01](2609-01-staged-sql-dataset-identity.md) | Staged, opt-in datasource identity for OpenLineage SQL datasets | +| [2609-02](2609-02-sql-metadata-keeps-1-0-0-filing.md) | SQL metadata keeps the 1.0.0 query/table_name filing; intent travels in `operation` | +| [2609-03](2609-03-resolving-sql-writes-and-statements.md) | How datasource-mode lineage resolves what a SQL write names | +| [2609-04](2609-04-sqlite-attached-databases.md) | SQLite attached databases are named by their own file |