Skip to content

[Test] add PDS Q2 end-to-end differential pipeline test (#198) - #254

Closed
Aram164 wants to merge 1 commit into
deem-data:mainfrom
Aram164:test/pds-q2-pipeline
Closed

Aram164 wants to merge 1 commit into
deem-data:mainfrom
Aram164:test/pds-q2-pipeline

Conversation

@Aram164

@Aram164 Aram164 commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Refs #198

Port PDS Q2 from polars-benchmark to a skrub DAG and check Stratum against the original pandas query on the same Arrow-backed fixtures, following the existing Q1 pipeline test in tests/application/pds/test_q1_pipeline.py.

What it checks

  • Five-table joins, the three filters, the grouped minimum, and the join-back on its shared filtered input, on both pandas and Polars.
  • Constant folding of var1/var2/var3, physical operator selection, and the pandas_query selection flag.
  • Projection, mixed-direction sorting, and head(100), exercised with 105 qualifying suppliers so the limit actually drops rows.

The fixtures include tied minimum-cost suppliers and a cheaper supplier outside the selected region, plus wrong-size, wrong-type, and Asia-only parts so each filter is isolated. The prefix comparison normalizes row order, the full-query comparisons check the final sorted order.

@codecov

codecov Bot commented Oct 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
unittests 92.78% <ø> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@e-strauss

Copy link
Copy Markdown
Collaborator

Thanks for the Q2 port, @Aram164.

Before merging I made a two minor changes:

  • Column selection: build_q2 now uses jn2[[...]] instead of jn2.skb.select([...]). Plain indexing is the more natural way to pick a fixed list of columns, and it's closer to the .loc[:, cols] in the original query. skb.select is mainly useful for skrub selectors. Indexing lowers to a ColumnProjectionOp instead of a ColumnSelectorOp, so test_q2_folds_the_sort_and_limit now checks that op, along with its .key and the PolarsColumnProjectionOp it binds to under greedy.
  • Greedy parity: test_q2_matches_reference now uses the polars fixture, so the column-order check against the reference runs with the greedy selector too.

@e-strauss e-strauss closed this in 8d71a2d Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants