CHORE: Add OneBranch release pipelines for mssql-python-odbc#664
Open
jahnvi480 wants to merge 33 commits into
Open
CHORE: Add OneBranch release pipelines for mssql-python-odbc#664jahnvi480 wants to merge 33 commits into
jahnvi480 wants to merge 33 commits into
Conversation
Adds the build and release pipelines for the standalone mssql-python-odbc package: per-platform build stages (Windows x64/arm64, macOS universal2, Linux manylinux_2_28 + musllinux_1_2 for x86_64/aarch64), a consolidate job that gathers all 7 py3-none wheels, the build orchestration pipeline, and the ESRP-based official release pipeline. The build stages run no compilation (data-only package); Linux builds run in tdslibrs ACR PyPA containers and the musllinux tag is auto-detected via libc_ver. Infra follow-ups (ESRP onboarding of the mssql-python-odbc name, registering the ADO build definition to set odbcBuildDefinitionId, and sharing the ESRP variable group) are marked with TODO(infra) comments.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Azure DevOps OneBranch build + official release pipeline YAMLs to produce and publish the standalone mssql-python-odbc “data-only” wheel set (7 platform-specific py3-none-<platform> wheels), aligning the CI/CD half of the ODBC-driver split with the existing mssql-python OneBranch approach.
Changes:
- Introduces a new OneBranch build pipeline that fans out into Windows/macOS/Linux wheel build stages and then consolidates artifacts.
- Adds per-OS stage templates for building the ODBC wheels (Windows x64/arm64, macOS universal2, Linux manylinux/musllinux x86_64/aarch64).
- Adds an official ESRP-based release pipeline to publish the consolidated wheels to PyPI (gated by
releaseToPyPI).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| OneBranchPipelines/build-release-odbc-pipeline.yml | Orchestrates multi-stage build for 7 ODBC wheels + consolidation; configures global SDL settings for this package. |
| OneBranchPipelines/stages/build-odbc-windows-stage.yml | Builds/stages Windows ODBC wheels per-arch and publishes per-stage artifacts. |
| OneBranchPipelines/stages/build-odbc-macos-stage.yml | Builds/stages macOS universal2 ODBC wheel and publishes per-stage artifacts. |
| OneBranchPipelines/stages/build-odbc-linux-stage.yml | Builds/stages Linux ODBC wheels via PyPA containers (manylinux/musllinux) and publishes per-stage artifacts. |
| OneBranchPipelines/jobs/consolidate-odbc-artifacts-job.yml | Downloads all stage artifacts, consolidates wheels into a single dist/, and publishes the consolidated artifact. |
| OneBranchPipelines/official-release-odbc-pipeline.yml | Downloads consolidated artifact and (optionally) releases to PyPI via ESRP. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
📊 Code Coverage Report
Diff CoverageDiff: main...HEAD, staged and unstaged changesNo lines with coverage information in this diff. 📋 Files Needing Attention📉 Files with overall lowest coverage (click to expand)mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.connection.connection.cpp: 76.2%
mssql_python.pybind.ddbc_bindings.cpp: 76.2%
mssql_python.__init__.py: 77.3%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.connection.py: 83.6%
mssql_python.logging.py: 85.5%🔗 Quick Links
|
…nux twine check + container cleanup, reword infra TODOs
…; drop unused targetArch var on Windows stage
Per review feedback, build and release the standalone mssql-python-odbc package from the existing Build-Release-Package-Pipeline (def 2199) and official release pipeline instead of separate ADO definitions. Build pipeline: add odbcWindows/Macos/LinuxConfigs params (2+1+4=7 data-only wheels), reference the 3 odbc stage templates alongside the mssql-python stages, and add a separate ConsolidateOdbc stage publishing drop_ConsolidateOdbc_ConsolidateArtifacts (distinct from mssql-python's drop_Consolidate_ConsolidateArtifacts). Release pipeline: add releasePackage parameter (mssql-python | mssql-python-odbc); select the consolidated artifact accordingly and skip symbol publishing + mssql-py-core version validation for odbc. Delete the redundant standalone orchestrators (build-release-odbc-pipeline.yml, official-release-odbc-pipeline.yml). This removes the need to register a new ADO build definition and the odbcBuildDefinitionId placeholder. Both packages now build from def 2199; the odbc stages consume setup_odbc.py from PR #663, so #663 must merge before these stages can build.
Def 2199 now builds mssql-python and mssql-python-odbc in one run. Both consolidate jobs used DownloadPipelineArtifact@2 with buildType: current, which downloads every artifact in the run and copies all *.whl into dist/. In the merged run this caused: (1) the main Consolidate to sweep the 7 odbc wheels into the mssql-python release (count is warning-only, so silent), and (2) ConsolidateOdbc to see 34 wheels while expecting 7 and hard-fail. Add itemPattern to scope each download to its own package's build-stage artifacts (main: drop_Win_*/drop_MacOS_*/drop_Linux_*; odbc: drop_ODBC_*), giving clean partitioning with no wheel leakage.
…-> 34) The matrix is 9 Windows + 5 macOS + 4 Linux stages x 5 Python versions (cp310-cp314) = 34 wheels across 18 stages. The diagnostic said 27 (7+4+16), stale from an older 3.10-3.13 / 7-Windows matrix. Count check remains warning-only (no behavior change).
official-release-pipeline.yml: gate 'Published symbols' dry-run/summary text to mssql-python so odbc runs no longer claim symbol publishing that never happens; add a mssql-python-only step asserting the wheel metadata pins mssql-python-odbc==18.6.0 (fails by design if released from a pre-pin build). build-odbc-{linux,macos,windows}-stage.yml: assert each built wheel actually contains the ODBC driver binary (msodbcsql*), guarding against the .gitignore '*.so' trap / silent sync_libs() skip shipping an empty data wheel that still passes twine check.
Flip the 34-wheel count check from WARNING-only to a hard exit 1, symmetric with the odbc consolidate job's count guard. A mismatch means a build stage silently produced a partial wheel set (e.g. a missing Python version) or the matrix changed without updating the expected count -- previously this shipped silently.
Mirrors the official-release-pipeline odbc changes: releasePackage parameter, consolidatedArtifactName selection for both artifact downloads, and gating the mssql-py-core version validation + symbol publishing to mssql-python only. Stays on Maven ContentType so it never publishes to PyPI.
Adds the mssql-python-odbc pin guard (gated to mssql-python, matching official-release-pipeline) and gates the 'symbols published' summary messaging on releasePackage==mssql-python so the dummy pipeline exercises the exact same mssql-python validation/messaging as the official pipeline for both packages.
Gate mssql-python and mssql-python-odbc stage groups behind a buildPackage parameter (mssql-python default) so only the selected package builds, halving run time and enabling concurrent per-package builds. Mirrors the releasePackage selector pattern in the release pipelines.
Add effectiveBuildPackage variable that forces 'both' on scheduled (Build.Reason=Schedule) runs while manual/PR runs honour the buildPackage parameter. Mirrors the effectiveOneBranchType pattern so the nightly keeps refreshing every artifact.
### Work Item / Issue Reference > GitHub Issue: #626 ------------------------------------------------------------------- ### Summary cursor.bulkcopy() opens a separate connection through mssql-py-core, which defaulted to a hardcoded 15s connect timeout with no way to override it from Python. I forward the cursor's query timeout (set via connect(timeout=X)) into pycore's connect_timeout when it's set, so the bulk copy connection honors the same limit. timeout=0 stays a no-override, leaving pycore on its 15s default. added tests covering the positive forward, the zero case, and that the cursor snapshot (not a later live connection change) is what's used. --------- Co-authored-by: Gaurav Sharma <223556219+Copilot@users.noreply.github.com> Co-authored-by: Saurabh Singh <saurabh500@gmail.com>
… before tests (#658) ### Work Item / Issue Reference > GitHub Issue: #656 ------------------------------------------------------------------- ### Summary on macOS the test job never removed system ODBC before running, so mssql-python could load a Homebrew unixODBC driver manager / msodbcsql18 from the default dyld path and pass even if the bundled copy was broken. every Linux job already uninstalls system ODBC before tests to force the bundled driver. this brings macOS in line and adds a static check of the bundle itself. - new step in the macOS test job uninstalls Homebrew msodbcsql18/mssql-tools18/unixodbc, deletes leftover libodbc/libodbcinst dylibs and odbcinst config from both Homebrew prefixes, then otool -L's the bundled driver to confirm it's intact. guarded so it's a no-op when nothing is installed. - new test (test_000_dependencies.py) walks the bundled driver's load chain with otool and asserts every inter-library reference is relocatable (@loader_path, not an absolute /opt/homebrew path), which is what breaks on machines without Homebrew. - brew trust on the mssql-release tap so the benchmark step can install msodbcsql18 for the pyodbc comparison (newer Homebrew refuses untrusted taps). --------- Co-authored-by: Gaurav Sharma <sharmag@microsoft.com> Co-authored-by: Gaurav Sharma <223556219+Copilot@users.noreply.github.com>
…edian, warmup) (#680) ### Work Item / Issue Reference > ADO Work Item: Fixed [AB#46419](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/46419) ------------------------------------------------------------------- ### Summary the CI perf benchmark flags false regressions on perf-neutral PRs. across 11 main runs of identical code, the same scenario's raw median swings 1.42x-1.73x (CV 11-18%) purely from shared-runner jitter, so the old 5% gate fired on ~34% of scenario comparisons (and ~30% false "improvements"). this reworks `benchmarks/perf-benchmarking.py` measurement and reporting. no pipeline changes, the step stays continueOnError: - metric is now the median, first sample discarded as warmup (no extra DB passes, so macOS timing is unchanged). - vs-main uses a normalized score instead of raw time: mssql-python's median relative to pyodbc measured on the same runner. pyodbc is pinned, so dividing by it cancels runner-to-runner speed. on the 11 historical runs this roughly halves variance on the I/O-bound scenarios (CV 18%->8%, 11%->5%) and trims the rest. - regression/highlight threshold raised to 20% (real perf changes move 30%+), gates only with >= 3 samples, and falls back to raw median for baselines that predate the normalized score. validated with deterministic unit checks, including the key case where absolute time doubles but the normalized score stays flat so nothing is flagged. --------- Co-authored-by: Gaurav Sharma <223556219+Copilot@users.noreply.github.com>
### Work Item / Issue Reference > GitHub Issue: #247 ------------------------------------------------------------------- ### Summary adds `.github/copilot-instructions.md`: a from-scratch, repo-wide guide for coding agents working on mssql-python. it covers the architecture (python -> pybind `ddbc_bindings` -> odbc, with py-core/tds for bulk copy), the validated build/test/validation-gate commands, the C++ shutdown/init/arch hazards, the credential-scanning localhost rule, PR conventions, and a trust-and-validate close. it references the existing `.github/prompts/` files instead of duplicating them, and every path plus the CI-enforced title prefixes were checked against the current tree. --------- Co-authored-by: Gaurav Sharma <sharmag@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bewithgaurav
requested changes
Jul 16, 2026
| # Builds the platform-specific `mssql-python-odbc` wheel for ONE libc/arch combo | ||
| # (manylinux_2_28 or musllinux_1_2 x x86_64 or aarch64). | ||
| # | ||
| # Data-only package: no native compilation, no per-Python matrix, no pytest. |
Collaborator
There was a problem hiding this comment.
the odbc wheels ship the driver but nothing runs pytest against them as of now
the pr-validation & daily build-release pipelines resolve the driver from bundled mssql_python/libs, so GetOdbcLibsBaseDir never gets hit in any case actually
GetOdbcLibsBaseDir runs on every connect, but CI only ever hits its fallback branch (bundled libs) since nothing installs the odbc wheel. the external-resolution branch never executes, so a break there ships green.
I think we need a separate pytest step here and have a nightly build routine for mssql-python-odbc as well
we should test installing the built package in a clean env with latest mssql-python build & run the same pytest suite
no new tests or smoketest neded, just the existing suite pointed at the built package
sumitmsft
previously approved these changes
Jul 16, 2026
Adds the build and release pipelines for the standalone mssql-python-odbc package: per-platform build stages (Windows x64/arm64, macOS universal2, Linux manylinux_2_28 + musllinux_1_2 for x86_64/aarch64), a consolidate job that gathers all 7 py3-none wheels, the build orchestration pipeline, and the ESRP-based official release pipeline. The build stages run no compilation (data-only package); Linux builds run in tdslibrs ACR PyPA containers and the musllinux tag is auto-detected via libc_ver. Infra follow-ups (ESRP onboarding of the mssql-python-odbc name, registering the ADO build definition to set odbcBuildDefinitionId, and sharing the ESRP variable group) are marked with TODO(infra) comments.
…nux twine check + container cleanup, reword infra TODOs
…; drop unused targetArch var on Windows stage
Per review feedback, build and release the standalone mssql-python-odbc package from the existing Build-Release-Package-Pipeline (def 2199) and official release pipeline instead of separate ADO definitions. Build pipeline: add odbcWindows/Macos/LinuxConfigs params (2+1+4=7 data-only wheels), reference the 3 odbc stage templates alongside the mssql-python stages, and add a separate ConsolidateOdbc stage publishing drop_ConsolidateOdbc_ConsolidateArtifacts (distinct from mssql-python's drop_Consolidate_ConsolidateArtifacts). Release pipeline: add releasePackage parameter (mssql-python | mssql-python-odbc); select the consolidated artifact accordingly and skip symbol publishing + mssql-py-core version validation for odbc. Delete the redundant standalone orchestrators (build-release-odbc-pipeline.yml, official-release-odbc-pipeline.yml). This removes the need to register a new ADO build definition and the odbcBuildDefinitionId placeholder. Both packages now build from def 2199; the odbc stages consume setup_odbc.py from PR #663, so #663 must merge before these stages can build.
Def 2199 now builds mssql-python and mssql-python-odbc in one run. Both consolidate jobs used DownloadPipelineArtifact@2 with buildType: current, which downloads every artifact in the run and copies all *.whl into dist/. In the merged run this caused: (1) the main Consolidate to sweep the 7 odbc wheels into the mssql-python release (count is warning-only, so silent), and (2) ConsolidateOdbc to see 34 wheels while expecting 7 and hard-fail. Add itemPattern to scope each download to its own package's build-stage artifacts (main: drop_Win_*/drop_MacOS_*/drop_Linux_*; odbc: drop_ODBC_*), giving clean partitioning with no wheel leakage.
…-> 34) The matrix is 9 Windows + 5 macOS + 4 Linux stages x 5 Python versions (cp310-cp314) = 34 wheels across 18 stages. The diagnostic said 27 (7+4+16), stale from an older 3.10-3.13 / 7-Windows matrix. Count check remains warning-only (no behavior change).
official-release-pipeline.yml: gate 'Published symbols' dry-run/summary text to mssql-python so odbc runs no longer claim symbol publishing that never happens; add a mssql-python-only step asserting the wheel metadata pins mssql-python-odbc==18.6.0 (fails by design if released from a pre-pin build). build-odbc-{linux,macos,windows}-stage.yml: assert each built wheel actually contains the ODBC driver binary (msodbcsql*), guarding against the .gitignore '*.so' trap / silent sync_libs() skip shipping an empty data wheel that still passes twine check.
Flip the 34-wheel count check from WARNING-only to a hard exit 1, symmetric with the odbc consolidate job's count guard. A mismatch means a build stage silently produced a partial wheel set (e.g. a missing Python version) or the matrix changed without updating the expected count -- previously this shipped silently.
Mirrors the official-release-pipeline odbc changes: releasePackage parameter, consolidatedArtifactName selection for both artifact downloads, and gating the mssql-py-core version validation + symbol publishing to mssql-python only. Stays on Maven ContentType so it never publishes to PyPI.
Adds the mssql-python-odbc pin guard (gated to mssql-python, matching official-release-pipeline) and gates the 'symbols published' summary messaging on releasePackage==mssql-python so the dummy pipeline exercises the exact same mssql-python validation/messaging as the official pipeline for both packages.
Gate mssql-python and mssql-python-odbc stage groups behind a buildPackage parameter (mssql-python default) so only the selected package builds, halving run time and enabling concurrent per-package builds. Mirrors the releasePackage selector pattern in the release pipelines.
Add effectiveBuildPackage variable that forces 'both' on scheduled (Build.Reason=Schedule) runs while manual/PR runs honour the buildPackage parameter. Mirrors the effectiveOneBranchType pattern so the nightly keeps refreshing every artifact.
… ADO design Add hardened wheel-installation-test-stage.yml (positive+negative driver proof, source-tree shadowing guards, TESTED==0 fails stage) and wire two 'both'-gated TestBothWheels_* stages (manylinux_2_28 + musllinux x86_64) after ConsolidateOdbc, replacing the old single 3.12 invocation.
…crosoft/mssql-python into jahnvi/odbc-release-pipelines # Conflicts: # OneBranchPipelines/build-release-package-pipeline.yml
| echo "Waiting for SQL Server to be ready..." | ||
| for i in {1..30}; do | ||
| if docker exec sqlserver-$(LINUX_TAG)-$(ARCH) /opt/mssql-tools18/bin/sqlcmd \ | ||
| -S localhost -U SA -P "$(DB_PASSWORD)" -C -Q "SELECT 1" >/dev/null 2>&1; then |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
Adds the ADO / OneBranch release pipelines for the standalone
mssql-python-odbcpackage. This is the CI/CD half of the ODBC driver split; the package code and build script are in companion PR #663.Files
OneBranchPipelines/build-release-odbc-pipeline.yml- build orchestration. Fans out to 7 platform build stages plus a consolidate stage, with path-filtered triggers scoped tosetup_odbc.py,mssql_python_odbc/,mssql_python/libs/, and these pipeline files. SDL keeps credscan / policheck / sbom / tsa and disables binskim / apiscan / armory / codeql (data-only package, no first-party compiled binaries) with justifications.OneBranchPipelines/stages/build-odbc-windows-stage.yml- Windows amd64 / arm64.OneBranchPipelines/stages/build-odbc-macos-stage.yml- macOS universal2.OneBranchPipelines/stages/build-odbc-linux-stage.yml- manylinux_2_28 and musllinux_1_2 (x86_64 / aarch64) built in tdslibrs ACR PyPA containers. No compilation happens here; the musllinux tag is auto-detected insetup_odbc.pyviaplatform.libc_ver(), so the hardcodedMANYLINUX_TAGonly affects glibc builds.OneBranchPipelines/jobs/consolidate-odbc-artifacts-job.yml- gathers all wheels and verifies the count is exactly 7.OneBranchPipelines/official-release-odbc-pipeline.yml- ESRP-based PyPI release, gated byreleaseToPyPI. TheEsrpRelease@9step is identical to the existing mssql-python release pipeline. A preflight step fails with a clear message ifodbcBuildDefinitionIdis still the0placeholder, so the artifact download can't fail cryptically.Wheels produced (7, py3-none-)
win_amd64,win_arm64,macosx_15_0_universal2,manylinux_2_28_x86_64,manylinux_2_28_aarch64,musllinux_1_2_x86_64,musllinux_1_2_aarch64.Infra follow-ups (marked "Infra setup:" in the YAML)
mssql-python-odbcpackage name to ESRP for PyPI release.odbcBuildDefinitionIdinofficial-release-odbc-pipeline.yml(currently placeholder0; the release pipeline now preflight-checks this and errors clearly if left at0), and confirmpipelines.sourcematches the registered definition name.ESRP Federated Creds (AME)variable group.Notes
These pipelines are inert until they are registered as ADO build definitions; merging the YAML alone runs nothing. Pairs with #663.