From 89f2787f0cd13f3cce5dee9bf6d0b3a8b7ead79e Mon Sep 17 00:00:00 2001 From: David Levy Date: Sat, 7 Feb 2026 23:17:20 -0600 Subject: [PATCH 1/2] DOC: Add llms.txt and copilot-instructions.md for LLM guidance Add two documentation files to help AI coding assistants work effectively with the mssql-python driver: llms.txt (user-facing, 256 lines): - Installation, Quick Start, connection string patterns - CRUD operations, parameterized queries, transactions - Stored procedures with OUTPUT param workaround - Error handling with full exception hierarchy - Connection pooling, pyodbc comparison and migration - Platform support matrix (SUSE x64-only noted) .github/copilot-instructions.md (contributor-facing, 263 lines): - Cross-references to .github/prompts/ workflow files - Build system instructions (Windows/macOS/Linux) - Project architecture with complete file tree - CI/CD pipeline details (5 GitHub workflows, ADO pipelines) - Anti-patterns, debugging quick reference - PR requirements and contributing guidelines --- .github/copilot-instructions.md | 274 ++++++++++++++++++++++++++++++++ llms.txt | 256 +++++++++++++++++++++++++++++ 2 files changed, 530 insertions(+) create mode 100644 .github/copilot-instructions.md create mode 100644 llms.txt diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 000000000..f104a0ca6 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,274 @@ +# Copilot Instructions for mssql-python + +## Repository Overview + +**mssql-python** is a Python driver for Microsoft SQL Server and Azure SQL databases that leverages Direct Database Connectivity (DDBC). It's built using **pybind11** and **CMake** to create native extensions, providing DB API 2.0 compliant database access with enhanced Pythonic features. + +- **Size**: Medium-scale project (~750KB total) +- **Languages**: Python (main), C++ (native bindings), CMake (build system) +- **Target Platforms**: Windows (x64, ARM64), macOS (Universal2), Linux (x86_64, ARM64) +- **Python Versions**: 3.10+ +- **Key Dependencies**: pybind11, azure-identity, Microsoft ODBC Driver 18 + +## Development Workflows + +This repository includes detailed prompt files for common tasks. Reference these with `#`: + +| Task | Prompt | When to Use | +|------|--------|-------------| +| First-time setup | `#setup-dev-env` | New machine, fresh clone | +| Build C++ extension | `#build-ddbc` | After modifying .cpp/.h files | +| Run tests | `#run-tests` | Validating changes | +| Create PR | `#create-pr` | Ready to submit changes | + +**Workflow order for new contributors:** +1. `#setup-dev-env` — Set up venv and dependencies +2. `#build-ddbc` — Build native extension +3. Make your changes +4. `#run-tests` — Validate +5. `#create-pr` — Submit + +## Build System and Validation + +### Prerequisites +**Always install these before building:** +```bash +# All platforms +pip install -r requirements.txt + +# Windows: Requires Visual Studio Build Tools with "Desktop development with C++" workload +# macOS: brew install cmake && brew install msodbcsql18 +# Linux: Install cmake, python3-dev, and ODBC driver per distribution +``` + +### Building the Project + +**CRITICAL**: The project requires building native extensions before testing. Extensions are platform-specific (`.pyd` on Windows, `.so` on macOS/Linux). + +#### Windows Build: +```bash +cd mssql_python/pybind +build.bat [x64|x86|arm64] # Defaults to x64 if not specified +``` + +#### macOS Build: +```bash +cd mssql_python/pybind +./build.sh # Creates universal2 binary (ARM64 + x86_64) +``` + +#### Linux Build: +```bash +cd mssql_python/pybind +./build.sh # Detects architecture automatically +``` + +**Build Output**: Creates `ddbc_bindings.cp{python_version}-{architecture}.{so|pyd}` in the `mssql_python/` directory. + +### Testing + +**IMPORTANT**: Tests require a SQL Server connection via `DB_CONNECTION_STRING` environment variable. + +```bash +# Run all tests with coverage +python -m pytest -v --cov=. --cov-report=xml --capture=tee-sys --cache-clear + +# Run specific test files +python -m pytest tests/test_000_dependencies.py -v # Dependency checks +python -m pytest tests/test_001_globals.py -v # Basic functionality +``` + +**Test Dependencies**: Tests require building the native extension first. The dependency test (`test_000_dependencies.py`) validates that all platform-specific libraries exist. + +### Linting and Code Quality + +```bash +# Python formatting +black --check --line-length=100 mssql_python/ tests/ + +# C++ formatting +clang-format -style=file -i mssql_python/pybind/*.cpp mssql_python/pybind/*.h + +# Coverage reporting (configured in .coveragerc) +python -m pytest --cov=. --cov-report=html +``` + +## Project Architecture + +### Core Components + +``` +mssql_python/ +├── __init__.py # Package initialization, connection registry, cleanup +├── connection.py # DB API 2.0 connection object +├── cursor.py # DB API 2.0 cursor object +├── db_connection.py # connect() function implementation +├── auth.py # Microsoft Entra ID authentication +├── pooling.py # Connection pooling implementation +├── logging.py # Logging configuration +├── exceptions.py # Exception hierarchy +├── connection_string_builder.py # Connection string construction +├── connection_string_parser.py # Connection string parsing +├── parameter_helper.py # Query parameter handling +├── row.py # Row object implementation +├── type.py # DB API 2.0 type objects +├── constants.py # ODBC constants +├── helpers.py # Utility functions and settings +├── ddbc_bindings.py # Platform-specific extension loader with architecture detection +├── mssql_python.pyi # Type stubs for IDE support +├── py.typed # PEP 561 type marker +└── pybind/ # Native extension source + ├── ddbc_bindings.cpp # Main C++ binding code + ├── ddbc_bindings.h # Header for bindings + ├── CMakeLists.txt # Cross-platform build configuration + ├── build.sh/.bat # Platform-specific build scripts + ├── configure_dylibs.sh # macOS dylib configuration + ├── logger_bridge.cpp/.hpp # Python logging bridge + ├── unix_utils.cpp/.h # Unix platform utilities + └── connection/ # Connection management + ├── connection.cpp/.h # Connection implementation + └── connection_pool.cpp/.h # Connection pooling +``` + +### Platform-Specific Libraries + +``` +mssql_python/libs/ +├── windows/{x64,x86,arm64}/ # Windows ODBC drivers and dependencies +├── macos/{arm64,x86_64}/lib/ # macOS dylibs +└── linux/{debian_ubuntu,rhel,suse,alpine}/{x86_64,arm64}/lib/ # Linux distributions +``` + +### Configuration Files + +- **`.clang-format`**: C++ formatting (LLVM style, 100 column limit) +- **`.coveragerc`**: Coverage configuration +- **`requirements.txt`**: Development dependencies +- **`setup.py`**: Package configuration with platform detection +- **`pyproject.toml`**: Modern Python packaging configuration +- **`.gitignore`**: Excludes build artifacts (*.so, *.pyd, build/, __pycache__) + +## CI/CD Pipeline Details + +### GitHub Workflows +- **`devskim.yml`**: Security scanning (runs on PRs and main) +- **`pr-format-check.yml`**: PR validation (title format, GitHub issue/ADO work item links) +- **`lint-check.yml`**: Python (Black) and C++ (clang-format) linting +- **`pr-code-coverage.yml`**: Code coverage reporting +- **`forked-pr-coverage.yml`**: Coverage for forked PRs + +### Azure DevOps Pipelines (`eng/pipelines/`) +- **`pr-validation-pipeline.yml`**: Comprehensive testing across all platforms +- **`build-whl-pipeline.yml`**: Wheel building for distribution +- **Platform Coverage**: Windows (LocalDB), macOS (Docker SQL Server), Linux (Ubuntu, Debian, RHEL, Alpine) with both x86_64 and ARM64 + +### Build Matrix +The CI system tests: +- **Python versions**: 3.10, 3.11, 3.12, 3.13 +- **Windows**: x64, ARM64 architectures +- **macOS**: Universal2 (ARM64 + x86_64) +- **Linux**: Multiple distributions (Debian, Ubuntu, RHEL, Alpine) on x86_64 and ARM64 + +## Common Build Issues and Workarounds + +### macOS-Specific Issues +- **dylib path configuration**: Run `configure_dylibs.sh` after building to fix library paths +- **codesigning**: Script automatically codesigns libraries for compatibility + +### Linux Distribution Differences +- **Debian/Ubuntu**: Use `apt-get install python3-dev cmake pybind11-dev` +- **RHEL**: Requires enabling CodeReady Builder repository for development tools +- **Alpine**: Uses musl libc, requires special handling in build scripts + +### Windows Build Dependencies +- **Visual Studio Build Tools**: Must include "Desktop development with C++" workload +- **Architecture Detection**: Build scripts auto-detect target architecture from environment + +### Known Limitations (from TODOs) +- Linux RPATH configuration pending for driver .so files +- Some Unicode support gaps in executemany operations +- Platform-specific test dependencies in exception handling + +## Architecture Detection and Loading + +The `ddbc_bindings.py` module handles architecture detection: +- **Windows**: Normalizes `win64/amd64/x64` → `x64`, `win32/x86` → `x86`, `arm64` → `arm64` +- **macOS**: Runtime architecture detection, always loads from universal2 binary +- **Linux**: Maps `x64/amd64` → `x86_64`, `arm64/aarch64` → `arm64` + +## Exception Hierarchy + +Critical for error handling guidance: + +``` +Exception (base) +├── Warning +└── Error + ├── InterfaceError # Driver/interface issues + └── DatabaseError + ├── DataError # Invalid data processing + ├── OperationalError # Connection/timeout issues + ├── IntegrityError # Constraint violations + ├── InternalError # Internal driver/database errors + ├── ProgrammingError # SQL syntax errors + └── NotSupportedError # Unsupported features/operations +``` + +## Critical Anti-Patterns (DO NOT) + +- **NEVER** hardcode connection strings - always use `DB_CONNECTION_STRING` env var for tests +- **NEVER** use `pyodbc` imports - this driver doesn't require external ODBC +- **NEVER** modify files in `mssql_python/libs/` - these are pre-built binaries +- **NEVER** skip `conn.commit()` after INSERT/UPDATE/DELETE operations +- **NEVER** use bare `except:` blocks - always catch specific exceptions +- **NEVER** leave connections open - use context managers or explicit `close()` + +## When Modifying Code + +### Python Changes +- Preserve existing error handling patterns from `exceptions.py` +- Use context managers (`with`) for all connection/cursor operations +- Update `__all__` exports if adding public API +- Add corresponding test in `tests/test_*.py` +- Follow Black formatting (line length 100) + +### C++ Changes +- Follow RAII patterns for resource management +- Use `py::gil_scoped_release` for blocking ODBC operations +- Update `mssql_python.pyi` type stubs if changing Python API +- Follow `.clang-format` style (LLVM style, 100 column limit) + +## Code Quality + +- **Keep it tight**: Minimal code to solve the problem. No duplicate logic, no redundant validation. +- **Comments explain why, not what**: Don't restate what the code does. Comment intent, edge cases, and non-obvious decisions. +- **One-line docstrings**: For test functions and simple helpers. No "Validates:" bullet lists or "Note:" paragraphs. +- **Docstring examples must match the API**: If the signature changes, update the examples. +- **Catch specific exceptions**: Use `DatabaseError`, `IntegrityError`, etc. — never `except Exception` or bare `except:`. +- **Let the test framework work**: Don't wrap test bodies in `try/except: pytest.fail()`. Let pytest show the real traceback. +- **Assertions must match claims**: If a test says "all types", check all of them. Cover the case that motivated the fix. +- **No stale references**: If you reference a file, function, or prompt — verify it exists first. + +## Debugging Quick Reference + +| Error | Cause | Solution | +|-------|-------|----------| +| `ImportError: ddbc_bindings` | Extension not built | Run `#build-ddbc` | +| Connection timeout | Missing env var | Set `DB_CONNECTION_STRING` | +| `dylib not found` (macOS) | Library paths | Run `configure_dylibs.sh` | +| `ODBC Driver not found` | Missing driver | Install Microsoft ODBC Driver 18 | +| `ModuleNotFoundError` | Not in venv | Run `#setup-dev-env` | + +## Contributing Guidelines + +### PR Requirements +- **Title Format**: Must start with `FEAT:`, `CHORE:`, `FIX:`, `DOC:`, `STYLE:`, `REFACTOR:`, or `RELEASE:` +- **Issue Linking**: Must link to either GitHub issue or ADO work item +- **Summary**: Minimum 10 characters of meaningful content under "### Summary" + +### Development Workflow +1. **Always build native extensions first** before running tests +2. **Use virtual environments** for dependency isolation +3. **Test on target platform** before submitting PRs +4. **Check CI pipeline results** for cross-platform compatibility + diff --git a/llms.txt b/llms.txt new file mode 100644 index 000000000..cf2e0b058 --- /dev/null +++ b/llms.txt @@ -0,0 +1,256 @@ +# mssql-python + +mssql-python is the official Microsoft Python driver for SQL Server and Azure SQL databases. It provides DB API 2.0 compliant database access with Direct Database Connectivity (DDBC) - no external ODBC driver manager required. + +## Installation + +```bash +pip install mssql-python +``` + +Or using uv (recommended for faster installs): +```bash +uv pip install mssql-python +``` + +## Quick Start + +> **Security Note**: The examples below use `TrustServerCertificate=yes` for local development with self-signed certificates. For production or remote connections, remove this option to ensure proper TLS certificate validation. + +```python +import mssql_python + +conn = mssql_python.connect( + "SERVER=localhost;DATABASE=mydb;Trusted_Connection=yes;Encrypt=yes;TrustServerCertificate=yes" +) +cursor = conn.cursor() +cursor.execute("SELECT * FROM users WHERE id = ?", (42,)) +rows = cursor.fetchall() +cursor.close() +conn.close() +``` + +## Connection String Formats + +### Windows Authentication (Trusted Connection) +```python +conn = mssql_python.connect( + "SERVER=localhost;DATABASE=mydb;Trusted_Connection=yes;Encrypt=yes;TrustServerCertificate=yes" +) +``` + +### SQL Server Authentication +```python +conn = mssql_python.connect( + "SERVER=localhost;DATABASE=mydb;UID=myuser;PWD=mypassword;Encrypt=yes;TrustServerCertificate=yes" +) +``` + +### Azure SQL with Entra ID (Interactive) +```python +conn = mssql_python.connect( + "SERVER=myserver.database.windows.net;DATABASE=mydb;Authentication=ActiveDirectoryInteractive;Encrypt=yes" +) +``` + +### Azure SQL with Managed Identity +```python +conn = mssql_python.connect( + "SERVER=myserver.database.windows.net;DATABASE=mydb;Authentication=ActiveDirectoryMSI;Encrypt=yes" +) +``` + +### Azure SQL with Service Principal +```python +conn = mssql_python.connect( + "SERVER=myserver.database.windows.net;DATABASE=mydb;" + "Authentication=ActiveDirectoryServicePrincipal;" + "UID=;PWD=;Encrypt=yes" +) +``` + +## Context Manager Usage (Recommended) + +```python +import mssql_python + +with mssql_python.connect(connection_string) as conn: + with conn.cursor() as cursor: + cursor.execute("SELECT @@VERSION") + version = cursor.fetchone() + print(version[0]) +``` + +## Query Patterns + +### Parameterized Queries (Prevent SQL Injection) +```python +cursor.execute("SELECT * FROM users WHERE email = ? AND active = ?", (email, True)) + +# Named parameters (pyformat style) +cursor.execute("SELECT * FROM users WHERE email = %(email)s", {"email": email}) +``` + +### Fetch Results +```python +row = cursor.fetchone() +rows = cursor.fetchmany(size=100) +all_rows = cursor.fetchall() +``` + +### Insert Data +```python +cursor.execute( + "INSERT INTO users (name, email) VALUES (?, ?)", + ("John Doe", "john@example.com") +) +conn.commit() # Don't forget to commit! +``` + +### Insert Multiple Rows (executemany) +```python +users = [ + ("Alice", "alice@example.com"), + ("Bob", "bob@example.com"), + ("Charlie", "charlie@example.com"), +] +cursor.executemany("INSERT INTO users (name, email) VALUES (?, ?)", users) +conn.commit() +``` + +### Update Data +```python +cursor.execute("UPDATE users SET active = ? WHERE id = ?", (False, user_id)) +conn.commit() +print(f"Rows affected: {cursor.rowcount}") +``` + +### Delete Data +```python +cursor.execute("DELETE FROM users WHERE id = ?", (user_id,)) +conn.commit() +``` + +## Transaction Management + +```python +from mssql_python import DatabaseError + +try: + cursor.execute("INSERT INTO orders (customer_id, total) VALUES (?, ?)", (1, 99.99)) + cursor.execute("UPDATE inventory SET quantity = quantity - 1 WHERE product_id = ?", (42,)) + conn.commit() # Commit both changes +except DatabaseError: + conn.rollback() # Rollback on error + raise +``` + +### Autocommit Mode +```python +conn = mssql_python.connect(connection_string) +conn.autocommit = True # Each statement commits immediately +``` + +## Working with Results + +### Get Column Metadata +```python +cursor.execute("SELECT id, name, email FROM users") +for column in cursor.description: + print(f"Column: {column[0]}, Type: {column[1]}") +``` + +## Stored Procedures + +```python +# Call a stored procedure +cursor.execute("EXEC GetUserById ?", (user_id,)) +result = cursor.fetchone() + +# OUTPUT parameters: use DECLARE/EXEC/SELECT pattern +# (callproc() is not supported; use execute() with anonymous code blocks) +cursor.execute(""" + DECLARE @count INT; + EXEC CountActiveUsers @count OUTPUT; + SELECT @count; +""") +count = cursor.fetchone()[0] +``` + +## Error Handling + +```python +from mssql_python import ( + DatabaseError, + IntegrityError, + ProgrammingError, + OperationalError, +) + +try: + cursor.execute("INSERT INTO users (id, name) VALUES (?, ?)", (1, "John")) + conn.commit() +except IntegrityError as e: + # Constraint violation (duplicate key, foreign key, etc.) + print(f"Constraint error: {e}") +except ProgrammingError as e: + # SQL syntax error or invalid query + print(f"SQL error: {e}") +except OperationalError as e: + # Connection issues, timeouts, etc. + print(f"Operational error: {e}") +except DatabaseError as e: + # General database error + print(f"Database error: {e}") +``` + +## Connection Pooling + +Connection pooling is enabled by default: + +```python +# Pooling is automatic - connections are reused +conn1 = mssql_python.connect(connection_string) +conn1.close() # Returns to pool + +conn2 = mssql_python.connect(connection_string) # Reuses pooled connection +``` + +## Comparison to pyodbc + +| Feature | mssql-python | pyodbc | +|---------|-------------|--------| +| Driver | Ships bundled | Requires separate ODBC driver | +| Installation | `pip install` only | pip install + ODBC driver setup | +| Azure Entra ID | Native Python API | Via ODBC driver connection keywords | +| Connection Pooling | Built-in | Via ODBC Driver Manager | +| DB API 2.0 | ✅ Compliant | ✅ Compliant | + +### Migration from pyodbc +```python +# pyodbc +import pyodbc +conn = pyodbc.connect("DRIVER={ODBC Driver 18 for SQL Server};SERVER=...;DATABASE=...") + +# mssql-python (no DRIVER needed) +import mssql_python +conn = mssql_python.connect("SERVER=...;DATABASE=...") +``` + +## Supported Platforms + +- Windows (x64, ARM64) +- macOS (ARM64, x86_64) +- Linux (x86_64, ARM64): Debian, Ubuntu, RHEL, Alpine (musl) +- Linux (x86_64 only): SUSE + +## Supported Python Versions + +- Python 3.10+ + +## Links + +- Documentation: https://github.com/microsoft/mssql-python/wiki +- PyPI: https://pypi.org/project/mssql-python/ +- GitHub: https://github.com/microsoft/mssql-python +- Issues: https://github.com/microsoft/mssql-python/issues From 7d5009e375f691c7b9a5941472e0ecf87275cac8 Mon Sep 17 00:00:00 2001 From: Gaurav Sharma Date: Wed, 15 Jul 2026 17:58:32 +0530 Subject: [PATCH 2/2] DOC: rewrite copilot-instructions.md from repo evidence, drop llms.txt rewrites .github/copilot-instructions.md from scratch against the current repo: the python -> pybind ddbc_bindings -> odbc call stack (py-core/TDS for bulkcopy), the validated build/test/validation-gate commands, the C++ shutdown and static-handle and error-code hazards, the credential-scanning localhost rule, PR conventions, and a trust-and-validate close. every path and the CI-enforced title prefixes were checked against the tree. drops llms.txt: it's a website-root convention, isn't shipped in the wheel (not in package_data) and nothing discovers it from a repo root, so it's inert here. the contributor instructions file is the surface that's actually consumed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 306 +++++++------------------------- llms.txt | 256 -------------------------- 2 files changed, 63 insertions(+), 499 deletions(-) delete mode 100644 llms.txt diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index f104a0ca6..60b9a4dba 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,274 +1,94 @@ -# Copilot Instructions for mssql-python +# mssql-python — Copilot instructions -## Repository Overview +mssql-python is Microsoft's pip-installable Python driver for SQL Server, Azure SQL, and Azure Synapse. It is DB API 2.0 (PEP 249) compliant. The core is a C++ pybind11 native extension (`ddbc_bindings`) built with CMake and wrapped by a pure-Python package; bulk copy is backed by a separate Rust/TDS core (`mssql_py_core`). Platform-specific wheels ship for Windows (x64, ARM64), macOS (x64, ARM64), and Linux (x64, ARM64). Python 3.10+. The ODBC driver is bundled in the wheel — no external driver manager is required. -**mssql-python** is a Python driver for Microsoft SQL Server and Azure SQL databases that leverages Direct Database Connectivity (DDBC). It's built using **pybind11** and **CMake** to create native extensions, providing DB API 2.0 compliant database access with enhanced Pythonic features. +## Architecture -- **Size**: Medium-scale project (~750KB total) -- **Languages**: Python (main), C++ (native bindings), CMake (build system) -- **Target Platforms**: Windows (x64, ARM64), macOS (Universal2), Linux (x86_64, ARM64) -- **Python Versions**: 3.10+ -- **Key Dependencies**: pybind11, azure-identity, Microsoft ODBC Driver 18 +Call stack, top to bottom: -## Development Workflows +1. **Pure-Python API** — `mssql_python/` (`connection.py`, `cursor.py`, `row.py`, `pooling.py`, `auth.py`, `exceptions.py`, type coercion). This is the DB API 2.0 surface. +2. **Extension loader** — `mssql_python/ddbc_bindings.py` detects platform/architecture and loads the correct native binary. +3. **pybind11 binding layer** — `mssql_python/pybind/` (`ddbc_bindings.cpp`/`.h`, `connection/`, `logger_bridge.*`, `unix_utils.*`), compiled to `ddbc_bindings.cp{ver}-{arch}.{so|pyd}` in `mssql_python/`. +4. **ODBC driver stack → SQL Server.** Bulk copy takes a different path: it uses `mssql_py_core` (TDS) directly, not ODBC. -This repository includes detailed prompt files for common tasks. Reference these with `#`: +Paths you will touch: -| Task | Prompt | When to Use | -|------|--------|-------------| -| First-time setup | `#setup-dev-env` | New machine, fresh clone | -| Build C++ extension | `#build-ddbc` | After modifying .cpp/.h files | -| Run tests | `#run-tests` | Validating changes | -| Create PR | `#create-pr` | Ready to submit changes | +- `mssql_python/pybind/CMakeLists.txt` — all platform/architecture build conditionals live here. +- `mssql_python/pybind/build.sh` / `build.bat` — build entry points; `configure_dylibs.sh` fixes macOS dylib paths. +- `mssql_python/libs/{windows,macos,linux}/...` — prebuilt ODBC binaries. **NEVER hand-edit these.** +- `setup.py` / `pyproject.toml` — packaging and wheel/platform tagging. +- `mssql_python/mssql_python.pyi` + `mssql_python/py.typed` — PEP 561 type stubs. Update stubs when the public API changes. +- `tests/` — mostly-numbered `test_NNN_*.py` files (a live-SQL-Server integration suite plus no-DB dependency checks). -**Workflow order for new contributors:** -1. `#setup-dev-env` — Set up venv and dependencies -2. `#build-ddbc` — Build native extension -3. Make your changes -4. `#run-tests` — Validate -5. `#create-pr` — Submit +## Development workflow -## Build System and Validation +Validated, step-by-step guides live in `.github/prompts/` — use them instead of rediscovering commands: -### Prerequisites -**Always install these before building:** -```bash -# All platforms -pip install -r requirements.txt - -# Windows: Requires Visual Studio Build Tools with "Desktop development with C++" workload -# macOS: brew install cmake && brew install msodbcsql18 -# Linux: Install cmake, python3-dev, and ODBC driver per distribution -``` +- `setup-dev-env.prompt.md` — venv, dependencies, ODBC headers, `DB_CONNECTION_STRING`, SQL Server. +- `build-ddbc.prompt.md` — build the C++ extension. +- `run-tests.prompt.md` — pytest categories and markers. +- `create-pr.prompt.md` — the PR flow (title / issue-link / summary confirmations). -### Building the Project +Core facts: -**CRITICAL**: The project requires building native extensions before testing. Extensions are platform-specific (`.pyd` on Windows, `.so` on macOS/Linux). - -#### Windows Build: -```bash -cd mssql_python/pybind -build.bat [x64|x86|arm64] # Defaults to x64 if not specified -``` +- **ALWAYS build the native extension before running Python tests.** Importing `mssql_python` needs the compiled `.so`/`.pyd`. Build with `cd mssql_python/pybind && ./build.sh` (or `build.bat` on Windows; pass `x64|arm64` as needed). +- **Tests need a live SQL Server** reachable through the `DB_CONNECTION_STRING` env var. `tests/test_000_dependencies.py` runs with no DB; most others require one. +- macOS wheels are universal2 (arm64 and x86_64); when touching dylib/rpath setup (`configure_dylibs.sh`), make sure both slices are handled, not just the build host. -#### macOS Build: -```bash -cd mssql_python/pybind -./build.sh # Creates universal2 binary (ARM64 + x86_64) -``` +## Validation gate (run before you finish — this mirrors CI) -#### Linux Build: ```bash -cd mssql_python/pybind -./build.sh # Detects architecture automatically -``` - -**Build Output**: Creates `ddbc_bindings.cp{python_version}-{architecture}.{so|pyd}` in the `mssql_python/` directory. - -### Testing - -**IMPORTANT**: Tests require a SQL Server connection via `DB_CONNECTION_STRING` environment variable. - -```bash -# Run all tests with coverage -python -m pytest -v --cov=. --cov-report=xml --capture=tee-sys --cache-clear - -# Run specific test files -python -m pytest tests/test_000_dependencies.py -v # Dependency checks -python -m pytest tests/test_001_globals.py -v # Basic functionality -``` - -**Test Dependencies**: Tests require building the native extension first. The dependency test (`test_000_dependencies.py`) validates that all platform-specific libraries exist. - -### Linting and Code Quality - -```bash -# Python formatting -black --check --line-length=100 mssql_python/ tests/ - -# C++ formatting -clang-format -style=file -i mssql_python/pybind/*.cpp mssql_python/pybind/*.h - -# Coverage reporting (configured in .coveragerc) -python -m pytest --cov=. --cov-report=html -``` - -## Project Architecture - -### Core Components - -``` -mssql_python/ -├── __init__.py # Package initialization, connection registry, cleanup -├── connection.py # DB API 2.0 connection object -├── cursor.py # DB API 2.0 cursor object -├── db_connection.py # connect() function implementation -├── auth.py # Microsoft Entra ID authentication -├── pooling.py # Connection pooling implementation -├── logging.py # Logging configuration -├── exceptions.py # Exception hierarchy -├── connection_string_builder.py # Connection string construction -├── connection_string_parser.py # Connection string parsing -├── parameter_helper.py # Query parameter handling -├── row.py # Row object implementation -├── type.py # DB API 2.0 type objects -├── constants.py # ODBC constants -├── helpers.py # Utility functions and settings -├── ddbc_bindings.py # Platform-specific extension loader with architecture detection -├── mssql_python.pyi # Type stubs for IDE support -├── py.typed # PEP 561 type marker -└── pybind/ # Native extension source - ├── ddbc_bindings.cpp # Main C++ binding code - ├── ddbc_bindings.h # Header for bindings - ├── CMakeLists.txt # Cross-platform build configuration - ├── build.sh/.bat # Platform-specific build scripts - ├── configure_dylibs.sh # macOS dylib configuration - ├── logger_bridge.cpp/.hpp # Python logging bridge - ├── unix_utils.cpp/.h # Unix platform utilities - └── connection/ # Connection management - ├── connection.cpp/.h # Connection implementation - └── connection_pool.cpp/.h # Connection pooling -``` - -### Platform-Specific Libraries - -``` -mssql_python/libs/ -├── windows/{x64,x86,arm64}/ # Windows ODBC drivers and dependencies -├── macos/{arm64,x86_64}/lib/ # macOS dylibs -└── linux/{debian_ubuntu,rhel,suse,alpine}/{x86_64,arm64}/lib/ # Linux distributions +black --check --line-length=100 mssql_python/ tests/ # BLOCKING in CI +python -m pytest -v # 'stress' marker excluded by default ``` -### Configuration Files - -- **`.clang-format`**: C++ formatting (LLVM style, 100 column limit) -- **`.coveragerc`**: Coverage configuration -- **`requirements.txt`**: Development dependencies -- **`setup.py`**: Package configuration with platform detection -- **`pyproject.toml`**: Modern Python packaging configuration -- **`.gitignore`**: Excludes build artifacts (*.so, *.pyd, build/, __pycache__) - -## CI/CD Pipeline Details - -### GitHub Workflows -- **`devskim.yml`**: Security scanning (runs on PRs and main) -- **`pr-format-check.yml`**: PR validation (title format, GitHub issue/ADO work item links) -- **`lint-check.yml`**: Python (Black) and C++ (clang-format) linting -- **`pr-code-coverage.yml`**: Code coverage reporting -- **`forked-pr-coverage.yml`**: Coverage for forked PRs - -### Azure DevOps Pipelines (`eng/pipelines/`) -- **`pr-validation-pipeline.yml`**: Comprehensive testing across all platforms -- **`build-whl-pipeline.yml`**: Wheel building for distribution -- **Platform Coverage**: Windows (LocalDB), macOS (Docker SQL Server), Linux (Ubuntu, Debian, RHEL, Alpine) with both x86_64 and ARM64 +- **`pr-format-check` (BLOCKING):** PR title must start with one of `FEAT: FIX: DOC: CHORE: STYLE: REFACTOR: RELEASE:`; the body must link a work item/issue and have a `### Summary` of at least 10 characters. +- `flake8`, `pylint`, `mypy`, `clang-format`, and `cpplint` run but are **informational**, not blocking. +- The authoritative cross-platform validation runs on **Azure DevOps** (broader OS / Python / arch coverage than the GitHub checks); consult the specific pipeline in `eng/pipelines/` for the exact matrix rather than assuming full coverage. A coverage bot posts a report comment on the PR. -### Build Matrix -The CI system tests: -- **Python versions**: 3.10, 3.11, 3.12, 3.13 -- **Windows**: x64, ARM64 architectures -- **macOS**: Universal2 (ARM64 + x86_64) -- **Linux**: Multiple distributions (Debian, Ubuntu, RHEL, Alpine) on x86_64 and ARM64 +## Code standards -## Common Build Issues and Workarounds - -### macOS-Specific Issues -- **dylib path configuration**: Run `configure_dylibs.sh` after building to fix library paths -- **codesigning**: Script automatically codesigns libraries for compatibility - -### Linux Distribution Differences -- **Debian/Ubuntu**: Use `apt-get install python3-dev cmake pybind11-dev` -- **RHEL**: Requires enabling CodeReady Builder repository for development tools -- **Alpine**: Uses musl libc, requires special handling in build scripts - -### Windows Build Dependencies -- **Visual Studio Build Tools**: Must include "Desktop development with C++" workload -- **Architecture Detection**: Build scripts auto-detect target architecture from environment - -### Known Limitations (from TODOs) -- Linux RPATH configuration pending for driver .so files -- Some Unicode support gaps in executemany operations -- Platform-specific test dependencies in exception handling - -## Architecture Detection and Loading - -The `ddbc_bindings.py` module handles architecture detection: -- **Windows**: Normalizes `win64/amd64/x64` → `x64`, `win32/x86` → `x86`, `arm64` → `arm64` -- **macOS**: Runtime architecture detection, always loads from universal2 binary -- **Linux**: Maps `x64/amd64` → `x86_64`, `arm64/aarch64` → `arm64` - -## Exception Hierarchy - -Critical for error handling guidance: - -``` -Exception (base) -├── Warning -└── Error - ├── InterfaceError # Driver/interface issues - └── DatabaseError - ├── DataError # Invalid data processing - ├── OperationalError # Connection/timeout issues - ├── IntegrityError # Constraint violations - ├── InternalError # Internal driver/database errors - ├── ProgrammingError # SQL syntax errors - └── NotSupportedError # Unsupported features/operations -``` +**Python** -## Critical Anti-Patterns (DO NOT) +- Black, line length 100. DB API 2.0 semantics take priority. +- Catch specific exceptions (`DatabaseError`, `IntegrityError`, `OperationalError`, …), **never a bare `except:`** — flake8 ignores E722, but the team enforces this in review. +- Use context managers for connections and cursors. Update `__all__` **and** `mssql_python/mssql_python.pyi` when changing the public API. Add a `tests/test_*.py` case for every fix. -- **NEVER** hardcode connection strings - always use `DB_CONNECTION_STRING` env var for tests -- **NEVER** use `pyodbc` imports - this driver doesn't require external ODBC -- **NEVER** modify files in `mssql_python/libs/` - these are pre-built binaries -- **NEVER** skip `conn.commit()` after INSERT/UPDATE/DELETE operations -- **NEVER** use bare `except:` blocks - always catch specific exceptions -- **NEVER** leave connections open - use context managers or explicit `close()` +**C++ / pybind11** (`mssql_python/pybind/`) — hazards to respect: -## When Modifying Code +- **Process-shutdown ordering for Python handles.** Destructors of static `py::object` caches (e.g. the datetime/decimal class cache in `ddbc_bindings.cpp`) run after Python finalizes; an ODBC or threading call from a destructor after the GIL is gone can deadlock or crash the whole test suite at exit. Guard shutdown cleanup with `Py_IsInitialized()`, register cleanup via `atexit`, and prefer not to add new static Python handles. +- **Don't let a failed initialization escape as a half-built object.** `Connection::setAttribute` returns `SQLRETURN`; `applyAttrsBefore` checks it and raises via `ThrowStdException` before the connection is exposed. Follow that pattern — translate a failing return into an exception rather than handing back a partially-initialized object. +- **Handle every shipped architecture, not just `$(uname -m)`.** Universal2 bundles arm64 and x86_64; dylib/rpath work that only touches the host arch ships a broken slice for the other. Verify both. +- Hot paths favor the raw CPython API (with correct refcounting and `PyErr_Occurred()` checks) over pybind11 per-cell calls. -### Python Changes -- Preserve existing error handling patterns from `exceptions.py` -- Use context managers (`with`) for all connection/cursor operations -- Update `__all__` exports if adding public API -- Add corresponding test in `tests/test_*.py` -- Follow Black formatting (line length 100) +## Testing conventions -### C++ Changes -- Follow RAII patterns for resource management -- Use `py::gil_scoped_release` for blocking ODBC operations -- Update `mssql_python.pyi` type stubs if changing Python API -- Follow `.clang-format` style (LLVM style, 100 column limit) +- Test files are mostly numbered `test_NNN_*.py`; `tests/test_000_dependencies.py` runs without a DB, most others need a live SQL Server. `-m "not stress"` is the default. +- Run segfault-prone or ODBC/pool global-state tests in an **isolated subprocess** so a crash or shared state cannot poison the rest of the suite. +- **Assert the contract, not just the output.** If a change's value is "we now call X once," assert the call/round-trip count — a correctness-only test won't catch a perf regression. +- For global type-mapping changes, add typed-NULL integration cases (VARBINARY, UNIQUEIDENTIFIER, XML, DECIMAL, stored-proc params) before applying the optimization broadly. -## Code Quality +## Security and credentials -- **Keep it tight**: Minimal code to solve the problem. No duplicate logic, no redundant validation. -- **Comments explain why, not what**: Don't restate what the code does. Comment intent, edge cases, and non-obvious decisions. -- **One-line docstrings**: For test functions and simple helpers. No "Validates:" bullet lists or "Note:" paragraphs. -- **Docstring examples must match the API**: If the signature changes, update the examples. -- **Catch specific exceptions**: Use `DatabaseError`, `IntegrityError`, etc. — never `except Exception` or bare `except:`. -- **Let the test framework work**: Don't wrap test bodies in `try/except: pytest.fail()`. Let pytest show the real traceback. -- **Assertions must match claims**: If a test says "all types", check all of them. Cover the case that motivated the fix. -- **No stale references**: If you reference a file, function, or prompt — verify it exists first. +- **Committed connection strings that contain `UID`/`PWD` must use `SERVER=localhost` (or `127.0.0.1`) with dummy values.** Real remote or Azure credentials come only from secrets or the `DB_CONNECTION_STRING` env var, and are never committed. Automated credential scanning (see `.config/CredScanSuppressions.json`, `.gdn/`) can block unsafe patterns. +- Do **not** put `Driver=` in a connection string — the bundled driver is selected automatically. +- `TrustServerCertificate=yes` is local-development only; never suggest it in remote or production examples. -## Debugging Quick Reference +## Contributing -| Error | Cause | Solution | -|-------|-------|----------| -| `ImportError: ddbc_bindings` | Extension not built | Run `#build-ddbc` | -| Connection timeout | Missing env var | Set `DB_CONNECTION_STRING` | -| `dylib not found` (macOS) | Library paths | Run `configure_dylibs.sh` | -| `ODBC Driver not found` | Missing driver | Install Microsoft ODBC Driver 18 | -| `ModuleNotFoundError` | Not in venv | Run `#setup-dev-env` | +- Branch naming: `/` (e.g. `bewithgaurav/fix-656-macos-dylib`). +- Link exactly one reference in the PR body: maintainers use `AB#` (ADO auto-close); external contributors use plain `#` — **never `Closes #N`**. +- Stage specific files; **never `git add .`**. Never commit build artifacts (`*.so`, `*.pyd`, `*.dll`, `*.dylib`) or a virtual environment. +- Keep changes surgical — fix the requested thing, leave unrelated code alone, and do not add stray files. -## Contributing Guidelines +## Working effectively in this repo -### PR Requirements -- **Title Format**: Must start with `FEAT:`, `CHORE:`, `FIX:`, `DOC:`, `STYLE:`, `REFACTOR:`, or `RELEASE:` -- **Issue Linking**: Must link to either GitHub issue or ADO work item -- **Summary**: Minimum 10 characters of meaningful content under "### Summary" +- **Reproduce before you claim.** Build and run a live repro against a real SQL Server before asserting a bug exists or that a fix works. Do not ship code-only assessments. +- Understand the linked issue and existing review threads before changing code. +- Search existing issues and PRs before opening a new one — do not create duplicates. +- Do not post PR or issue comments as part of a change unless explicitly asked. +- Verify that mutating git/gh operations actually landed (re-read the branch or PR) before reporting done. -### Development Workflow -1. **Always build native extensions first** before running tests -2. **Use virtual environments** for dependency isolation -3. **Test on target platform** before submitting PRs -4. **Check CI pipeline results** for cross-platform compatibility +## Trust these instructions +Trust the commands, paths, and conventions above — they were validated against the current repository. Only fall back to searching the codebase when something here is missing or proves incorrect, and when you find a gap, update these instructions. diff --git a/llms.txt b/llms.txt deleted file mode 100644 index cf2e0b058..000000000 --- a/llms.txt +++ /dev/null @@ -1,256 +0,0 @@ -# mssql-python - -mssql-python is the official Microsoft Python driver for SQL Server and Azure SQL databases. It provides DB API 2.0 compliant database access with Direct Database Connectivity (DDBC) - no external ODBC driver manager required. - -## Installation - -```bash -pip install mssql-python -``` - -Or using uv (recommended for faster installs): -```bash -uv pip install mssql-python -``` - -## Quick Start - -> **Security Note**: The examples below use `TrustServerCertificate=yes` for local development with self-signed certificates. For production or remote connections, remove this option to ensure proper TLS certificate validation. - -```python -import mssql_python - -conn = mssql_python.connect( - "SERVER=localhost;DATABASE=mydb;Trusted_Connection=yes;Encrypt=yes;TrustServerCertificate=yes" -) -cursor = conn.cursor() -cursor.execute("SELECT * FROM users WHERE id = ?", (42,)) -rows = cursor.fetchall() -cursor.close() -conn.close() -``` - -## Connection String Formats - -### Windows Authentication (Trusted Connection) -```python -conn = mssql_python.connect( - "SERVER=localhost;DATABASE=mydb;Trusted_Connection=yes;Encrypt=yes;TrustServerCertificate=yes" -) -``` - -### SQL Server Authentication -```python -conn = mssql_python.connect( - "SERVER=localhost;DATABASE=mydb;UID=myuser;PWD=mypassword;Encrypt=yes;TrustServerCertificate=yes" -) -``` - -### Azure SQL with Entra ID (Interactive) -```python -conn = mssql_python.connect( - "SERVER=myserver.database.windows.net;DATABASE=mydb;Authentication=ActiveDirectoryInteractive;Encrypt=yes" -) -``` - -### Azure SQL with Managed Identity -```python -conn = mssql_python.connect( - "SERVER=myserver.database.windows.net;DATABASE=mydb;Authentication=ActiveDirectoryMSI;Encrypt=yes" -) -``` - -### Azure SQL with Service Principal -```python -conn = mssql_python.connect( - "SERVER=myserver.database.windows.net;DATABASE=mydb;" - "Authentication=ActiveDirectoryServicePrincipal;" - "UID=;PWD=;Encrypt=yes" -) -``` - -## Context Manager Usage (Recommended) - -```python -import mssql_python - -with mssql_python.connect(connection_string) as conn: - with conn.cursor() as cursor: - cursor.execute("SELECT @@VERSION") - version = cursor.fetchone() - print(version[0]) -``` - -## Query Patterns - -### Parameterized Queries (Prevent SQL Injection) -```python -cursor.execute("SELECT * FROM users WHERE email = ? AND active = ?", (email, True)) - -# Named parameters (pyformat style) -cursor.execute("SELECT * FROM users WHERE email = %(email)s", {"email": email}) -``` - -### Fetch Results -```python -row = cursor.fetchone() -rows = cursor.fetchmany(size=100) -all_rows = cursor.fetchall() -``` - -### Insert Data -```python -cursor.execute( - "INSERT INTO users (name, email) VALUES (?, ?)", - ("John Doe", "john@example.com") -) -conn.commit() # Don't forget to commit! -``` - -### Insert Multiple Rows (executemany) -```python -users = [ - ("Alice", "alice@example.com"), - ("Bob", "bob@example.com"), - ("Charlie", "charlie@example.com"), -] -cursor.executemany("INSERT INTO users (name, email) VALUES (?, ?)", users) -conn.commit() -``` - -### Update Data -```python -cursor.execute("UPDATE users SET active = ? WHERE id = ?", (False, user_id)) -conn.commit() -print(f"Rows affected: {cursor.rowcount}") -``` - -### Delete Data -```python -cursor.execute("DELETE FROM users WHERE id = ?", (user_id,)) -conn.commit() -``` - -## Transaction Management - -```python -from mssql_python import DatabaseError - -try: - cursor.execute("INSERT INTO orders (customer_id, total) VALUES (?, ?)", (1, 99.99)) - cursor.execute("UPDATE inventory SET quantity = quantity - 1 WHERE product_id = ?", (42,)) - conn.commit() # Commit both changes -except DatabaseError: - conn.rollback() # Rollback on error - raise -``` - -### Autocommit Mode -```python -conn = mssql_python.connect(connection_string) -conn.autocommit = True # Each statement commits immediately -``` - -## Working with Results - -### Get Column Metadata -```python -cursor.execute("SELECT id, name, email FROM users") -for column in cursor.description: - print(f"Column: {column[0]}, Type: {column[1]}") -``` - -## Stored Procedures - -```python -# Call a stored procedure -cursor.execute("EXEC GetUserById ?", (user_id,)) -result = cursor.fetchone() - -# OUTPUT parameters: use DECLARE/EXEC/SELECT pattern -# (callproc() is not supported; use execute() with anonymous code blocks) -cursor.execute(""" - DECLARE @count INT; - EXEC CountActiveUsers @count OUTPUT; - SELECT @count; -""") -count = cursor.fetchone()[0] -``` - -## Error Handling - -```python -from mssql_python import ( - DatabaseError, - IntegrityError, - ProgrammingError, - OperationalError, -) - -try: - cursor.execute("INSERT INTO users (id, name) VALUES (?, ?)", (1, "John")) - conn.commit() -except IntegrityError as e: - # Constraint violation (duplicate key, foreign key, etc.) - print(f"Constraint error: {e}") -except ProgrammingError as e: - # SQL syntax error or invalid query - print(f"SQL error: {e}") -except OperationalError as e: - # Connection issues, timeouts, etc. - print(f"Operational error: {e}") -except DatabaseError as e: - # General database error - print(f"Database error: {e}") -``` - -## Connection Pooling - -Connection pooling is enabled by default: - -```python -# Pooling is automatic - connections are reused -conn1 = mssql_python.connect(connection_string) -conn1.close() # Returns to pool - -conn2 = mssql_python.connect(connection_string) # Reuses pooled connection -``` - -## Comparison to pyodbc - -| Feature | mssql-python | pyodbc | -|---------|-------------|--------| -| Driver | Ships bundled | Requires separate ODBC driver | -| Installation | `pip install` only | pip install + ODBC driver setup | -| Azure Entra ID | Native Python API | Via ODBC driver connection keywords | -| Connection Pooling | Built-in | Via ODBC Driver Manager | -| DB API 2.0 | ✅ Compliant | ✅ Compliant | - -### Migration from pyodbc -```python -# pyodbc -import pyodbc -conn = pyodbc.connect("DRIVER={ODBC Driver 18 for SQL Server};SERVER=...;DATABASE=...") - -# mssql-python (no DRIVER needed) -import mssql_python -conn = mssql_python.connect("SERVER=...;DATABASE=...") -``` - -## Supported Platforms - -- Windows (x64, ARM64) -- macOS (ARM64, x86_64) -- Linux (x86_64, ARM64): Debian, Ubuntu, RHEL, Alpine (musl) -- Linux (x86_64 only): SUSE - -## Supported Python Versions - -- Python 3.10+ - -## Links - -- Documentation: https://github.com/microsoft/mssql-python/wiki -- PyPI: https://pypi.org/project/mssql-python/ -- GitHub: https://github.com/microsoft/mssql-python -- Issues: https://github.com/microsoft/mssql-python/issues