* Removed pytestmark db_test from the elasticsearch providers tests (apache#52139) * Remove pytestmark and add db_test marker to relevant tests (apache#52140) * Fix Task Instance “No Status” Filter (apache#51880) * Support no_status alias in TaskInstance state filter for REST API * Allow 'no_status' state filter and include no_status in valid state list; skip date filters when filtering for null state * Fix NULL-state filtering in get_mapped_task_instances by coalescing date fields * Refactor datetime_range_filter_factory: coalesce only start_date and end_date filters * Add a test * Add Pattern to companies using Airflow (apache#52149) Pattern is The Premier Accelerator for Global Ecommerce * Add a button to collapse/expand the information panel (apache#51946) * Add a button to collapse/expand the information panel for better visualizing DAG * remove transform of IconButton * change Box width * add translations for aria-label (en, zh-TW) * change translations for zh-TW * Add chart index.yaml step back to chart release guide (apache#52160) This was removed in apache#50464, but we still need to do this step. * fix(provider): Fix kwargs handling in Azure Data Lake Storage V2 Hook methods (apache#51847) * Require release flags in breeze helm chart issue command (apache#52162) We need these, so fail early if they are missing (say, you missed escaping a newline 😂). * fix mypy errors in otel_tracer (apache#52170) * Remove unused code from `models/dag.py` (apache#52173) These were not used and aren't part of public interface. * Update PostgreSQL to 16 in example docker-compose.yaml and docs. (apache#52174) * Remove unused `SimpleTaskInstance` (apache#52176) These isn't used in Airflow 3 and isn't part of public interface. * Add deprecation to `airflow/sensors/base.py` (apache#52178) Had to todo earlier, resolved that. * Remove @pytest.mark.db_test for cncf (apache#52153) * Use PythonOperator import from standard provider in ydb providers example (apache#52165) * Remove unused import Case from dagrun.py (apache#52179) * Remove old Serialization enums (apache#52183) This aren't used anymore -- these were initially part of AIP-44 but were missed during cleanup * Add description of what kind of changes we cherry-pick (apache#52148) Following the discussion in devlist - this PR adds description of what kind of changes we cherry-pick: https://lists.apache.org/thread/f3off4vtn2h6ctznjd5wypxvj1t38xlf * Ignore mypy errors for deprecated executors (apache#52187) I removed SimpleTaskInstance in apache#52176 since it isn't used in Airflow 3. This caused failure in hybrid executors like `LocalKubernetesExecutor` and `CeleryKubernetesExecutor` -- which aren't suported in Airflow 3. Hence we can ignore mypy errors. * Update alibaba example dags (apache#52163) * remove pytest db_test marker where unnecessary (apache#52171) * Fix spelling in edge provider (apache#52169) * Revert "Add deprecation to `airflow/sensors/base.py` (apache#52178)" (apache#52193) This reverts commit 54f9bff. * Refactor asana operator tests free from db access (apache#52192) * Move type-ignores up one line (apache#52195) The apache#52187 added ignores in a bit wrong place due to auto-reformatting * Add default conn name to asana provider operators (apache#52185) * Add default conn name to asana provider operators * Update tests * Helm: add custom annotations to jwt secret (apache#52166) * Add few small improvements in publishing workflow: (apache#52136) * allow to use any reference not only tag when publishing docs * autocomplete destinations for publish-to-s3 command * Fix archival for cascading deletes by archiving dependent tables first (apache#51952) Co-authored-by: Jed Cunningham <jedcunningham@apache.org> * Chart: Use api-server instead of webserver in NOTES.txt for Airflow 3.0+ (apache#52194) * Update providers metadata 2025-06-24 (apache#52188) * Doc update to install git in docker image prior 3.0.2 (apache#52190) * Doc update to install git in docker image prior 3.0.2 Prior to Airflow 3.0.2, docker image needs git installed on it to be able to use the git dag bundles feature. Adding this note to the docs * Fix static checks * Add Airflow 3.0+ Task SDK support to AWS Batch Executor (apache#52121) Added Task SDK support for AWS BatchExecutor to enable compatibility with Airflow 3.0+. The AWS BatchExecutor lacked support for handling Task SDK workloads, which has already been supported in the AWS ECSExecutor, changes were made to add this functionality. This meant the executor couldn't properly function with the latest Airflow architecture. Changes: - Implemented handling for ExecuteTask workloads in queue_workload method - Added _process_workloads method to properly process Task SDK workloads - Modified execute_async to handle cases where a workload object is passed instead of a direct command - Added serialization logic to convert workloads to JSON for execution in AWS Batch containers - Added new test case to verify Task SDK integration works correctly * Automatically add "backport" label to dev tool changes (apache#52189) * Added additional steps to QuickSights test prerequisites (apache#52198) * OS platform dependent code changed to platform independent (#59) Co-authored-by: Satish Ch <satishchinthanippu@gmail.com> * Bumping min version of pagerduty to 2.3.0 (apache#52214) * Bumping min version of pagerduty to 2.3.0 * Bumping min version of pagerduty to 2.3.0 * Bteq platform independent (#61) * OS platform dependent code changed to platform independent * mac platform verified and adjusted code to work with zsh and normal shell --------- Co-authored-by: Satish Ch <satishchinthanippu@gmail.com> * Fix whitespace handling in DAG owners parsing for multiple owners (apache#52216) * DEL: pytestmark in test_opensearch.py (apache#52213) * Fixing upgrade checks on main (apache#52210) * Add more diagnostics for Airflow installation inside CI image (apache#52223) * Separate out creation of default Connections for tests and non-tests (apache#52129) * Airbyte test fixes, make mock JobResponse response id as int (apache#52134) * Airbyte test fixes, make mock JobResponse response id as int * Airbyte test fixes, make mock JobResponse response id as int * Nuke unused latest flag for preparing helm chart release (apache#52229) * Remove HDFSHook, HdfsRegexSensor, HdfsSensor, HdfsFolderSensor (apache#52217) These classes generated RuntimeError since version 4.0.0 * Remove db_tests from openlineage provider (apache#52239) Part of apache#52020 Still some tests left. * Remove unused LoggerMutationHelper (apache#52241) This was removed in Airflow 3.0 as part of the TaskSDK rewrite and is not used anymore * Fix xdist compatibility for test_local_to_gcs test (apache#52244) The test used hard-coded "/tmp" folder to create and delete files and used the same files in several tests, when running it as non-db test with xdist, that caused sometimes failures because the tests could randomly override each-others-data. This PR fixes it by switching to pytest fixture instead of setup/teardown and using tmp_path fixture to use different tmp folder for different invocations of test methods. * Bump the core-ui-package-updates group across 1 directory with 2 updates (apache#52167) Bumps the core-ui-package-updates group with 2 updates in the /airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui directory: [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) and [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite). Updates `typescript-eslint` from 8.34.1 to 8.35.0 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.35.0/packages/typescript-eslint) Updates `vite` from 6.3.5 to 7.0.0 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/create-vite@7.0.0/packages/vite) --- updated-dependencies: - dependency-name: typescript-eslint dependency-version: 8.35.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: core-ui-package-updates - dependency-name: vite dependency-version: 7.0.0 dependency-type: direct:development update-type: version-update:semver-major dependency-group: core-ui-package-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Show tooltip when hovering on the button handling details panel (apache#52212) * Fixed external links in Navigation buttons (apache#52220) * Set downstream option to default on task instance clear (apache#52130) * Set downstream default when clear task instance * Set downstream default when mark TI success or failed * feat: added `request_body` support in the `PowerBIDatasetRefreshOperator` (enables support for enhanced dataset refreshes) (apache#51397) * feat: initial draft implementation for `request_body` support in the `PowerBIDatasetRefreshOperator` - not tested yet * fix: reference to correct URL in case API changes in future * test: update `TestPowerBITrigger` * chore: pre-commit checks * chore: remove TODOs * test: add sample request_body to `TestPowerBIDatasetRefreshOperator` * test: add example of `request_body` to system tests / examples * chore: missing trailing comma * Remove db usage from http provider tests (apache#52227) * Remove db usage from http provider tests * Remove db usage from http provider tests * Fix http hook tests * Fix operators and triggers tests * Document taskflow decorators and fix setup/teardown docstrings (apache#52181) * Move `EdgeInfoType` to Task SDK (apache#52180) This is an internal class -- moving it where it belongs. Doesn't need newsfragment. * Add deprecation to `airflow/sensors/base.py` (apache#52249) * Clean up middlewares (apache#52116) * Add token API for `KeycloakAuthManager` (apache#52112) * Remove side-effects in models/tests_dags affecting plugin manager tests (apache#52258) I guess CI must not run this exact combination of tests together, but prior to this change if you ran `pytest airflow-core/tests/unit/models/test_dag.py::TestDag::test_bulk_write_to_db_assets airflow-core/tests/unit/plugins/test_plugins_manager.py::TestPluginsManager::test_registering_plugin_listeners` you would get a test failure. The issue was caused by having two fixtures of the same name, a module level `clean_plugins`, and a class level one. This is by design in Pytest and is how to override plugins at different scopes. This also explains why we had `listener_manager.clear()` in a finally block when it should have been handled by the fixture * Remove latest flag from core release issue generation cli (apache#52256) We always provide the current/prior flags anyways, so we do not need to support this flag. * Updating issue content generation in GH workflows (apache#52271) * Fix docstring typo in dag_processing/manager.py (apache#52266) * Clean up remaining DB-dependent tests from OpenSearch provider (apache#52235) * DEL: remove pytestmark * DEL: remove pytestmark in os_response * DEL: remove pytestmark in operator * CHG: opensearch in .pre-commit-config.yaml and Mark DB-dependent tests in test_os_task_handler with @pytest.mark.db_test * CHORE: Enable db_test pre-commit check for OpenSearch hooks/operators * DEL: check-pytest-mark-db-test-in-providers about opensearch in pre-commit-config.yaml * Fix multi line release command in CI (apache#52281) * Enhanced the BTEQ operator to ensure platform independence. (apache#52252) * OS platform dependent code changed to platform independent (#59) Co-authored-by: Satish Ch <satishchinthanippu@gmail.com> * Bteq platform independent (#61) * OS platform dependent code changed to platform independent * mac platform verified and adjusted code to work with zsh and normal shell --------- Co-authored-by: Satish Ch <satishchinthanippu@gmail.com> --------- Co-authored-by: Satish Ch <satishchinthanippu@gmail.com> * Unify selecting constraints option when installing airflow (apache#52274) Due to the way how it historically got added - we had two ways of selecting whether we are installing airlfow dyanmically in breeze with or without constraints: * --install-airflow-with-constraints - was used in a few places * --skip-airflow-constraints - was used in other places The logic to handle those were broken at places where they contradicted each other. This PR unifies it and only uses the --install-airflow-with-constraints flag in all the places where we need to determine whether constraints are used or not and it fixes the logic. The logic of installation had been reviewed, refactored into separate methods doing smaller tasks and more diagnostics was added. * Enhance Variable set method to use upsert instead of delsert (apache#48547) * Enable Serde for Pydantic BaseModel and Subclasses (apache#51059) This adds serialization and deserialization support for arbitrary pydantic objects, while still maintaining security. --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Documentation improved * Use base AWS classes in Glue Trigger / Sensor and implement custom waiter (apache#52243) * Adjusted the GlueJobSensor to inherit from AwsBaseSensor * Changed timeout logic and added further tests * Renamed test case due to removal of max_retries param * Added custom GlueJob waiter * Added new params to GlueJobOperator and fixed GlueTrigger tests * Refined params of operator, trigger and hook * Handle exceptions when fetching status in GlueJobHook (apache#52262) * Handle exceptions when fetching status in GlueJobHook * Add api_retry_args to the ALLOWED_THICK_HOOKS_PARAMETERS dictionary for GlueJobHook * Ensure `HttpHook.run()` does not alter `extra_options` passed to it (apache#51893) * Prevent alteration of the extra_options dict * Removing TODO's * Updating unit test naming, no change in logic * Removing pop logic in favor of get, where applicable * Changing deepcopy to shallow copy * Validating that extra_options is not modified * Remove double call to plugin init (apache#52291) * Remove unused import Sequence from the celery_executor.py (apache#52290) * Deprecated import fix for TimeDeltaSensorAsync in example dags (apache#52285) Co-authored-by: Atul Singh <atulsingh@Atuls-MacBook-Pro.local> * Grid view optimization (apache#51805) The headline here is, with 6k tasks in a dag, loading time for 10 runs drops from 1.5m to < 10s in a quick local test. I split it into smaller more purpose-specific requests that each do less. So we have one request for just the structure, and another one for TI states (per dag run). I also find ways to stop refreshing when there's no active dag run (or the particular dag run is not active and its tis don't need refreshing. I also changed the "latest dag run" query (which checks for a new run triggered externally to be simpler dedicated endpoint. It runs every couple seconds even when there is nothing going on and now it takes 10ms instead of 300ms. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Add React Apps to plugin (apache#52255) Unrelated CI failure. * Skip test that needs the .git folder when it is missing (apache#52305) When you run breeze tests in breeze - by default .git folder is missing because it is not mounted to inside breeze. This can be remediated with `breeze shell --mount all` but this test should simply not run if .git folder is missing. * Python versions in shell params are strings (apache#52306) The versions were "floats" and it accidentally worked because they were coerced to strings, but with 3.10 it will be coerced to the "3.1" string. * Bump pymssql version to 2.3.5 (apache#52307) There is a problem with 2.3.4 that the .whl files for MacOS are broken / missing and when installing with Python 3.10 on MacOS, pymssql installation fails. Since this is only pymssql, we can easily bump it to 2.3.5 to avoid it - it is the latest version installed anyway in main now. * Remove pre-commit check-daysago-import-from-utils (apache#52304) * Remove pre-commit check-daysago-import-from-utils * fixes * Use proper show-only value in test_worker.py (apache#52300) * Revert "Enable Serde for Pydantic BaseModel and Subclasses (apache#51059)" (apache#52312) This reverts commit a041a2a. * Fix GlueJobOperator deferred waiting (apache#52314) In apache#52243 the waiting was moved from custom code within the glue hook to using the aws base waiters when deferring Glue jobs. The Trigger was given inappropriate inputs which caused it to wait for zero attempts, which causes our tests to fail. This change moves to using the common parameters we use for other operators in deferrable with the same defaults as the Trigger has. Note: previously this Operator used to wait indefinitely for the job to either complete or fail. The default now waits for 75 minutes. The aws base waiter has no ability to wait indefinitely, nor do I think it should, that feels like a bug to me. So I'm considering this slight behaviour change a bug fix of a bug fix. * cleanup stale dependency of methodtools (apache#52310) * Enable DatabricksJobRunLink for Databricks plugin, skip provide_session usage in Airflow3 (apache#52228) This PR introduces support for the "See Databricks Job Run" extra link in the Databricks workflow provider plugin for Airflow 3. The implementation stores the job run URL in XCom during task execution and retrieves it when the extra link is accessed. Additionally, when using Airflow 3, the PR refactors the plugin code to eliminate the use of `@provide_session` and direct database access for compatibility with Airflow 3. These changes address the concerns raised in [issue apache#49187](apache#49187) regarding the Databricks provider plugin. Support for the Databricks workflow repair functionality in Airflow 3 is still pending. A follow-up issue apache#52280 has been filed to explore a new approach for implementing repair in Airflow 3. Related: apache#49187 * Fix mypy errors in GCP `generative_model` (apache#52321) * fix: task-sdk AssetEventOperations.get to use alias_name when specified (apache#52303) * Fix AssetEventOperations.get to use alias_name when specified * Use syntax compatible with python 3.9 * Fix: Unclosed aiohttp ClientSession and TCPConnector in DatabricksRunNowOperator (deferrable=True) (apache#52119) Closes: apache#51910 Fixes unclosed `aiohttp.ClientSession` and `TCPConnector` warnings when using `DatabricksRunNowOperator` with `deferrable=True` in Airflow 3.0.2 and Databricks Provider 7.4.0. ### Background As described in apache#51910, the following errors appear during deferrable task execution: ``` Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x...> Unclosed connector connector: <aiohttp.connector.TCPConnector object at 0x...> ``` These indicate improper async resource cleanup during trigger polling. ### Fix - Ensures `aiohttp.ClientSession` and `TCPConnector` are properly closed - Applies best practices for async resource lifecycle management in the trigger --------- Co-authored-by: Salikram Paudel <paudelsalik4@gamil.com> * Use BaseSensorOperator from task sdk in providers (apache#52296) * Use BaseSensorOperator from task sdk in providers * Use BaseSensorOperator from task sdk in providers * Use BaseSensorOperator from task sdk in providers * Fix tests * Fix tests * feat: Add new query related methods to SnowflakeSqlApiHook (apache#52157) * Attempt2: Fix mypy in gcp generative_model (apache#52331) * Attempt2: Fix mypy in gcp generative_model * Remove private class imports * Replace occurences of 'get_password' with 'password' to ease migration (apache#52333) * Replace `models.BaseOperator` to Task SDK one for Standard Provider (apache#52292) The Providers should use the BaseOperator from Task SDK for Airflow 3.0+. * Drop support for Python 3.9 (apache#52072) * Drop support for Python 3.9 * fixes * fix casandra * fix casandra * fix PreviewGenerativeModel * fix PreviewGenerativeModel * fix static checks * fix datetime.py * Replace usage of 'set_extra' with 'extra' for athena sql hook (apache#52340) * Replace `models.BaseOperator` to Task SDK one for Alibaba & Airbyte (apache#52335) Follow-up of apache#52292 for Alibaba & Airbyte * chore: use task_instance as source for all airflow identifiers used in listener (apache#52339) * Bump google-cloud-bigquery>=3.24.0 (apache#52337) * Cleanup stale Python3.9 dependencies (apache#52344) * Make airflow-ctl test_login safe for parallel execution by using temp AIRFLOW_HOME (apache#52345) * Handle directory creation for tests more robustly in airflow-ctl * generalising it to temp home * Improve safety for external views (apache#52352) * Set snowflake-snowpark-python for Python 3.12 (apache#52356) * Set snowflake-snowpark-python for Python 3.12 * fix * Bump ibmcloudant>=0.10.0 (apache#52354) * Fix UnboundLocalError for `edge_job_command_len` (apache#52328) * fix: fix UnboundLocalError for `edge_job_command_len` * Fix `edge_job_command_len` UnboundLocalError (explicitly init) * Chart: Fix JWT secret name (apache#52268) * Fix indexerror in _find_caplog_in_def selective check function (apache#52369) * Bump microsoft kiota packages to 1.9.4 and update tests (apache#52367) * Check chart annotations with pre-commit (apache#52365) It's easy to get "valid" helm annotations, but still be invalid artifacthub annotations because they are strings with yaml in them. Let's validate the strings are valid yaml too. * Add new `breeze run` command for non-interactive command execution (apache#52370) Add a new `breeze run` command that allows running commands in the Breeze environment without entering an interactive shell. This is useful for automated testing, and one-off command execution which is useful for AI too. * Bump ``uv`` to ``0.7.16`` (apache#52372) `0.7.16` was just released. * Replace `models.BaseOperator` to Task SDK one for Google Provider (apache#52366) Follow-up of apache#52292 for Google provider. * Add Python <=> Airflow compat filtering for breeze (apache#52386) * docstring update for gcp dataplex operator and hook (apache#52387) * Run release tests always - not only in canary runs (apache#52389) * Add plural per-language forms in check-translations script (apache#52391) Different languages have different plural forms. Our script should take the original English forms and convert them into the right plural forms for the language. Also noticed that sorting order is slightly different than the one that eslint uses. The "eslint" sorting order is now used when generating missing keys. * Update click requirement in /dev/breeze (apache#52361) Updates the requirements on [click](https://github.com/pallets/click) to permit the latest version. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.1.8...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Limit click back to 8.2.0 because it has an ENVVAR bug for flags (apache#52404) There is a bug in Click 8.2.0 and 8.2.1 that makes flag variables do not properly evaluate "false"-y values set in environment variables. See the issue pallets/click#2952 * Replace `models.BaseOperator` to Task SDK one for Asana & Arangodb (apache#52374) Follow-up of apache#52292 for Asana & Arangodb * Replace `models.BaseOperator` to Task SDK one for Atlassian (apache#52376) Follow-up of apache#52292 * Replace `models.BaseOperator` to Task SDK one for Apache Pig (apache#52375) Follow-up of apache#52292 * Replace `models.BaseOperator` to Task SDK one for DBT & Databricks (apache#52377) * Reduce timeout for task-sdk/airflow-ctl tests job workflow (apache#52399) * Add timeout for distribution tests job * Add timeout for distribution tests job * Provider Migration: Update trino for Airflow 3.0 compatibility (apache#52383) * ADD: impport for BaseOperator in version_compat.py * CHG: import change * Add missing Polish Translations including proper plural forms (apache#52395) Update airflow-core/src/airflow/ui/public/i18n/locales/pl/components.json Co-authored-by: Kacper Muda <mudakacper@gmail.com> * enhance error message for `breeze --backend none` to suggest setting a valid backend (apache#52318) * CHG: option_backend in breeze * CHG: when backend=none modify message in breeze * CHG: Apply common_option change for backend validation (pre-commit) * CHG: update sentence Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com> * CHG: Supported values msg and delete mssql * CHG: fix(breeze): exit with error if START_AIRFLOW=true and --backend=none * DEL: . in sentence * CHG: reflect pre-commit output * Update scripts/in_container/check_environment.sh --------- Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> * Adding some intelligence to classifying provider commits (apache#52407) * Provider Migration: Update github provider for Airflow 3.0 compatibility (apache#52415) * Provider Migration: Update github provider for Airflow 3.0 compatibility * refactor: move context if-else conditions into version_compat * Bring back providers compatibility checks (apache#52398) The compatibility checks were removed in apache#52072 accidentally. This one brings them back: * Python 3.10 * do not add cloudant (it was not working for Python 3.9) * Change analytics-python to segment-analytics-python (apache#52401) * Change analytics-python to segment-analytics-python * fix import * Provider Migration: Update airbyte provider for Airflow 3.0 compatibility (apache#52418) * Sanitize Username (apache#52419) Escape user.username in flash banners to prevent potential HTML injection * Add a script to report outdated versions in constraints (apache#52406) * Skip check-airflow-providers-bug-report-template in non main branch (apache#52426) * Clean some leftovers of Python 3.9 removal - Airflow core pieces (apache#52424) * Clean some leftovers of Python 3.9 removal - Github pieces (apache#52423) * Add inline dependencies for uv run and colors to dependencies script (apache#52428) * Make sure all test version imports come from test_common (apache#52425) * Provider Migration: Update Oracle for Airflow 3.0 compatibility (apache#52382) * Update BaseOperator imports for Airflow 3.0 compatibility * update based on latest instruction * Provider Migration: Update Weaviate for Airflow 3.0 compatibility (apache#52381) * Update BaseOperator imports for Airflow 3.0 compatibility * update according to latest instruction * remove type ignore since it is not a mock context * Add selected packages and explain why to the package scripts (apache#52433) The script has now two more parameters: * --selected-packages with comma separated list of packages * --explain-why - explaining whhy the latest version of packages is not installed. * Fix failing static check for Oracle provider (apache#52436) * Replace `models.BaseOperator` to Task SDK one for SFTP (apache#52435) * Replace models.BaseOperator to Task SDK one for SFTP * Resolve MC, adding PokeReturnValue to version_compat.py * Clean some leftovers of Python 3.9 removal - Task-SDK (apache#52434) * Clean some leftovers of Python 3.9 removal - Airflow CTL pieces (apache#52430) * Add keycloak to providers removed when running Airflow 2 (apache#52442) We we are using "--use-airflow-version" and use Airlfow 2 we uninstall all providers that are Airflow 2 only mounted from sources, because Provider's Manager (correctly) fails if Airflow 3 provider is installed. Recently added keycloak was missing from the list. * i18n(Ko): Add missing translations in admin.json and common.json (apache#52417) * i18n(Ko): Add missing translations in admin.json and common.json * Fix some translations * Fix editing connection with sensitive extra field (apache#52403) * Handle unchanges json * Remove the redact from connections * Fix the static checks * Replace `models.BaseOperator` to Task SDK one for Apache TinkerPop (apache#52400) Follow-up of apache#52292 for Apache TinkerPop * Force the definition of `execution_api_server_url` based on `api_url` (apache#52184) * Force the definition of execution_api_server_url * Add more tests * Improve constraints updated version check script (apache#52446) This script is now much nicer, and more useful: * it has been refactored and split into smaller methods * --verbose flag is added to help with diagnostics * the "regular" and "explain why" loops are now merged into a single loop * table is always printed now - even in "--explain-why" mode, we print the table as list of packages is being traversed and then "explain why" summary is printed at the end. * typing is added everywhere * Update documentation for forcing core execution_api_server_url (apache#52447) * Add colors to go tests output in CI (apache#52454) * add: version_compat (apache#52448) * Improve terminal handling for breeze commands (apache#52452) Console width has been hard-coded in CI commands, which often limited what was written in CI (Github action's CI does not have a terminal, nor terminal width so we allocate pseudo-terminal there) However when running breeze locally we should be able to use all terminal width. This PR: * increases length of CI terminal as we tend to have longer paths now after we moved stuff to subdirectories * only fixes terminal size on CI and leaves it None (auto) for local runs * adds --tty (default auto) to `breeze run` command to allow to use it both locally and in CI. * Remove old, unused generate SVG airflowctl pre-commit and fix width (apache#52457) The command was duplicated - an old version of it was also defined using cli folder that does not exist any more. Also column width is fixed when generating the help files which will make it independent from where the generation is run. * i18n(Ko): Replace 연결 as 커넥션 (apache#52440) * i18n(Ko): Replace 연결 as 커넥션 * Remove trailing comma in admin.json * Speed-up constraints generation (apache#52449) Constraints generation was slow because we run them in a loop and we tried to run them all on a single machine - trying to utilize the fact that we only have to build airflow and provider packages once. But those are pretty fast, comparing to constraint generation and it's much better to parallelize the constraint jobs and run them on separatae workers. This will speed up constraint generation delays that will allow building PROD images and kubernetes checks faster. * Wire-in dependency check script in CI "finalize" job (apache#52450) After constraints are committed, we should generate and print summary of dependencies that could be upgraded. * Clean some leftovers of Python 3.9 removal - All the rest (apache#52432) * Clean some leftovers of Python 3.9 removal - All the rest * Fix static checks * Fix generate-constraints run on different python than base (apache#52464) It turns out that when we are installing Breeze we were using the "Image" python version and not the 'default" python version to install breeze, and Python 3.12 and 3.11 are not installed by default when generate-constraints runs. This change fixes this problem, also it changes the name of the generate-constraints job to only show the python version used. * Add GITHUB_TOKEN when preparing image for dependency summary (apache#52472) We need GITHUB_TOKEN to load the image from artifact. * Clean some leftovers of Python 3.9 removal - Files in root (apache#52463) * Rmeove --tty specification for running the dependency script (apache#52489) Apparently default --tty auto should be eough. * Filter only provided integration paths for breeze integration testing (apache#52462) * Filter only provided integration paths * Fix tests * Rename gremline integration name to tinkerpop * Fix selective_checks test * Update @integration pytest marker with tinkerpop * Provider Migration: Update docker for Airflow 3.0 compatibility (apache#52465) * Provider Migration: Replace `models.BaseOperator` to Task SDK for apache/impala (apache#52455) * Replace models.BaseOperator to Task SDK for apache/impala * Add test_version_compat ignore * Provider Migration: Replace `models.BaseOperator` to Task SDK for apache/hive (apache#52453) * Replace models.BaseOperator to Task SDK one for Common Providers * Fix static errors * Fix StopIteration in snowflake sql tests (apache#52394) * Cleanup unused args example_pyspark.py (apache#52492) * Cleanup unused args example_pyspark.py * Cleanup unused args example_pyspark.py * Make the dependency script executable (apache#52493) * Close German language gap June 28th (apache#52459) * Close German language gap June 28th * Review feedback Co-authored-by: Tamara Janina Fingerlin <90063506+TJaniF@users.noreply.github.com> * Review feedback --------- Co-authored-by: Tamara Janina Fingerlin <90063506+TJaniF@users.noreply.github.com> * Replace models.BaseOperator to Task SDK one for Common Providers (apache#52443) Part of apache#52378 * Generally do not force version_compat.py to have pytests (apache#52496) * Provider Migration: Update Apache Druid for Airflow 3.0 compatibility (apache#52498) * Update BaseOperator imports for Airflow 3.0 compatibility merge updates from master * remove version_compat.py update as PR 52496 * Replace models.BaseOperator to Task SDK for http (apache#52506) * Replace models.BaseOperator to Task SDK for apache/livy (apache#52499) * Replace models.BaseOperator to Task SDK for apache/hdfs (apache#52505) * Update BaseOperator imports for Airflow 3.0 compatibility (apache#52503) * Update BaseOperator imports for Airflow 3.0 compatibility (apache#52504) * Revert "Replace models.BaseOperator to Task SDK for http (apache#52506)" (apache#52515) This reverts commit a9a7fcc. * [OpenLineage] Added operator_provider_version to task event (apache#52468) * added another attribute containing the provider package version of the operator being used. Signed-off-by: Rahul Madan <madan.rahul9@gmail.com> * precommit run Signed-off-by: Rahul Madan <madan.rahul9@gmail.com> --------- Signed-off-by: Rahul Madan <madan.rahul9@gmail.com> * Add a bunch of no-redef ignores so Mypy is happy (apache#52507) * Update Jenkins for Airflow 3.0 `BaseOperator` compatibility (apache#52510) Part of apache#52378 * Provider Migration: Update mysql for Airflow 3.0 compatibility (apache#52500) Follow-up of apache#52292. Part of apache#52378 * feat: Add explicit support for DatabricksHook to Ol helper (apache#52253) * Fix various incompatibilities with SQLAlchemy 2.0 (apache#52518) * Workaround to allow using `Base` as a superclass in sqla 2.0 * Add SQLA-version-dependent dialect kwarg generator * Fix test_connection.py * Fix test_import_error.py * Fix test_exceptions.py * Fix dag_run.py * Fix test_sqlalchemy_config.py * Fix rotate_fernet_key_command.py * Fix dag_version & test_scheduler_job * Fix db isolation between tests in test_collection.py * Update ERD diagram * One more redef needing ignore (apache#52525) * Provider Migration: Update Cohere for Airflow 3.0 compatibility (apache#52379) * feat: Add explicit support for SnowflakeSqlApiHook to Ol helper (apache#52161) * Provider Migration: Replace `BaseOperator` to Task SDK for `apache/http` (apache#52528) Part of apache#52378 Credits to @bdsoha for apache#52506 * Provider Migration: Update yandex provider for Airflow 3.0 compatibility (apache#52422) Part of apache#52378 * Replace models.BaseOperator to Task SDK one for Mongo (apache#52566) * fix: enable iframe script execution (apache#52257) * fix: enable iframe script execution * fix: include vite env variables when transpiling typescripts * fix: add explanations to sandbox settings * fix: remove csp change * Add the `upgrade_sqlalchemy` breeze flag (apache#52559) + Fix some existing shellcheck violations * Fix airflow pin for fab provider (apache#52351) * feat: Add real-time clock updates to timezone selector (apache#52414) * feat: Add real-time clock updates to timezone selector * Add real-time clock update to user setting button Nav * Allow Providers Iframe script execution (apache#52569) * Provider Migration: Replace `BaseOperator` to Task SDK for `ssh` (apache#52558) Part of apache#52378 * Provider Migration: Replace `BaseOperator` to Task SDK for `Papermill` (apache#52565) Part of apache#52563 * Provider Migration: Replace `BaseOperator` to Task SDK for `OpenAI` (apache#52561) Part of apache#52378 * Provider Migration: Replace `BaseOperator` to Task SDK for `Pinecone` (apache#52563) Part of apache#52378 * Marking test_process_dags_queries_count as flaky (apache#52535) * Fix ParseImportError query in get_import_errors endpoint (apache#52531) * Fix ParseImportError query in get_import_errors endpoint An and_ is required in a join condition but was missing. This fixes the issue that the bundle_name filter does not have any effect. * fixup! Fix ParseImportError query in get_import_errors endpoint * Migrate segment provider to af3 (apache#52579) * Set prefix to generate correctly the FAB Auth Manager API ref (apache#52329) * set prefix to generate correctly the documentation because it is a FastAPI subapplication of the main one mounted in /auth * autogenerated openapi yaml file generated by pre-commits * Move compat shim in Standard Provider to `version_compat.py` (apache#52567) Moves the conditional imports to `version_compat.py` * Provider Migration: Replace `BaseOperator` to Task SDK for `singularity` (apache#52590) * Provider Migration: Replace `BaseOperator` to Task SDK for `samba` (apache#52588) * Provider Migration: Replace `BaseOperator` to Task SDK for `salesforce` (apache#52587) * Revert "Run release tests always - not only in canary runs (apache#52389)" (apache#52594) This reverts commit 7596539. * Fix deferrable mode for SparkKubernetesOperator (apache#51956) * Increase dependency epoch to trigger pip cache invalidation (apache#52599) After removal of analytics-python we still keep it in the constraints. This change is likely to build cache from the scratch an avoid analytics-python in our constraints. * Add Google Cloud VertexAI and Translate datasets import data verification (apache#51364) For the: - Google Cloud VertexAI datasets. - Google Cloud Trasnalation native model datasets. Co-authored-by: Oleg Kachur <kachur@google.com> * Refactor the google cloud DataprocCreateBatchOperator tests (apache#52573) - replce un-called method mock - add logging checks - populate labels checks Co-authored-by: Oleg Kachur <kachur@google.com> * Upgrade ruff to latest version (0.12.1) (apache#52562) Fixes apache#52551 * Fix SBOM commands to work for Airfow 2 (apache#52591) Airflow 3 will need to be updated with package-json.lock but for now we are fixing the sbom command to work for Airflow 2 (and generate airflow 2.11 SBOMS. Changes: * passing --github-token parameter which might be helpful to not rate-limit GitHub calls * allowing to pass either `--airflow-site-archive-path` or `--airflow-root-path` depending where we want to generate sbom - it can be generated in `archive` folder directly (when we want to update historical data) or in the airflow source directory when we want to add SBOM to **just** generated documentation during the doc-building phase * airflowctl: transition of bulk operations to return BulkResponse (apache#52458) * bulkactionresponse to bulkresponse * modified pool cmd * Provider Migration: Update presto for Airflow 3.0 compatibility (apache#52608) * ADD: add conditional import for BaseOperator * CHG: change import path * Provider Migration: Update opensearch for Airflow 3.0 compatibility (apache#52609) * ADD: add conditional import for BaseOperator * CHG: change import path * Provider Migration: Update neo4j for Airflow 3.0 compatibility (apache#52610) * NEW: add conditional import for BaseOperator * CHG: change import path * Provider Migration: Replace `BaseSensorOperator` to Task SDK for `datadog` (apache#52583) * Provider Migration: Replace BaseSensorOperator to Task SDK for datadog * Apply suggestion from @kaxil --------- Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Provider Migration: Replace `BaseOperator` to Task SDK for `dingding` (apache#52577) * Provider Migration: Replace BaseOperator to Task SDK for dingding * Apply suggestion from @kaxil --------- Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Fix symlink handling for static assets when installed in editable mode with uv (apache#52612) * Update app.py Add follow_symlink for StaticFiels * Update simple_auth_manager.py add follow_symlink for StaticFiles * Replace models.BaseOperator to Task SDK one for Slack Provider (apache#52347) * replace baseOperator to Task SDK * fix version compat * update imports * fix (apache#52607) * Add regional support for google secret manager hook (apache#52124) * Add regional support for google secret manager hook * Change property name from location_id to location * Remove backward compatibility comment. * Fix static check failing * Add more dependency reports (apache#52606) The dependency reports of ours will be different for: * different python versions * different constraint modes We change the job to produce the reports into matrix of jobs producing reports for all combinations of those. * Correctly treat requeues on reschedule sensors as resetting after each reschedule (apache#51410) * Update `BaseOperator` and `BaseSensorOperator` imports for Airflow 3.0 compatibility in `qdrant` provider (apache#52600) * Provider Migration: Replace `models.BaseOperator` to Task SDK for `smtp` (apache#52596) Related apache#52378 * Upgrade uv to 0.7.17 (apache#52615) * Ensuring XCom return value can be mapped for dynamically-mapped `@task_group`'s (apache#51556) * Added same logic to @task_group as is in @task for mapping over invalid XCom arg * Added same logic to @task_group as is in @task for mapping over invalid XCom arg * Fixing linting * Add support for templating the DockerOperator parameter (apache#52451) * Update `grpc` BaseOperator imports for Airflow 3.0 compatibility (apache#52603) * Update grpc BaseOperator imports for Airflow 3.0 compatibility * Apply suggestions from code review --------- Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Provider Migration: Update Apache Kylin for Airflow 3.0 compatibility (apache#52572) * Update influxdb BaseOperator imports for Airflow 3.0 compatibility (apache#52602) * Update influxdb BaseOperator imports for Airflow 3.0 compatibility * Apply suggestions from code review --------- Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Revert "Fix symlink handling for static assets when installed in editable mode with uv (apache#52612)" (apache#52620) This reverts commit d1f4420. * Replace `models.BaseOperator` to Task SDK one for OpsGenie (apache#52564) * Improve dependency report and uppgrading (apache#52619) Our dependencies should be set in "upgrade to newer dependencies" mode every time every single pyproject.toml changes - this is slower as it triggers full builds with all versions but it also prevents some errors when dependencies from one provider are impacting what will be resolved in the CI image. As part of it - whenever we run the dependency report with "source constraints" we use exactly the same `uv sync` command as used during image build with "ugprade to newer dependencies" - this way the report is more accurate as it includes some dependencies from dev dependency groups that have not been included in the current reports. * Allow more empty loops before stopping log streaming (apache#52614) In apache#50715 we starting short-circuiting if we hit 5 iterations of no new log messages. This works well, except in the scenario where there are no log messages at all. ES log handler has it's own short-circuit for that scenario, but it triggers based on time and that works out to ~7 iterations. Let's let ES have the first crack at it so the user gets a better message. Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com> * Honor `index_urls` when venv is created with `uv` in `PythonVirtualenvOperator` (apache#52287) * Use `index_urls` when venv is created with `uv` * Fix formatting * Remove conditional creation of `pip.conf` * Set Python package index for uv with environment variables * Update documentation * Fix unit tests * Provider Migration: Update cassandra for Airflow 3.0 compatibility (apache#52623) Co-authored-by: Natanel Rudyuklakir <natanelrudyuklakir@gmail.com> * Bump pyarrow to 16.1.0 minimum version for several providers (apache#52635) Pyarrow < 16.1.0 does not play well with numpy 2. Bumping it to 16.1.0 as minimum version should make compatibility tests to not downgrade to versions that are not compoatible when numpy 2 is already installed. It should also prevent our users from accidentally downgrading pyarrow or not upgrading it when numpy is upgraded to >= 2.0.0. * Disable UP038 ruff rule and revert mandatory `X | Y` in insintance checks (apache#52644) This came in to effect once we swapped to Py 3.10 as the minimum version. This is inplace because of the ruff rule [UP038], and as we have discovered (after changing it to this style in the first place) the docs for the rule say: > **Warning: This rule is deprecated and will be removed in a future release.** So lets change it back [UP038]: https://docs.astral.sh/ruff/rules/non-pep604-isinstance/#deprecation * Replace `models.BaseOperator` to Task SDK one for Tableau, Telegram, and Teradata (apache#52642) Part of apache#52378 --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Rahul Madan <madan.rahul9@gmail.com> Co-authored-by: Dominik <105610163+dominikhei@users.noreply.github.com> Co-authored-by: Yeonguk Choo <choo121600@gmail.com> Co-authored-by: Ankit Chaurasia <8670962+sunank200@users.noreply.github.com> Co-authored-by: Shaunak Sontakke <ssontakke@users.noreply.github.com> Co-authored-by: Wei-Yu Chen <60685934+Jasperora@users.noreply.github.com> Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: omrdyngc <omerdayangac@outlook.com> Co-authored-by: Christos Bisias <christosbis@gmail.com> Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> Co-authored-by: Josef Šimánek <josef.simanek@gmail.com> Co-authored-by: GPK <gopidesupavan@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Dov Benyomin Sohacheski <b@kloud.email> Co-authored-by: Aakcht <aakcht@gmail.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com> Co-authored-by: Jed Cunningham <jedcunningham@apache.org> Co-authored-by: Dheeraj Turaga <dheerajturaga@gmail.com> Co-authored-by: Isaiah Iruoha <76797894+isaiahiruoha@users.noreply.github.com> Co-authored-by: Satish Ch <satishchinthanippu@gmail.com> Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com> Co-authored-by: Kyungjun Lee <kyungjunlee.me@gmail.com> Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com> Co-authored-by: Ash Berlin-Taylor <ash@apache.org> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: BBQing <33732539+BBQing@users.noreply.github.com> Co-authored-by: humit <jhjang1005@naver.com> Co-authored-by: Ramon Vermeulen <ramonvermeulen98@gmail.com> Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com> Co-authored-by: Seongho Kim <55729930+ksh24865@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Aryan Khurana <101019909+AryanK1511@users.noreply.github.com> Co-authored-by: Jake Roach <116606359+jroachgolf84@users.noreply.github.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> Co-authored-by: Atul Singh <atulkumar.singh@astronomer.io> Co-authored-by: Atul Singh <atulsingh@Atuls-MacBook-Pro.local> Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> Co-authored-by: Przemysław Mirowski <miretpl@gmail.com> Co-authored-by: Niko Oliveira <onikolas@amazon.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Justyn Harriman <jharriman@openai.com> Co-authored-by: Salikram Paudel <36765526+SalikramPaudel@users.noreply.github.com> Co-authored-by: Salikram Paudel <paudelsalik4@gamil.com> Co-authored-by: Kacper Muda <mudakacper@gmail.com> Co-authored-by: Yanshi <8177510+yanshil@users.noreply.github.com> Co-authored-by: Junmin Ahn <36021827+junminahn@users.noreply.github.com> Co-authored-by: arvindp25 <58228701+arvindp25@users.noreply.github.com> Co-authored-by: Zhen-Lun (Kevin) Hong <zhenlun.hong01@gmail.com> Co-authored-by: bu <69127692+SCH227@users.noreply.github.com> Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com> Co-authored-by: Shubham Raj <48172486+shubhamraj-git@users.noreply.github.com> Co-authored-by: Farhan <52888929+ahmadtfarhan@users.noreply.github.com> Co-authored-by: Geonwoo Kim <78521993+kgw7401@users.noreply.github.com> Co-authored-by: Wonseok Yang <deadline.yang@gmail.com> Co-authored-by: Idris Adebisi <104902482+hardeybisey@users.noreply.github.com> Co-authored-by: Tamara Janina Fingerlin <90063506+TJaniF@users.noreply.github.com> Co-authored-by: Rahul Madan <34760210+rahul-madaan@users.noreply.github.com> Co-authored-by: Dev-iL <Gid.CTO+github@gmail.com> Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> Co-authored-by: Joel Pérez Izquierdo <joelperez91@gmail.com> Co-authored-by: Maksim <maksimy@google.com> Co-authored-by: olegkachur-e <qwsqad@gmail.com> Co-authored-by: Oleg Kachur <kachur@google.com> Co-authored-by: jj.lee <63435794+jx2lee@users.noreply.github.com> Co-authored-by: magic_frog <Chenyzsjtu@gmail.com> Co-authored-by: Harikrishna D <64940963+harikrishna12524@users.noreply.github.com> Co-authored-by: Collin McNulty <collin.mcnulty@gmail.com> Co-authored-by: Karen Braganza <karenbraganza15@gmail.com> Co-authored-by: Guangyang Li <2060045+gyli@users.noreply.github.com> Co-authored-by: fweilun <william20020503@gmail.com> Co-authored-by: Daniel Wolf <95075445+wolfdn@users.noreply.github.com> Co-authored-by: Nataneljpwd <104094207+Nataneljpwd@users.noreply.github.com> Co-authored-by: Natanel Rudyuklakir <natanelrudyuklakir@gmail.com>

ruff is held by apache-airflow #52551

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions