Cherry-pick build/test updates from main - #7532
Merged
Merged
Conversation
When the ISA95 smoke test pipeline generates the config file for aziot-edged, it generates slightly different values for `agent.config.auth.serveraddress` and `agent.config.image`. In the latter, it appends `:443` to the container registry's address. It seems that some recent change to the test environment has caused Docker to complain that serveraddress and image are different and then fail the pull operation. This leaves IoT Edge in an inconsistent state and the tests fail. This change removes the logic that appends the port in the image. It also removes trailing whitespace in the script. I confirmed that the ISA95 smoke tests pass with this change. ## Azure IoT Edge PR checklist:
The agents that run the ISA95 smoke tests for ARM64 are running Ubuntu 24.04, but the tests are failing because they try to install IoT Edge binaries for Ubuntu 22.04. This change corrects the problem. To test, I ran the ISA95 smoke test pipeline and confirmed that it passes. ## Azure IoT Edge PR checklist:
…t reconcile (Azure#7523) ## Summary Fixes the consistently failing `RouteMessageL3LeafToL4Module` nested end-to-end test. The failure is a test race: the AMQP test group starts before the L4/L5 deployment has finished reconciling, so the L4 edgeHub restart triggered by that deployment lands in the middle of the test. The core change is a reconcile-wait guard after `set-modules`. It also keeps two supporting changes that made the failure debuggable and the repro reliable. ## Root cause The `RouteMessageL3LeafToL4Module` timeout is caused by a missing synchronization step in the test pipeline, not by a network fault. After the MQTT tests, the pipeline calls `az iot edge set-modules` for L5 and L4 and then starts the AMQP tests immediately. `set-modules` only updates the desired twin; it does not wait for edgeAgent to apply the deployment. The L4 deployment changes the upstream protocol, edgeHub configuration/routes, and modules, so L4 edgeHub restarts after the AMQP tests have already started. L3 loses its parent connection for about 15 seconds during that restart, and the in-flight device-to-module message is delayed long enough to miss the test's 300-second receive window. Verified in run [171488751](https://msazure.visualstudio.com/One/_build/results?buildId=171488751&view=results): L3 lost its parent at `03:11:37.193` and reconnected at `03:11:52.212`, a 15.018-second outage coinciding with the L4 edgeHub restart. The `iotedgeApiProxy` / `IoTEdgeAPIProxy` casing difference between the MQTT and AMQP deployment files adds reconciliation churn, but it is not the complete cause; any deployment change that restarts edgeHub would expose the same missing wait. Normalizing the casing is not the fix. `RouteMessageL3LeafToL4Module` is the second AMQP test. The first, `QuickstartCerts`, runs immediately after `set-modules` and acts as an unintended buffer while the L4 reconcile completes in the background. Whether RouteMessage passes comes down to a race between that buffer and the L4 reconcile/edgeHub-restart, both of which vary run to run depending on the test agents' state (for example how long modules take to reach the `running` state, which depends on image-cache warmth). This is why the test is timing-sensitive rather than deterministically broken. ## Is this masking a 1.6 product regression? No. This was the first thing we checked, since the test began failing around the 1.6 timeframe and a test-only fix would be wrong if it hid a product regression. The decisive evidence is a controlled comparison of the **same binary and commit** run at different times: | Test Run | Date | Commit / CI artifact | Pre-RouteMessage buffer | RouteMessage | |---|---|---|---|---| | [20260529.1](https://msazure.visualstudio.com/One/_build/results?buildId=166045060&view=results) | 2026-05-29 | `b3531a737` / `20260528.1` | 78.6s | **pass**, 21.7s | | [20260710.12](https://msazure.visualstudio.com/One/_build/results?buildId=171795768&view=results) | 2026-07-10 | `b3531a737` / `20260528.1` | 33.1s | **timeout**, 300s | Identical bits, identical commit, identical parameters, clean identity namespace: it passed in May and times out today. A binary cannot regress against itself, so the change is environmental/timing (the buffer shrank relative to the reconcile), not a code regression in 1.6. Supporting data across a range of runs (all AMQP full-suite, on `main` unless noted): | Build | Date | Commit / CI artifact | Buffer | RouteMessage result | |---|---|---|---|---| | [161293998](https://msazure.visualstudio.com/One/_build/results?buildId=161293998&view=results) | 04-22 | `621ff98b` / `20260420.1` | 30.4s | pass 29.5s | | [162168027](https://msazure.visualstudio.com/One/_build/results?buildId=162168027&view=results) | 04-29 | `621ff98b` / `20260427.1` | 30.3s | pass 29.5s | | [166045060](https://msazure.visualstudio.com/One/_build/results?buildId=166045060&view=results) | 05-29 | `b3531a737` / `20260528.1` | 78.6s | pass 21.7s | | [170483773](https://msazure.visualstudio.com/One/_build/results?buildId=170483773&view=results) | 07-01 | `36d6c426` / `20260629.1` | 48.1s | timeout 300s | | [171767938](https://msazure.visualstudio.com/One/_build/results?buildId=171767938&view=results) | 07-10 | `36d6c426` / `20260706.1` | 58.2s | timeout 300s | | [171795768](https://msazure.visualstudio.com/One/_build/results?buildId=171795768&view=results) | 07-10 | `b3531a737` / `20260528.1` | 33.1s | timeout 300s | The outcome does not track a clean buffer threshold (April passed with a 30s buffer; a July run timed out with a 58s buffer), which is expected for a race where both the buffer and the concurrent reconcile time vary. The point the table establishes is narrower and solid: a known-good pre-1.6 binary that passed now times out, so the recent failures are timing/environmental, not a 1.6 product regression. The guard removes the dependency on that accidental buffer entirely by waiting for the actual reconcile, so it is the correct fix regardless of the exact timing. > Some earlier runs (for example [166823784](https://msazure.visualstudio.com/One/_build/results?buildId=166823784&view=results), 06-04) failed fast on `DeviceAlreadyExists` from a leaf identity orphaned by a prior aborted run, a separate issue addressed by the idempotent-identity change below, not the timeout this PR targets. ### Separate, out-of-scope product improvement A ~15s parent restart can strand an in-flight message for minutes because the edgeHub SDK connection-status callback is disabled and edgeHub falls back to slow connectivity polling. That is a genuine product improvement, tracked separately as Fix B in [Azure#7527](Azure#7527), and is out of scope here. This PR fixes the test race that makes the failure reproduce in CI. ## Changes 1. **Reconcile-wait guard (the fix).** Adds a guard to the shared `nested-deploy-config.yaml` template, applied to L5 then L4. After `set-modules`, it reads the desired edgeAgent `$version` and polls `reported.lastDesiredVersion` plus `lastDesiredStatus.code`, continuing only when the exact desired version reports status `200`. On a 300-second timeout it prints `lastDesiredStatus.description` and reported module states so the two failure modes (deploy never picked up vs picked up but reported non-200) are distinguishable from the pipeline log. 2. **Collect L4/L5 support bundles.** Adds `Collect_Nested_Bundles` to publish support bundles and raw module logs from L3, L4, and L5. The existing pipeline only preserved L3 evidence, which is why the parent (L4) edgeHub restart was invisible in earlier investigations. Kept because per-level evidence is broadly useful for nested-edge test debugging. 3. **Idempotent leaf-identity creation.** Deletes and recreates a leaf device identity orphaned when a prior run dies before teardown (the pipeline's identity cleanup only removes the L4/L5 parent devices, not the test-created leaf). Prevents a spurious `DeviceAlreadyExistsException` on the shared hub. ## Validation | Build | Harness | RouteMessage | |---|---|---| | [171795530](https://msazure.visualstudio.com/One/_build/results?buildId=171795530&view=results) | with guard | pass 26.8s (full AMQP suite 25/25) | | [171618229](https://msazure.visualstudio.com/One/_build/results?buildId=171618229&view=results) | with guard | pass 21.7s (full AMQP suite 25/25) | | [171795768](https://msazure.visualstudio.com/One/_build/results?buildId=171795768&view=results) | no guard | timeout 300s | With the guard, RouteMessage runs only after the L4/L5 deployment has reconciled, so the edgeHub restart no longer lands mid-test.
The cleanup stage of the nested end-to-end tests has begun failing recently, even though all the jobs within the stage pass. This is because the time it takes to complete the "Clean up identities" job has increased to right around 2 minutes, which is the timeout value for the job. So the job passes, but orchestration at the stage level times out. This change updates the 2 minute timeouts to 5 minutes, which should provide ample time to complete the work. ## Azure IoT Edge PR checklist:
arsing
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This update cherry-picks changes to build/test code from main:
Confirmed the following pipelines pass:
Azure IoT Edge PR checklist:
This checklist is used to make sure that common guidelines for a pull request are followed.
General Guidelines and Best Practices
Testing Guidelines