e2e: fix RouteMessageL3LeafToL4Module by waiting for nested deployment reconcile

[jlian]

Summary

Fixes the consistently failing RouteMessageL3LeafToL4Module nested end-to-end test. The failure is a test race: the AMQP test group starts before the L4/L5 deployment has finished reconciling, so the L4 edgeHub restart triggered by that deployment lands in the middle of the test.

The core change is a reconcile-wait guard after set-modules. It also keeps two supporting changes that made the failure debuggable and the repro reliable.

Root cause

The RouteMessageL3LeafToL4Module timeout is caused by a missing synchronization step in the test pipeline, not by a network fault.

After the MQTT tests, the pipeline calls az iot edge set-modules for L5 and L4 and then starts the AMQP tests immediately. set-modules only updates the desired twin; it does not wait for edgeAgent to apply the deployment. The L4 deployment changes the upstream protocol, edgeHub configuration/routes, and modules, so L4 edgeHub restarts after the AMQP tests have already started. L3 loses its parent connection for about 15 seconds during that restart, and the in-flight device-to-module message is delayed long enough to miss the test's 300-second receive window.

Verified in run 171488751: L3 lost its parent at 03:11:37.193 and reconnected at 03:11:52.212, a 15.018-second outage coinciding with the L4 edgeHub restart. The iotedgeApiProxy / IoTEdgeAPIProxy casing difference between the MQTT and AMQP deployment files adds reconciliation churn, but it is not the complete cause; any deployment change that restarts edgeHub would expose the same missing wait. Normalizing the casing is not the fix.

RouteMessageL3LeafToL4Module is the second AMQP test. The first, QuickstartCerts, runs immediately after set-modules and acts as an unintended buffer while the L4 reconcile completes in the background. Whether RouteMessage passes comes down to a race between that buffer and the L4 reconcile/edgeHub-restart, both of which vary run to run depending on the test agents' state (for example how long modules take to reach the running state, which depends on image-cache warmth). This is why the test is timing-sensitive rather than deterministically broken.

Is this masking a 1.6 product regression?

No. This was the first thing we checked, since the test began failing around the 1.6 timeframe and a test-only fix would be wrong if it hid a product regression. The decisive evidence is a controlled comparison of the same binary and commit run at different times:

Test Run	Date	Commit / CI artifact	Pre-RouteMessage buffer	RouteMessage
20260529.1	2026-05-29	b3531a737 / 20260528.1	78.6s	pass, 21.7s
20260710.12	2026-07-10	b3531a737 / 20260528.1	33.1s	timeout, 300s

Identical bits, identical commit, identical parameters, clean identity namespace: it passed in May and times out today. A binary cannot regress against itself, so the change is environmental/timing (the buffer shrank relative to the reconcile), not a code regression in 1.6.

Supporting data across a range of runs (all AMQP full-suite, on main unless noted):

Build	Date	Commit / CI artifact	Buffer	RouteMessage result
161293998	04-22	621ff98b / 20260420.1	30.4s	pass 29.5s
162168027	04-29	621ff98b / 20260427.1	30.3s	pass 29.5s
166045060	05-29	b3531a737 / 20260528.1	78.6s	pass 21.7s
170483773	07-01	36d6c426 / 20260629.1	48.1s	timeout 300s
171767938	07-10	36d6c426 / 20260706.1	58.2s	timeout 300s
171795768	07-10	b3531a737 / 20260528.1	33.1s	timeout 300s

The outcome does not track a clean buffer threshold (April passed with a 30s buffer; a July run timed out with a 58s buffer), which is expected for a race where both the buffer and the concurrent reconcile time vary. The point the table establishes is narrower and solid: a known-good pre-1.6 binary that passed now times out, so the recent failures are timing/environmental, not a 1.6 product regression.

The guard removes the dependency on that accidental buffer entirely by waiting for the actual reconcile, so it is the correct fix regardless of the exact timing.

Some earlier runs (for example 166823784, 06-04) failed fast on DeviceAlreadyExists from a leaf identity orphaned by a prior aborted run, a separate issue addressed by the idempotent-identity change below, not the timeout this PR targets.

Separate, out-of-scope product improvement

A ~15s parent restart can strand an in-flight message for minutes because the edgeHub SDK connection-status callback is disabled and edgeHub falls back to slow connectivity polling. That is a genuine product improvement, tracked separately as Fix B in #7527, and is out of scope here. This PR fixes the test race that makes the failure reproduce in CI.

Changes

1. Reconcile-wait guard (the fix). Adds a guard to the shared nested-deploy-config.yaml template, applied to L5 then L4. After set-modules, it reads the desired edgeAgent $version and polls reported.lastDesiredVersion plus lastDesiredStatus.code, continuing only when the exact desired version reports status 200. On a 300-second timeout it prints lastDesiredStatus.description and reported module states so the two failure modes (deploy never picked up vs picked up but reported non-200) are distinguishable from the pipeline log.

2. Collect L4/L5 support bundles. Adds Collect_Nested_Bundles to publish support bundles and raw module logs from L3, L4, and L5. The existing pipeline only preserved L3 evidence, which is why the parent (L4) edgeHub restart was invisible in earlier investigations. Kept because per-level evidence is broadly useful for nested-edge test debugging.

3. Idempotent leaf-identity creation. Deletes and recreates a leaf device identity orphaned when a prior run dies before teardown (the pipeline's identity cleanup only removes the L4/L5 parent devices, not the test-created leaf). Prevents a spurious DeviceAlreadyExistsException on the shared hub.

Validation

Build	Harness	RouteMessage
171795530	with guard	pass 26.8s (full AMQP suite 25/25)
171618229	with guard	pass 21.7s (full AMQP suite 25/25)
171795768	no guard	timeout 300s

With the guard, RouteMessage runs only after the L4/L5 deployment has reconciled, so the edgeHub restart no longer lands mid-test.

[jlian]

Closing and reopening to try to trigger the license check