test: persist nested-e2e edgeHub store across container recreate (diagnostic)

[jlian]

Summary

The nested-E2E edgeHub stores store-and-forward messages at the default /tmp/edgeHub, which lives in the container's ephemeral writable layer. When the AMQP-group redeploy recreates the L4 edgeHub container mid-test, any queued or undelivered message is lost with the wiped store.

This mounts a host directory (/etc/iotedge/storage) into edgeHub and points its StorageFolder at it, so the message store survives an edgeHub container recreate. edgeHub's entrypoint (hubStart.sh) creates and chowns $StorageFolder/edgeHub to the edgehub uid at startup, so no host-side preparation is needed.

Applies to both middle-layer AMQP deployments (nestededge_middleLayerBaseDeployment_amqp.json and nestededge_middleLayer_e2e_amqp.json).

Why (diagnostic)

Investigating why RouteMessageL3LeafToL4Module fails, the message the test waits on is stranded when the L4 edgeHub container is delete+recreated by the mid-test deployment reconcile. Because the store is ephemeral, the queued message is destroyed with the old container rather than being retried on reconnect.

This change is a diagnostic to separate two hypotheses:

• If RouteMessage passes with a persistent store, the failing message was purely lost to the container-recreate store wipe (a test-rig gap).
• If it still stalls, the message is orphaned within a surviving store, which isolates the defect to edgeHub's store-and-forward re-pump on reconnect.

Draft scope

Draft, not for merge as-is. This is a targeted experiment to inform the durable edgeHub store-and-forward fix. If persistent storage is the right long-term posture for the nested-E2E rig (arguably it is, since store-and-forward across a restart is a real customer scenario), the mount path and provisioning should be reviewed for the CI agents.

[jlian]

Result of the diagnostic run (172076620, this branch @ f3c7076, ci-build 20260706.1 = plain main, no #7528): RouteMessage still failed at +300s, but the run does not actually test the hypothesis, for two reasons.

1. The bind landed on the wrong container. The mount + StorageFolder were added to the AMQP deployment files only. But the L4 edgeHub that receives the message during the pre-restart window is the MQTT-base container (iotedgeApiProxy, created 23:52:54), which has no bind, so its store is still ephemeral. Deployment 3 (AMQP, with the bind) then removes that MQTT container at 00:29:49 and creates the AMQP one at 00:30:02, i.e. the persistent mount only exists on the container born after the store was already wiped.

2. The message never queues for relayer1 anyway. relayer1 + the ToRelayer1 route only exist in the AMQP deployment, which isn't applied until 00:29:55, while the leaf sends at 00:28:32. So the message goes upstream un-relayed regardless of store persistence (see the edgeHub: resume store-and-forward promptly after reconnect #7528 thread for the full timeline).

To make this a valid experiment it would need the bind on the MQTT base too (shared host path) and relayer1 + route present continuously so the message actually queues before the recreate. Given that the RouteMessage failure is a routing/reconcile-timing race (fixed by the merged guard #7523) and Q2 is separately provable in isolation, I don't think this diagnostic is on the critical path anymore. Suggest closing this unless we specifically want to nail the container-recreate message-loss mechanism for its own sake.