Poll the chaincode container listings in the e2e suite - #5557
Merged
Conversation
The e2e suite reads docker's running-container list once and asserts on it in four places, and each single read races the peer. The spec "recreates terminated chaincode containers" removes every chaincode container, invokes once per peer so that each peer relaunches its chaincode, and then lists the running chaincode containers, expecting one per peer. The peer treats a chaincode as launched as soon as the chaincode process registers over gRPC: RuntimeLauncher.Launch returns on launchState.Done() while the goroutine that called DockerVM.Start is still inside ContainerStart, and dockerd only adds a container to the running list at the end of its start path, after persisting the container state. In that window the chaincode is registered and endorsing but `docker ps` does not list it. Endorsement, ordering, commit and event delivery all happen on localhost and fit inside the window whenever dockerd's tail takes more than a few tens of milliseconds, so the listing taken right after the last invoke can come back one container short: Expected <[]container.Summary | len:3, cap:4>: [...] to have length 4 Recording docker events alongside the spec shows dockerd emitting its `start` event for each relaunched container about 20 ms after the peer has already finished executing chaincode inside it, and in failing runs the last container's `start` event is timestamped after the listing. On hosts with a slow dockerd start path (docker inside an unprivileged LXC container on ZFS) the spec fails about half the time, on main and on unrelated branches alike; the other e2e specs pass every time. The spec "basic lifecycle operations for chaincode install and update" has the same shape twice, listing the containers right after the init invoke that launches them, and a third read in the other direction: after a definition upgrade it expects the superseded containers to be gone, but the peer stops those from the lifecycle custodian's background worker, not as part of block commit, so nothing orders that stop before the listing either. Poll all four listings with Eventually through a shared helper, as the rest of the suite does for side effects, and check the recreated container IDs against the originals as before. Assisted-by: anthropic:claude-fable-5-1 Signed-off-by: Ry Jones <ry@linux.com>
pfi79
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The e2e suite reads docker's running-container list once and asserts on it in four places, and each single read races the peer.
The spec "recreates terminated chaincode containers" removes every chaincode container, invokes once per peer so that each peer relaunches its chaincode, and then lists the running chaincode containers, expecting one per peer. The peer treats a chaincode as launched as soon as the chaincode process registers over gRPC: RuntimeLauncher.Launch returns on launchState.Done() while the goroutine that called DockerVM.Start is still inside ContainerStart, and dockerd only adds a container to the running list at the end of its start path, after persisting the container state. In that window the chaincode is registered and endorsing but
docker psdoes not list it. Endorsement, ordering, commit and event delivery all happen on localhost and fit inside the window whenever dockerd's tail takes more than a few tens of milliseconds, so the listing taken right after the last invoke can come back one container short:Expected <[]container.Summary | len:3, cap:4>: [...] to have length 4
Recording docker events alongside the spec shows dockerd emitting its
startevent for each relaunched container about 20 ms after the peer has already finished executing chaincode inside it, and in failing runs the last container'sstartevent is timestamped after the listing. On hosts with a slow dockerd start path (docker inside an unprivileged LXC container on ZFS) the spec fails about half the time, on main and on unrelated branches alike; the other e2e specs pass every time.The spec "basic lifecycle operations for chaincode install and update" has the same shape twice, listing the containers right after the init invoke that launches them, and a third read in the other direction: after a definition upgrade it expects the superseded containers to be gone, but the peer stops those from the lifecycle custodian's background worker, not as part of block commit, so nothing orders that stop before the listing either.
Poll all four listings with Eventually through a shared helper, as the rest of the suite does for side effects, and check the recreated container IDs against the originals as before.
Type of change
Related issues
Fixes #5556