CMP-4631: Wait for MachineConfigPools to observe new config before rescanning - #85
Open
yuumasato wants to merge 1 commit into
Open
CMP-4631: Wait for MachineConfigPools to observe new config before rescanning#85yuumasato wants to merge 1 commit into
yuumasato wants to merge 1 commit into
Conversation
After applying dependency-gated remediations, the e2e test's "wait for MachineConfigPools to be updated" gate could return immediately: the MachineConfigPool still reported Updated=True from the *previous* rollout because the Machine Config Operator had not yet reacted to the just-created MachineConfigs. The final rescan then ran while nodes were still rebooting onto staggered MachineConfig generations, so the compliance-operator saw different results per node and aggregated them as INCONSISTENT. usbguard was the most exposed rule set (a two-wave dependency chain), producing 12 spurious INCONSISTENT results across rhcos4-high and rhcos4-stig. Fix, mirroring compliance-operator's WaitForMachinePoolUpdate: - Snapshot each pool's Status.ObservedGeneration before a wave's remediations are applied (snapshotMachineConfigPoolGenerations) and require the pool to advance past that baseline (it has observed a new rendered config) before it can be considered done. A pool that never changes is accepted once a short reaction grace elapses, so genuinely-unchanged pools don't hang. - Tighten isMachineConfigPoolUpdated to also require the targeted rendered config to be realized (Spec.Configuration.Name == Status.Configuration.Name), catching a pool mid-rollout even if machine counts momentarily look settled. Add a unit test for isMachineConfigPoolUpdated covering the mid-rollout case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yuumasato
force-pushed
the
cmp-4631-mcp-convergence-race
branch
from
August 31, 2026 11:56
bfdc656 to
c19019e
Compare
taimurhafeez
approved these changes
Sep 1, 2026
Collaborator
|
/test e2e-aws-openshift-platform-compliance |
Collaborator
|
/test e2e-aws-openshift-node-compliance |
|
@yuumasato: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After applying newly-available (dependency-gated) remediations, the e2e test's "wait for MachineConfigPools to be updated" gate could return immediately: the MachineConfigPool still reported
Updated=Truefrom the previous rollout, because the Machine Config Operator had not yet reacted to the just-created MachineConfigs. The final rescan then ran while nodes were still rebooting onto staggered MachineConfig generations, so the compliance-operator saw different results per node in a pool and aggregated them as INCONSISTENT.usbguardis the most exposed rule set because it is a two-wave dependency chain (install package → reboot → enable service + write rules/auditbackend config → second reboot). This produced 12 spurious INCONSISTENT results acrossrhcos4-highandrhcos4-stig(configure-usbguard-auditbackend,service-usbguard-enabled,usbguard-allow-hid-and-hubfor both master and worker).This is a test-harness timing/rollout-race artifact, not a compliance-operator content or remediation defect (must-gather confirmed all MCPs
Updated, all usbguard MachineConfigs rendered, all nodesDone).Fix
Mirrors compliance-operator's
WaitForMachinePoolUpdate, which keys offStatus.ObservedGenerationcaptured before the change-causing action:snapshotMachineConfigPoolGenerations— snapshot each pool'sStatus.ObservedGenerationbefore a wave's remediations are applied.WaitForMachineConfigPoolsUpdated(tc, c, baseline)— a pool is only "done" once it has advanced its generation past the baseline (i.e. observed a new rendered config) and finished rolling out. A pool that never changed is accepted after a short reaction grace (mcpReactionGrace, 5m), so genuinely-unchanged pools don't hang.isMachineConfigPoolUpdated— additionally require the targeted rendered config to be realized (Spec.Configuration.Name == Status.Configuration.Name), catching a pool mid-rollout even if machine counts momentarily look settled.Test
Added
helpers/mcp_test.go— a unit test forisMachineConfigPoolUpdatedcovering the mid-rollout case (spec config not yet realized in status → not updated), which the previous counts-only check missed. The baseline/reaction-grace loop is exercised by the live e2e run (no fake client is vendored).Depends on #84 for the
make test-unittarget /unitCI job that runs this test.🤖 Generated with Claude Code