Skip to content

[Celestica] [m4061clsc] Fix AgentAclTableGroupTrafficTest warmboot race condition by waiting for post-warmboot stats sync - #1867

Open
gang-tao wants to merge 1 commit into
meta:mainfrom
gang-tao:m4061clsc_agent_fix1
Open

gang-tao wants to merge 1 commit into
meta:mainfrom
gang-tao:m4061clsc_agent_fix1

Conversation

@gang-tao

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • [ ✓ ] I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • [ ✓ ] pre-commit run
clang-format.............................................................Passed
shellcheck...........................................(no files to check)Skipped
shfmt................................................(no files to check)Skipped
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check json...........................................(no files to check)Skipped
check for merge conflicts................................................Passed
ruff check...........................................(no files to check)Skipped
ruff format..........................................(no files to check)Skipped
Check copyright headers in new FBOSS C/C++ files.........................Passed
Prevent sai_impl in fboss manifest.......................................Passed
[INFO] Restored changes from /work/home/gangtao/.cache/pre-commit/patch1793359488-2644314.

Summary

In Split-Agent / Multi-Switch test environments, AgentAclTableGroupTrafficTest test cases (specifically VerifyDscpMarkingAndTtlAclTable and VerifyQueuePerHostAclTableAndTtlAclTable) occasionally failed during warmboot with assertion timeouts:

  /var/FBOSS/fboss/fboss/agent/test/agent_hw_tests/AgentAclTableGroupTrafficTests.cpp:365: Failure
  Expected equality of these values:
    dscpAclPkts - beforeAclPkts.first
      Which is: 280
    (1 * 2 * 2 * (utility::kUdpPorts().size() + utility::kTcpPorts().size()))
      Which is: 40

  /var/FBOSS/fboss/fboss/agent/test/agent_hw_tests/AgentAclTableGroupTrafficTests.cpp:368: Failure
  Expected equality of these values:
    ttlAclPkts - beforeAclPkts.second
      Which is: 100
    (2 * (utility::kUdpPorts().size() + utility::kTcpPorts().size()))
      Which is: 20

Root Cause

In Split-Agent architecture, hardware counters are asynchronously published by HwAgent to SwSwitch via the syncHwStats Thrift stream every second.
During cold boot, the test executes a multi-second setup() routine, allowing ample time for initial hardware statistics to synchronize.
However, during warm boot, setup() is skipped and the test runner immediately enters verify(), calling pktCounterHelper() within milliseconds of SwSwitch initialization. At this instant, SwSwitch has not yet received its first post-warmboot syncHwStats snapshot from Agent, causing getAclInOutPackets to return a default baseline of 0.
When HwAgent subsequently pushes the first statistics event (~100ms later), it carries the cumulative ASIC counters inherited across warm boot (DSCP: 240, TTL: 80) plus newly injected packets (DSCP: 40, TTL: 20), resulting in dscpAclPkts - beforeAclPkts.first = 280 - 0 = 280 expected 40), triggering a test timeout failure.

Solution

In AgentAclTableGroupTrafficTests.cpp, invoke this->getNextUpdatedPortStats(this->masterLogicalPortIds()[0]) before querying initial ACL counters in both _verifyHelperDscpTtlAclTables and _verifyHelperMultipleAclTables.
This ensures the test runner blocks until HwAgent publishes a fresh post-warmboot statistics snapshot with updated timestamps, guaranteeing the baseline counter accurately reflects hardware state before packet transmission begins.

This follows the exact same established pattern as commit 186ac01 : Wait for HW stats before ACL baseline in AgentQueuePerHostTest) and AgentMPLSTtlTests, ensuring the test runner blocks until HwAgent publishes a fresh post-warmboot statistics snapshot before packet transmission begins.

Test Plan

Both test cases were verified on icephoton unit with 100% pass rate across multiple Coldboot + Warmboot iterations:

  1. AgentAclTableGroupTrafficTest.VerifyDscpMarkingAndTtlAclTable (3 consecutive runs):
    • Run 1:
    [ PASSED ] cold_boot.AgentAclTableGroupTrafficTest.VerifyDscpMarkingAndTtlAclTable (23668 ms)
    [ PASSED ] warm_boot.AgentAclTableGroupTrafficTest.VerifyDscpMarkingAndTtlAclTable (12130 ms)
    • Run 2:
    [ PASSED ] cold_boot.AgentAclTableGroupTrafficTest.VerifyDscpMarkingAndTtlAclTable (23580 ms)
    [ PASSED ] warm_boot.AgentAclTableGroupTrafficTest.VerifyDscpMarkingAndTtlAclTable (12127 ms)
    • Run 3:
    [ PASSED ] cold_boot.AgentAclTableGroupTrafficTest.VerifyDscpMarkingAndTtlAclTable (23857 ms)
    [ PASSED ] warm_boot.AgentAclTableGroupTrafficTest.VerifyDscpMarkingAndTtlAclTable (12130 ms)

  2. AgentAclTableGroupTrafficTest.VerifyQueuePerHostAclTableAndTtlAclTable (2 consecutive runs):
    • Run 1:
    [ PASSED ] cold_boot.AgentAclTableGroupTrafficTest.VerifyQueuePerHostAclTableAndTtlAclTable (25331 ms)
    [ PASSED ] warm_boot.AgentAclTableGroupTrafficTest.VerifyQueuePerHostAclTableAndTtlAclTable (14132 ms)
    • Run 2:
    [ PASSED ] cold_boot.AgentAclTableGroupTrafficTest.VerifyQueuePerHostAclTableAndTtlAclTable (25865 ms)
    [ PASSED ] warm_boot.AgentAclTableGroupTrafficTest.VerifyQueuePerHostAclTableAndTtlAclTable (14133 ms)

@gang-tao
gang-tao requested a review from a team as a code owner October 30, 2026 11:51
@meta-cla meta-cla Bot added the CLA Signed label Oct 30, 2026
…ce condition by waiting for post-warmboot stats sync
@gang-tao
gang-tao force-pushed the m4061clsc_agent_fix1 branch from e7033f5 to 333bae7 Compare October 30, 2026 12:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant