feat(tests): add Leios endorser block log test - #3642
Merged
Conversation
Check that the node reports the expected Leios endorser block activity in its logs: EB announcements, votes and certificates on every pool, and forging, announcing, storing and certifying an EB on at least one pool. The producer side messages are reported only by the pool that won the EB election, so they cannot be required on every pool log. The search waits for new blocks in rounds. Each round searches only the part of a log that was appended since the previous round, and only for the messages that are still missing there, so waiting for many blocks doesn't mean reading the whole log over and over. A stalled chain and a log file that could not be searched are reported together with the messages that were missing, so that a failure is not just "no EB messages found". The test is skipped outside the Dijkstra era and when no Tx load generator is enabled - an EB is forged from exactly the mempool backlog that doesn't fit into a Praos block, so without load there is no EB, no announcement, no votes and no certificate. For the same reason the block production rate is checked once a cluster instance is assigned: on variants that produce a block every few seconds the mempool never builds up a backlog. Add `ENABLE_TX_CENTRIFUGE` and `ENABLE_TX_FIREHOSE` to the configuration, so far they were read only by `runner/regression.sh`.
The Leios tests cannot be marked `testnets` - they need the node logs of a local cluster and wait for many blocks - so the Leios regression run, which selects `testnets`, would not run them. Mark them `leios` and select that marker as well, in the launcher script and in the workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
test_leios_blocks.pywith a test that checks the node reports the expectedLeios endorser block activity in its logs. The traces live in two namespaces,
Consensus.LeiosKernel.*andConsensus.LeiosPeer.*, and the test requires themin two tiers. Every pool must report the messages that all block producers take
part in - EB announcements from peers and accepted announcements, acquired EB
bodies, txs and votes, its own vote and the assembled certificate. Forging,
announcing, storing and certifying an EB is reported only by the pool that won
the EB election, so those are required in the log of at least one pool. The
namespaces are anchored with
\]so thatCertifieddoes not also matchCertifiedAndAnnounced.The test is skipped outside the Dijkstra era, where the traces don't exist, and
when no Tx load generator is enabled. An EB is forged from exactly the mempool
backlog that doesn't fit into a Praos block, so with no load there is no EB, no
announcement, no votes and no certificate. For the same reason the block
production rate is checked once a cluster instance is assigned, which cannot be
a
skipifmarker:leios_fastproduces a block every 20 seconds and gets EBs,while the 10x faster block rate of
local_fastdrains the mempool.The search waits for new blocks in rounds. Each round searches only the part of
a log that was appended since the previous round, and only for the messages that
are still missing there, so waiting for many blocks doesn't mean reading the
whole trace heavy log over and over, and a pool whose messages are all found is
not searched at all. The new search position is recorded before the search, so
that lines appended while it runs are not skipped in the next round. A stalled
chain and a log file that could not be searched are reported together with the
messages that were missing, so a failure is not just "no EB messages found".
The test cannot be marked
testnets- it needs the node logs of a local clusterand waits for many blocks - so the Leios regression run, which selects
testnets, would not run it. Add aleiosmarker, mark the module with it andselect it as well in
scripts/test_leios.shand in the Leios workflow.ENABLE_TX_CENTRIFUGEandENABLE_TX_FIREHOSEare added to the configuration,so far they were read only by
runner/regression.sh.The expected messages were verified against the cluster logs of the 2026-08-18
leios_fastregression run - all of them are present in each of the three poollogs. The skip paths were verified against a running cluster. The test itself has
not been run against a live Dijkstra cluster with a load generator.