Skip to content

test: Check why a blockchain test block was rejected - #1623

Merged
chfast merged 1 commit into
masterfrom
test/blockchain-invalid-verdict
Aug 10, 2026
Merged

test: Check why a blockchain test block was rejected#1623
chfast merged 1 commit into
masterfrom
test/blockchain-invalid-verdict

Conversation

@chfast

@chfast chfast commented Aug 9, 2026

Copy link
Copy Markdown
Member

A blockchain test block expected to be invalid counted as correctly rejected however evmone rejected it, so a block failing a rule other than the one the test is about still passed. The header-mismatch branches were the weakest: any difference from the claimed header ended the case silently.

Compare the rejection against the fixture's exception at every exit: block validation, a rejected transaction, requests collection, and each header mismatch, which now names the BlockException it is the symptom of.

Two known gaps are documented rather than checked: transaction senders come from the fixture instead of the signature, so a broken signature surfaces only as a state-root mismatch, and ommer headers are not validated.

🤖 Generated with Claude Code

@chfast
chfast force-pushed the test/statetest-reject-reason branch from 19d0bc2 to 3d555c5 Compare August 9, 2026 23:47
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 3bc40bc to a5099a1 Compare August 9, 2026 23:52
@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.92683% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.70%. Comparing base (8ec683f) to head (8068e29).

Files with missing lines Patch % Lines
test/blockchaintest/blockchaintest_runner.cpp 80.55% 6 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1623      +/-   ##
==========================================
- Coverage   97.73%   97.70%   -0.04%     
==========================================
  Files         171      171              
  Lines       15565    15593      +28     
  Branches     3604     3606       +2     
==========================================
+ Hits        15213    15235      +22     
- Misses        263      269       +6     
  Partials       89       89              
Flag Coverage Δ
eest-develop 88.61% <77.50%> (-0.09%) ⬇️
eest-develop-gmp 26.66% <4.87%> (-0.05%) ⬇️
eest-legacy 17.19% <65.85%> (+0.17%) ⬆️
eest-libsecp256k1 28.86% <4.87%> (-0.05%) ⬇️
eest-stable 88.61% <77.50%> (-0.09%) ⬇️
evmone-unittests 93.13% <4.87%> (-0.18%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
core 96.08% <ø> (ø)
tooling 91.86% <82.92%> (-0.22%) ⬇️
tests 99.80% <ø> (ø)
Files with missing lines Coverage Δ
test/utils/block_transition.cpp 100.00% <100.00%> (ø)
test/utils/error_matching.cpp 95.83% <100.00%> (+0.37%) ⬆️
test/utils/t8n.cpp 96.74% <100.00%> (ø)
test/blockchaintest/blockchaintest_runner.cpp 83.51% <80.55%> (-0.63%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@chfast
chfast force-pushed the test/statetest-reject-reason branch from 3d555c5 to 3ddecd8 Compare August 10, 2026 06:17
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from a5099a1 to 85903da Compare August 10, 2026 06:24
@chfast
chfast force-pushed the test/statetest-reject-reason branch from 3ddecd8 to 13146b5 Compare August 10, 2026 06:30
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 85903da to 80f22df Compare August 10, 2026 06:30
@chfast
chfast force-pushed the test/statetest-reject-reason branch from 13146b5 to 528d969 Compare August 10, 2026 06:47
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 80f22df to 65c90ed Compare August 10, 2026 06:47
@chfast
chfast force-pushed the test/statetest-reject-reason branch 5 times, most recently from 3271aac to 29faa6c Compare August 10, 2026 13:16
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 65c90ed to 582fff4 Compare August 10, 2026 13:44
@chfast
chfast force-pushed the test/statetest-reject-reason branch from 29faa6c to 04fdfcb Compare August 10, 2026 13:54
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 582fff4 to 0b87e1b Compare August 10, 2026 13:54
Base automatically changed from test/statetest-reject-reason to master August 10, 2026 14:51
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 0b87e1b to 84f53a7 Compare August 10, 2026 15:00
@chfast
chfast requested a balanced review from Copilot August 10, 2026 15:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Tightens blockchain-test rejection checks so invalid blocks must fail for the fixture’s expected reason.

Changes:

  • Preserves structured transaction errors for matching.
  • Validates transaction, request, block, and header-mismatch rejection reasons.
  • Documents signature-recovery and ommer-validation gaps.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
test/utils/t8n.cpp Renders preserved transaction errors.
test/utils/block_transition.hpp Stores rejection error codes.
test/utils/block_transition.cpp Propagates transaction error codes.
test/blockchaintest/blockchaintest_runner.cpp Adds rejection-reason verification.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/blockchaintest/blockchaintest_runner.cpp Outdated
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 84f53a7 to 4b40369 Compare August 10, 2026 15:42
@chfast
chfast requested a balanced review from Copilot August 10, 2026 15:45

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

test/blockchaintest/blockchaintest_runner.cpp:396

  • This disables reason checking for every executed block that contains an ommer, not only fixtures whose expected failure is an unvalidated ommer-header rule. Consequently, an unrelated block with an ommer can fail any header check above for the wrong reason and still be counted as correctly rejected. Gate the exemption on the specific unsupported ommer exception(s), rather than merely on a nonempty ommer list.
                const auto expect_fixture_names = [&](std::string_view names) {
                    if (!names_spec_exception || ommers_not_validated)
                        return;

test/blockchaintest/blockchaintest_runner.cpp:334

  • The new rejection-reason behavior has no regression coverage: the existing test/integration/blockchaintest/*.json cases contain no invalid block with expectException. Add invalid fixtures that verify both a matching reason and a deliberately different reason (including validation, rejected-transaction, and computed-header exits), so future broad matching or silent exits cannot reintroduce false positives.
                    EXPECT_TRUE(
                        is_expected_block_exception(block_error, test_block.expected_exception))

Comment on lines +54 to +58
{state::INCORRECT_BLOCK_FORMAT,
"BlockException.GAS_USED_OVERFLOW|"
"BlockException.IMPORT_IMPOSSIBLE_UNCLES_OVER_PARIS|"
"BlockException.RLP_STRUCTURES_ENCODING|"
"BlockException.RLP_INVALID_FIELD_OVERFLOW_64"},

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the coarseness, and I have added a comment stating it: nine validate_block() branches share INCORRECT_BLOCK_FORMAT, so to the matcher these four names are interchangeable, and separating them needs a distinct code per branch.

Worth noting it is not new behaviour introduced here. Before this commit the loader's map_legacy_exception rewrote those same three spec names to INCORRECT_BLOCK_FORMAT outright, so the interchangeability already existed — it was just hidden in the loader, and it destroyed the fixture's stated expectation on the way. This change removes those rewrites and puts the equivalence in one visible table, which is what makes the granularity gap reviewable at all. Giving the branches distinct codes is a change to state/errors.hpp plus validate_block(), deliberately outside this PR.

A block expected to be invalid only had to differ from its fixture somewhere:
every header mismatch counted as "correctly rejected", so a block that failed
the state root of a test about the receipts root passed just as well. A rejected
transaction only had to be rejected for some reason, with "TransactionException."
appearing anywhere in the expectation.

Each header mismatch is now matched against the BlockException it is the symptom
of, and the transaction reason goes through is_expected_tx_exception(), the same
mapping the state test runner uses. That one takes the error code rather than its
message, so apply_block() reports the code.

Two rules evmone does not implement keep their old verdict, each marked with a
TODO: transaction senders come from the fixture instead of being recovered from
the signature, and ommer headers are not validated.
@chfast
chfast force-pushed the test/blockchain-invalid-verdict branch from 4b40369 to 8068e29 Compare August 10, 2026 16:07
@chfast
chfast merged commit 22c6d37 into master Aug 10, 2026
21 of 23 checks passed
@chfast
chfast deleted the test/blockchain-invalid-verdict branch August 10, 2026 16:39
chfast added a commit that referenced this pull request Aug 10, 2026
Follow-up to #1623. The unrecovered-sender gap was patched only in the
state-root check's names, but the blob-gas check runs before it, so a
block whose bad signature the runner cannot see was judged against a
rule it does not break. Skip the whole ladder instead, as already done
for the unvalidated ommer headers.

No existing fixture reaches that path, so the regression test is a new
one: a Cancun block carrying a blob transaction with a broken signature
and a `blobGasUsed` the execution contradicts. Without the fix it
reports

```
the block failed the check for BlockException.INCORRECT_BLOB_GAS_USED|BlockException.BLOB_GAS_USED_ABOVE_LIMIT,
expected TransactionException.INVALID_SIGNATURE_VRS
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants