Skip to content

[WIP] sszgen fixes found while FVing - #26

Closed
syjn99 wants to merge 9 commits into
OffchainLabs:mainfrom
syjn99:syjn99/sszgen-fv-fixes
Closed

syjn99 wants to merge 9 commits into
OffchainLabs:mainfrom
syjn99:syjn99/sszgen-fv-fixes

Conversation

@syjn99

@syjn99 syjn99 commented Nov 12, 2026

Copy link
Copy Markdown
Member

No description provided.

syjn99 and others added 9 commits November 12, 2026 19:04
For Bitvector[N], when N is not a multiple of 8, the spec serializer
does not write the top 8 - N%8 bits of the last byte. These bits are
always 0 (consensus-specs ssz/simple-serialize.md, EthLib paddingBits
error). The generated decoder did not check these bits. It accepted
0x10 for Bitvector4. A formal-verification counterexample found this.

- ssz.ValidateBitvector(buf, bits): checks that the length is
  (bits+7)/8 and that the padding bits are 0.
- Decoder: adds a ValidateBitvector call for each go-bitfield
  BitvectorN with N%8 != 0 (at this time, only Bitvector4). When
  N%8 == 0, the generated code does not change.
- The encoder also rejects the same values. The EthLib value layer has
  no padding bits. Thus a Go value with a padding bit set does not
  represent a spec value. If the encoder emitted such a value, it would
  make bytes that the decoder of the same type rejects. This is the
  same kind of bug as the Bitlist length check in the marshaler.
- The hash tree root (HTR) does not change. It still uses hh.PutBytes,
  which has no error path.

Tests: ssz TestValidateBitvector (N = 4, 9, 16, padding 0 or 1, wrong
length). TestBitvectorPadding in a new compiled fixture,
sszgen/testutil/genfixture. The generated file is committed, and
sszgen.TestGenFixture pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Bitlist encoder used tmplGenerateMarshalValueList
(`len(field) > MaxSize`). field is a byte array, but MaxSize is a limit
in bits. Thus the encoder encoded bitlists of 258 to 2048 bytes without
an error, and the decoder of the same type (ssz.ValidateBitlist)
rejected that output. The encoder also accepted an empty array and a
last byte of 0. A formal-verification counterexample found this.

The spec serializer (consensus-specs ssz/simple-serialize.md
BitList[N]) emits e//8+1 bytes and puts the delimiter bit at bit e.
Thus the encoding is not empty, the last byte is not 0, and e <= N.
This rule is the same as ssz.ValidateBitlist in the decoder.

- Non-progressive bitlist: marshal calls ssz.ValidateBitlist(field,
  limit).
- Progressive bitlist: marshal calls ssz.ValidateProgressiveBitlist(field).
  Like other progressive lists, it does not check a limit. The spec has
  no limit.
- A bitlist in a list or vector element position uses the same path.

Tests: genfixture TestBitlistLimit (empty array, last byte 0, 2048
bits, 2049 bits, 258 bytes, 0 bits). The test checks that the decoder
and the encoder give the same result. It also checks the round trip of
the accepted values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For a Go array vector field such as [N]uint64, UnmarshalSSZ emitted
`c.A = make([]uint64, N)`. This code did not compile (unmarshalVectorTmpl
in sszgen/render/unmarshal.go). An array destination already has N
elements. Thus the decoder now emits no make and writes the elements in
place. The vector template for variable-size elements gets the same
condition.

Tests: genfixture gets Limbs{A [4]uint64}. The generated code compiles.
TestUint64Array checks the spec serialization (4 little-endian uint64
values), the round trip and the root. The value is one chunk, so the
root is equal to the serialization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The container decoder checked the first variable offset with
`off != fixedSize → ErrInvalidVariableOffset` and then with
`off > size → ErrOffset`. The size check at the start of UnmarshalSSZ
already returns ErrSize when size < fixedSize. Thus, at the second
check, off == fixedSize <= size. No input reaches the ErrOffset branch.
An example is the dead ErrOffset in ListU64. No meaningful input can
make this branch reachable, so this change removes it. The behavior
does not change.

Tests: genfixture gets ListU64{A uint64; L []uint64 ssz-max:"4"}.
TestListU64FirstOffset checks that the decoder accepts offset 12. It
also checks that the decoder rejects 0, 11, 13, 20 and 0xffffffff with
ErrInvalidVariableOffset (EthLib/remerkleable: first offset = size of
the fixed part). This change regenerates the generated golden files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The generated MarshalSSZTo writes the offset table and the earlier
fields before it checks the lengths of later fields (ErrBytesLength,
ErrListTooBig and others). Examples are Scalars, ListU64 and FixedVec.
On error, partial output stays in the array past len(dst).

This change documents the behavior (option b). It does not move the
checks before the writes (option a). Reasons:
- All writes are appends, so only the array past len(dst) changes. On
  error, the function returns nil. Thus dst[:len(dst)], which the
  caller sees, does not change.
- Option a needs a separate pass that checks the full value before any
  write. The MarshalSSZTo of nested containers and of delegated types
  writes before its own checks. Thus, if only the top-level checks move
  to the start, partial writes still occur. Option a is not the
  shortest change.

The ssz.Marshaler documentation now states the promise (nil result,
dst[:len(dst)] does not change) and the unspecified range (past
len(dst)). The test TestMarshalSSZToErrorKeepsPrefix pins only the
promised part.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
merkleizeChunks in ssz/htr.go calls gohashtree.Hash(elements,
elements). The input and the output use the same array. The
formal-verification work did not prove that this aliasing is safe. It
was a premise.

To check it, I read the source of gohashtree v0.0.5-beta (the version
in go.mod). The results are:
- generic: reads chunks 2k and 2k+1, then writes digest k, in k order.
- amd64 avx1 x4/x1, avx2 x8, avx512 x16, shani x2/x1: each loop
  completes all DATA_PTR loads of the batch before it stores to
  OUTPUT_PTR.
- arm64 neon x4/x1, sha2 x1: the same. In x4, the WORD-encoded
  instructions are LD4 (base R1 = DATA_PTR, in rounds 0-15) and ST4
  (base R0 = OUTPUT_PTR, at the end).
A batch k0..k0+L-1 writes only below k0+L. The next batch reads from
2(k0+L). Thus the in-place call is safe. A comment on that line records
this result.

Note: an arm64 host does not run only the generic path.
hash_arm64.go sets supportedCPU = true. The test host (Apple M5 Pro)
has cpuid SHA2 = true, so it uses the arm64 sha2 path. With
GOARCH=amd64 (Rosetta, no AVX2 or SHA), it uses the generic path.

Test TestMerkleizeInPlaceHash: for 1..40 pairs, the in-place call, a
call with a separate output, and crypto/sha256 (the spec definition)
give the same result. The test passed on both paths (arm64 sha2 and
generic).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When bits >= MaxUint64-6, the uint64 addition in `(bits+7)/8`
overflows and the result is 0..6. Then an empty buf passes the length
check, and the next line reads buf[-1] and panics. A review found this.
This change computes the byte count from bits/8 and bits%8.

Tests: TestValidateBitvector gets new cases. It checks the error return
for N = MaxUint64, MaxUint64-6 (empty buf) and MaxUint64-3 (1 byte). It
also checks that N = 0 with an empty buf is accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a successful decode, checkBits encoded the input `in` again.
Thus the test passed even when the decoder succeeded but damaged the
value. A review found this: the test did not detect the mutation
`c.V4[0] = 0`. Now the test compares the fields of the decoded `dec`
with the input. It also encodes `dec` again and compares the result
with `enc`. I put in the same mutation and checked that the test now
fails.

This change also adds a compile and run test for arrays with
variable-size elements. genfixture gets Inner{B []byte ssz-max:"8"}
and InnerPair{P [2]*Inner}. TestVariableElementArray compares the
encoding with bytes made by hand from the spec serialization (outer
offset 4, element offsets 8 and 14). It also checks the decoded values
and the rejection of a wrong first element offset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A review asked for a narrower supported scope in the comments. This
change narrows it.

- ssz.Marshaler: the comment now says that the "dst[:len(dst)] does not
  change" promise of the generated code holds only when each delegated
  MarshalSSZTo that it calls keeps the same contract.
- render marshalOp.Overlay: the check applies only to go-bitfield
  types. Delegated types and HTR are out of scope. core.Dispatch
  delegates before Overlay. HTR still hashes a BitvectorN with padding
  bits set.
- Progressive bitlist: the encoder applies no limit. The decoder
  applies a local limit when the field has ssz-max. The comment now
  records this existing asymmetry.

The generated code does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@syjn99

syjn99 commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

Superseded by #27

@syjn99 syjn99 closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant