Repository navigation
docs: refactor CONTRIBUTING.md and split out TESTING.md #1139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
andrew-fleming
wants to merge
11
commits into
OpenZeppelin:main
Choose a base branch
from
andrew-fleming:refactor-contributing
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
629f5a9
refactor CONTRIBUTING
andrew-fleming 6f1de5a
move testing doc to its own md
andrew-fleming 7b322c0
update CONTRIBUTING links
andrew-fleming f4b301e
Merge branch 'OpenZeppelin:main' into refactor-contributing
andrew-fleming ff50eda
add comming soon to broken links, apply sem linefeeds
andrew-fleming 07304b5
Merge branch 'refactor-contributing' of https://github.com/andrew-fle…
andrew-fleming 7b87c68
fix unit test comment
andrew-fleming 04f473b
fix contributing example
andrew-fleming 96fd9ae
fix links
andrew-fleming 98eb033
remove anticipated links
andrew-fleming 857c19d
Merge branch 'main' into refactor-contributing
andrew-fleming File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| blank_issues_enabled: false | ||
| contact_links: | ||
| - name: OpenZeppelin Midnight Contracts Community Support | ||
| url: https://github.com/OpenZeppelin/midnight-contracts/discussions | ||
| url: https://github.com/OpenZeppelin/compact-contracts/discussions | ||
| about: Please ask and answer questions here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| # Testing | ||
|
|
||
| How to run the test suites locally and how the live suite runs in CI. What every change must | ||
| be tested with lives in [`GUIDELINES.md`](GUIDELINES.md#testing); the coverage gate lives in | ||
| [`CONTRIBUTING.md`](CONTRIBUTING.md#code-quality-standards). | ||
|
|
||
| Run all commands from the repository root. Enable Corepack once (`corepack enable`) so `yarn` | ||
| resolves to the version pinned in `package.json`. | ||
|
|
||
| ## Unit tests | ||
|
|
||
| Unit tests run against an in-process mock backend: no network, ZK proving skipped. | ||
|
|
||
| ```bash | ||
| yarn test | ||
| yarn test:coverage # run locally (not yet a CI check) | ||
| ``` | ||
|
|
||
| ## Integration tests | ||
|
|
||
| Composed-contract specs under `contracts/test/integration/specs`: several modules assembled | ||
| into one contract under `test/integration/_mocks`, then deployed and driven as a unit. Same | ||
| mock backend as the unit tests. | ||
|
|
||
| ```bash | ||
| yarn test:integration | ||
| ``` | ||
|
|
||
| ## Live tests | ||
|
|
||
| Live tests run against a local Midnight network (node, indexer and proof server) defined in | ||
| [`local-env.yml`](local-env.yml). They need [Docker](https://docs.docker.com/get-docker/) and a | ||
| completed `yarn install`. The runner builds the target's contracts with real ZK keys, which is | ||
| slow; do not interleave `SKIP_ZK=true` builds with live runs, since a key-less build forces a | ||
| full regeneration. | ||
|
|
||
| ```bash | ||
| yarn test:live # every live category, sequentially | ||
| yarn test:live multisig # one category | ||
| yarn test:live multisig ShieldedTreasury # files within a category | ||
| yarn test:live integration # the composed-contract specs (or: yarn test:integration:live) | ||
| yarn test:live --list # the live targets, as the CI matrix reads them | ||
| ``` | ||
|
|
||
| A run compiles the target's slice, starts the stack (`make env-up`, itself a reset), runs a | ||
| harness smoke, then the specs, each category on a freshly reset node. The runner stops the | ||
| stack on every exit path, Ctrl-C included; no manual `env:up` or `env:down` is needed. | ||
| `MIDNIGHT_LIVE_KEEP_ENV=1` leaves it running for inspection. Container logs land in `logs/` | ||
| either way. | ||
|
|
||
| ### Targets and filters | ||
|
|
||
| Every `contracts/src/<category>` with tests is a live category; today that is `access`, | ||
| `crypto`, `multisig`, `security`, `token` and `utils`. A category that must not run live is | ||
| an explicit opt-out in `EXCLUDED_CATEGORIES` in [`scripts/live/targets.ts`](scripts/live/targets.ts) | ||
| (only the legacy `archive` today). | ||
|
|
||
| `integration` is a target of its own, not a category: an unscoped run skips it, so ask for it | ||
| by name. Only one target runs per invocation, since both live projects draw wallets from the | ||
| same genesis-funded pool. The integration live block deploys the composed contract and reads | ||
| its supply back; the functional integration specs stay dry-only because they mutate private | ||
| state mid-test, which the live backend rejects. | ||
|
|
||
| The first argument names the target. Any further argument is a filename filter, matched the | ||
| way vitest matches it (a case-insensitive substring of the path) against that target's own | ||
| spec files, so a name that prefixes others runs all of them. In an unscoped run, a target the | ||
| filter matches nothing under is skipped; a filter that matches nothing anywhere aborts. | ||
|
|
||
| ### Flake check and exit codes | ||
|
|
||
| If any files fail, a second round re-runs just those files on a fresh node with one worker: | ||
|
|
||
| - Fails round 1, passes round 2: **FLAKY** (exit 0, reported loudly). | ||
| - Fails both rounds, or never reports in round 2: **REAL** (exit 1). | ||
| - Infrastructure abort (stack, compile, lock): exit 2. | ||
|
|
||
| The same semantics apply to a scoped run, so a green result means the same thing as for the | ||
| full suite. | ||
|
|
||
| ### Fresh node rules | ||
|
|
||
| All live tests share one node, so state left by an earlier run can fail a later one. A guard | ||
| enforces two rules and fails fast, before any wallet build: | ||
|
|
||
| 1. **Start from a fresh node.** Stale state makes shielded spends fail with node | ||
| `Custom error: 103`. The guard aborts if it finds any shielded coin event beyond genesis. | ||
| The runner resets for you; reset manually with `yarn env:up`. | ||
| 2. **One live run at a time.** A pid-stamped lock (`contracts/logs/.live-run.lock`) makes a | ||
| second concurrent run abort. | ||
|
|
||
| | Variable | Default | Effect | | ||
| | --- | --- | --- | | ||
| | `MIDNIGHT_LIVE_WORKERS` | 3 | Parallel spec files under `unit-live` (max 3, one genesis-funded deployer each). | | ||
| | `MIDNIGHT_LIVE_ALLOW_DIRTY` | unset | `1` skips the freshness check. | | ||
| | `MIDNIGHT_LIVE_MAX_COIN_EVENTS` | 0 | Coin events beyond genesis tolerated before "not fresh". | | ||
| | `MIDNIGHT_LIVE_MAX_SCAN_BLOCKS` | 3600 | Above this indexer head, the guard asks you to `env:up` rather than scan. | | ||
| | `MIDNIGHT_LIVE_KEEP_ENV` | unset | `1` leaves the stack running after the run. | | ||
|
|
||
| ### Reading the output | ||
|
|
||
| `unit-live` runs up to 3 workers in parallel, so output interleaves. It is tagged per worker: | ||
| a `▶ live worker N/3 ready` banner when a worker's wallets are funded, a `[wN] ❯ <file>` line | ||
| as each spec file starts, and a `[wN] ✓ <test> (<ms>) [done/total]` line per test. Each worker | ||
| also writes `logs/live-harness-wN.log`. `integration-live` runs one worker. | ||
|
|
||
| To keep a readable log, force color and pipe to `tee`: piping makes vitest print one line per | ||
| result instead of a spinner, and `FORCE_COLOR=1` keeps the color. | ||
|
|
||
| ```bash | ||
| FORCE_COLOR=1 yarn test:live multisig 2>&1 | tee logs/live-multisig.ansi | ||
| ``` | ||
|
|
||
| Render the `.ansi` file rather than reading it raw: `less -R` in a terminal, or an ANSI | ||
| extension such as [`iliazeus.vscode-ansi`](https://marketplace.visualstudio.com/items?itemName=iliazeus.vscode-ansi) | ||
| in VS Code. On Linux, prefix `systemd-inhibit --why="live tests"` for a long run. | ||
|
|
||
| ### Live tests in CI | ||
|
|
||
| The live suite takes hours, so [`live.yml`](.github/workflows/live.yml) runs it separately | ||
| from the PR checks and is never a required check: | ||
|
|
||
| - **Nightly** on `main`. A failed nightly opens (or comments on) a `live-nightly` tracking | ||
| issue, which closes automatically on the next green run. | ||
| - **On demand**: run `Live Test Suite` from the Actions tab (or `gh workflow run live.yml`), | ||
| optionally scoped with the `target` and `filter` inputs. | ||
| - **On a PR**: apply a `live-tests:<target>` label (e.g. `live-tests:multisig`), or | ||
| `live-tests:all` for every target. Re-apply the label for a fresh run after new pushes. The | ||
| bare `live-tests` label is rejected by the plan job. | ||
|
|
||
| The run is one pipeline per target. A plan job asks the runner which targets exist and which | ||
| spec files each holds. Each target gets a compile job that builds its slice once with real ZK | ||
| keys, feeding a matrix with one job per spec file that runs its file on its own runner and | ||
| stack. A file whose estimated duration exceeds the leg budget (30 tests at ~55 s each, or the | ||
| previous run's measured timings when available) is split by its describe blocks into several | ||
| legs via [`scripts/ci/split.ts`](scripts/ci/split.ts). `KNOWN_TARGETS` in `live.yml` must | ||
| list every target; a new `src/` category without a job pair fails the plan in seconds. The | ||
| reasoning behind this layout is in the header comment of `live.yml`. | ||
|
|
||
| Every job runs the same `yarn test:live` entry point a local run does, so the flake semantics | ||
| are identical. The two flags CI adds are `--compile-only` (build the slice and stop) and | ||
| `--prebuilt` (run against a downloaded build, and abort rather than compile if it is | ||
| incomplete). A `filter` input narrows the matrix as well as the run, matched exactly as vitest | ||
| does. Each job's verdict lands in its job summary; JSON reports upload as a | ||
| `live-reports-<target>-<file>` artifact on every run, and service and worker logs as | ||
| `live-logs-<target>-<file>` on failure. | ||
|
|
||
| The workflow holds no logic: matrix resolution and nightly reporting live in | ||
| [`scripts/live-ci.ts`](scripts/live-ci.ts), so both are unit tested and runnable locally. | ||
|
|
||
| ```bash | ||
| yarn test:scripts # dry unit tests for scripts/live and scripts/ci | ||
| yarn types:scripts # type-check them (also part of `yarn types`) | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 blocking: this PR's own title (
Refactor contributing.md) does not follow the rule it adds, and the squash commit takes the PR title. Retitle to e.g.docs: refactor CONTRIBUTING.md and split out TESTING.md.added by claude (dev3-code-review skill)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
edited the title