feat: add --json flag for machine-readable output

[tamtamchik]

Summary

Adds --json / -J: instead of the human-readable log, stdout carries a single JSON report meant for scripts and coding agents. Logs still go to digest/<ts>/logs.txt (path included in the report), prompts are skipped, and the exit code is unchanged.

The format is specified in docs/json-output.md (linked from README, docs/cli.md, CLAUDE.md, the debug-diff skill, and --help). Highlights:

• top level: status (passed | failed | error), exit_code, duration_seconds, log_file, summary counters, flat contracts list with a config field per entry
• per contract: source / bytecode with status, facets + reason of the matching allowed_diffs rule, diff hunks with line ranges (only files that differ or are missing), uncovered bytecode ranges, compile/simulation error text, and a ready suggested_rule
• keys with null or empty values are omitted, so a clean contract is just {"status": "exact", "files": 26}

A fatal error (missing config, bad token) still produces a report with status: "error"; the traceback goes to stderr.

Side finding

With fail_on_bytecode_comparison_error: false, a contract that errors out (e.g. explorer 401) was silently dropped from all stats and the run exited 0. Such errors are now recorded in the result and surface in the JSON as contracts[].error / summary.contract_errors, turning status into error. The exit code itself is not changed here.

Other changes

• Logger.stdout_enabled switch; two raw print calls in binary_verifier.py routed through it so JSON mode stays clean
• config discovery extracted into _collect_config_paths

Verification

• 395 tests pass, black, mypy, pre-commit green
• real runs: lido-earn/mainnet/earn-factory.yaml gives clean JSON and empty stderr vs 422 log lines; hypernative-guard/mainnet/config.yaml reports 4 contracts allowed via immutables in 86 lines
• independent review of the first commit with codex exec review --uncommitted: no actionable bugs found

[Copilot]

🟡 Changes recommended

Interrupted runs and unmatched contract filters can produce incorrect documented JSON statuses.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds machine-readable JSON output for automated Diffyscan consumers.

Changes:

• Adds --json / -J, structured reports, and suppressed stdout logging.
• Exposes comparison details and previously swallowed contract errors.
• Documents and tests the JSON schema.

File summaries

File	Description
diffyscan/diffyscan.py	Implements JSON reporting and error aggregation.
diffyscan/utils/logger.py	Adds stdout suppression.
diffyscan/utils/binary_verifier.py	Routes output through the logger.
tests/test_json_output.py	Tests JSON generation and CLI behavior.
tests/test_diffyscan_allowlist_runtime.py	Updates expected error metadata.
docs/json-output.md	Specifies the JSON schema.
docs/cli.md	Documents the new flag.
docs/how-to.md	Adds automation guidance.
README.md	Links the JSON documentation.
CLAUDE.md	Adds a machine-readable command example.
.claude/skills/debug-diff/SKILL.md	Recommends JSON for automated debugging.

Review details

• Files reviewed: 11/11 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[TheDZhon]

Codex:

---

Two findings at 8e9a74c:

1. [P2] Fatal errors discard completed contract results. If one contract finishes comparison and fetching the next raises, process_config() never returns and its accumulated results never reach all_results. Reproduced a source mismatch followed by an explorer 401: JSON returned summary: {} and contracts: [], losing the detected mismatch. Preserve partial results when reporting the fatal error. Code (

   diffyscan/diffyscan/diffyscan.py

   Lines 1035 to 1056 in 8e9a74c

   	try:
   	for config_path in _collect_config_paths(args.path):
   	result = process_config(
   	config_path,
   	args.support_brownie,
   	enable_binary_comparison,
   	args.cache_explorer,
   	args.cache_github,
   	args.yes or args.json,
   	args.contract_filter,
   	)
   	all_results.append(result)
   	if result["interrupted"]:
   	# Partial results must never read as a verified run
   	error = "KeyboardInterrupt: run interrupted by user"
   	break
   	except Exception as exc:
   	# In JSON mode the report must still reach stdout; keep the traceback on stderr.
   	if not args.json:
   	raise
   	traceback.print_exc()
   	error = f"{type(exc).__name__}: {exc}"

   )
2. [P2] JSON mode loses opcode diff diagnostics. The replacement logger.stdout() calls do not write to the log file. With stdout disabled, instruction rows disappear entirely; the JSON only provides mismatch ranges. Reproduced a differing PUSH1 that appears in normal stdout but is absent from both JSON-mode stdout and the saved log. Persist these rows before suppressing terminal output. Code (

   diffyscan/diffyscan/utils/binary_verifier.py

   Lines 371 to 382 in 8e9a74c

   	def _print_instruction_diffs(instruction_pairs, checkpoints, immutables):
   	for prev_idx, cur_idx in zip(checkpoints, checkpoints[1:]):
   	if prev_idx != cur_idx - 1:
   	logger.stdout("...")
   	local_instruction, remote_instruction = instruction_pairs[cur_idx]
   	(opcode, opname, params), _ = _format_instruction_diff(
   	local_instruction,
   	remote_instruction,
   	immutables,
   	)
   	logger.stdout(f"{to_hex(cur_idx, 4)} {opcode} {opname} {params}")

   )

Validation: 397 tests passed. Both findings were reproduced offline; no live explorer or RPC checks were run.

[TheDZhon]

Claude:

---

What ran

All runs used the PR head with real Etherscan, GitHub and RPC credentials from this workspace. Secrets are replaced with <...> below.

Case	Command (abridged)	Result
Clean pass	hypernative-guard/mainnet/config.yaml --json -E -G	exit 0, status: passed, 4 source exact, 4 bytecode allowed via immutables, 86 lines, stderr empty. Matches the PR body.
Real diff	lido-earn/mainnet/earn-factory.yaml --json -E -G	exit 1, well-formed JSON, stderr empty. Bytecode failed for MellowStrategyFactory with uncovered: ["offset=329 length=20 immutable", "offset=627 length=20 immutable"] and a ready suggested_rule. Pre-existing on main, see side observation.
RPC error text	same, second run hit a DRPC free-plan 408	bytecode.error contained the full RPC URL including the key. Finding 1.
Empty directory	configs/lido-earn --json (holds only mainnet/)	exit 0, status: passed, summary: {}, contracts: []. Finding 2.
Missing config	does-not-exist.yaml --json	exit 1, status: error, error: "FileNotFoundError: ...", traceback on stderr only. As documented.
Bad token, fail_on_bytecode_comparison_error: false	earn-factory with an invalid Etherscan key	exit 0, status: error, summary.contract_errors: 2, both contracts carry error. Human mode exits 0 with "Total contracts analyzed: 0". Side finding confirmed.
Bad token, fail_on...: true	hypernative-guard with an invalid key	exit 1, status: error, top-level error: "ExplorerError: ...".
SIGINT at 0.4 s and at 12 s	timeout -s INT --preserve-status N .venv/bin/diffyscan earn-factory.yaml --json	exit 1, status: error, error: "KeyboardInterrupt: run interrupted by user", valid JSON both times. Copilot fix works.
SIGINT at 12 s, human mode	same with -Y	exit 1. Output ends with "Keyboard interrupt by user" then "Done in 11.956s". No summary, no line that explains exit 1. Finding 4.

Note for anyone repeating the SIGINT test: a background job started from a non-interactive bash script has SIGINT ignored, so kill -INT does nothing. Run the venv binary in the foreground under timeout -s INT.

Code claims in docs/json-output.md checked against source: _compact drops None and [] only; statuses are exactly exact | allowed | failed (evaluate_source_rules, evaluate_bytecode_rules); hunks come from SequenceMatcher.get_opcodes() with +1 starts (normalize_source_hunks); facet names match allowed_diffs.py; uncovered labels match summarize_bytecode_uncovered; the only prompt is gated by skip_user_input, which --json sets; the only remaining raw print sites are --version and the JSON dump; solc runs under subprocess.run(..., capture_output=True).

Findings

1. Credentials leak into error fields — fix before merge

_handle_request_errors in diffyscan/utils/http_client.py builds the message as f"HTTP error: {exc}{body}". requests.HTTPError text includes the request URL: "408 Client Error: Request Timeout for url: https://lb.drpc.org/ethereum/<DRPC_KEY> Response: {...}". The RPC URL is the whole credential for DRPC-style endpoints, and _get_contract_from_etherscan appends &apikey=<token> to the explorer URL, so an HTTP-level explorer error (403, 429, 5xx) leaks the Etherscan key the same way. Reproduced live: the second earn-factory run returned this in contracts[].bytecode.error.

The same text already reaches stdout and digest/<ts>/logs.txt in human mode through logger.error(str(exc)), so the exposure is not new. The PR widens it materially: the JSON report is the artifact agents will save, diff and paste into issues, and the PR adds three new homes for the text (contracts[].bytecode.error, contracts[].error, top-level error). The codebase already masks these URLs everywhere it logs them on purpose (mask_text in fetch, pull, node_handler), so the raw text in exceptions is an oversight, not a design choice.

Suggested fix, in the wrapper where the URL is known:

def wrapper(*args, **kwargs):
    url = args[0] if args else kwargs.get("url", "")
    def _safe(text: str) -> str:
        return text.replace(url, mask_text(url)) if url else text
    try:
        ...
    except requests.exceptions.HTTPError as exc:
        ...
        raise error_class(_safe(f"HTTP error: {exc}{body}"))
    except requests.exceptions.RequestException as exc:
        raise error_class(_safe(str(exc)))

Add a test that an HTTPError for a URL with apikey= and a key-bearing path never appears unmasked in NodeError/ExplorerError text. Consider a second line of defense: run every string that lands in the JSON report through a redactor seeded with the values load_env(..., masked=True) loaded.

2. status: "passed" when nothing was checked — fix before merge

_collect_config_paths returns [] for a directory without .json/.yaml/.yml files. main then reports status: "passed", exit_code: 0, summary: {}, contracts: []. After PR 211 every project directory has this shape (configs/lido-earn contains only mainnet/), so an agent that points at a project instead of a network gets a green verdict for zero contracts. A config with an empty contracts map has the same effect. The documentation's definition of passed ("each contract verified") is vacuously true here.

Human mode exits 0 in the same case today ("Total contracts analyzed: 0"), so this is pre-existing, but the JSON turns it into a machine-consumable false positive. The --contract filter already has a guard for "matched nothing"; the same guard should apply without a filter:

config_paths = _collect_config_paths(args.path)
if not config_paths:
    raise FileNotFoundError(f"No config files (.json, .yaml, .yml) found in {args.path}")
...
nothing_checked = error is None and sum(r["matched_count"] for r in all_results) == 0

and let nothing_checked drive the existing failed + exit 1 path (with the filter-specific message kept when a filter was given). Document the case in json-output.md.

3. any rule hides the suppressed simulation error — medium

In process_config, a DeploymentSimulationError suppressed by an any: true rule produces _bytecode_result(status="allowed", matched_facets=["any"]) without error. The JSON then reads {"status": "allowed", "facets": ["any"], "reason": ...}, indistinguishable from "compared, diff allowed". No comparison ran. Pass error=str(exc) there and document that error may accompany allowed when the facet is any. Code-confirmed, not run live.

4. Interrupted runs now exit 1 in both modes; human mode lost its summary — minor

Commit 8e9a74c makes an interrupted run exit 1 in human mode too, where main exits 0 when the partial results passed. That is the right call, but the PR body still says "the exit code is unchanged" and the exit-status paragraph in docs/cli.md does not mention interruption. In human mode the partial summary is no longer printed and no log line explains the non-zero exit (reproduced: output ends at "Done in 11.956s"). Suggest a logger.error("Run interrupted; partial results are not a verdict") line and keeping the partial summary under a clear heading.

5. A KeyboardInterrupt outside the per-contract loop escapes JSON mode — minor

process_config catches KeyboardInterrupt only around the contract loop. An interrupt during load_config, _load_explorer_token, _setup_binary_comparison, or between two configs propagates past except Exception in main, so no report is printed. My 0.4 s interrupt happened to land inside the loop (during get_chain_id). Catching KeyboardInterrupt in main and routing it to the same error string closes the gap.

6. Documentation precision — nits

• with_diffs also counts missing files: a file absent on GitHub yields one insert hunk from an empty left side, so bool(hunks) is true. Say "files that differ or are missing".
• Bytecode error is not limited to "compile, calldata, or deployment-simulation": any BaseCustomException from run_bytecode_diff lands there, including RPC (NodeError) and explorer errors. Say "the error that stopped the bytecode comparison".
• log_file is relative to the working directory. Emit os.path.abspath(LOGS_PATH) so consumers in another directory can open it.
• "Later versions may add keys" invites a diffyscan_version (or schema_version) top-level key; __version__ is already imported.

7. Suggestion: provenance block — non-blocking

The report does not say what was compared against what. For the agents it targets, a per-config inputs object with explorer_hostname, explorer_chain_id, remote_chain_id, github_repo.url, github_repo.commit and the dependency pins would make the artifact self-describing and let a consumer notice a wrong chain or a moved pin without opening the log. This follows the trust-anchor lens from the PR 210 review: the RPC and the pinned commit are the unverified reference suppliers, and the report should name them.

Side observation: configs/lido-earn/mainnet/earn-factory.yaml is red on main

Independent of this PR. On main cbbf63b the same config fails bytecode comparison for MellowStrategyFactory 0x8Fac09FD82F031D390B94622E2E4baBf16Fd2236 with uncovered immutables at offsets 329 and 627 (observed values at 317 and 615 equal the StrategyCallForwarder address 0x7305...0Bb8). Cause, from the pinned source at vaults-wrapper@37b0999: STRATEGY_CALL_FORWARDER_IMPLEMENTATION = address(new StrategyCallForwarder()) in the constructor, so the immutable holds a CREATE address that depends on the factory's own deployment address and nonce, which the eth_call simulation cannot reproduce. The config needs an immutables rule for those two slots with that reason; the PR's suggested_rule output gives the exact offsets and values. The directory is not in the CI matrix, so this stays invisible until someone runs it.

[TheDZhon]

Codex

---

Both original findings are fixed: completed results survive later fatal errors, and opcode diff rows reach the log in JSON mode. Both reproductions now pass.

Three additional issues from the discussion were independently confirmed:

1. [P2] Credentials appear in JSON error fields. Synthetic HTTP failures exposed complete RPC-path and explorer-query credentials through exception text. Redact credentials before serializing errors. The underlying exception leakage predates this PR; the JSON report adds another exposure surface. Code (

   diffyscan/diffyscan/diffyscan.py

   Line 735 in e453eaa

   error=str(exc),

   )
2. [P2] Zero checks produce a passing verdict. diffyscan configs/lido-earn --json returns exit 0, status: "passed" and empty results because discovery ignores its subdirectories. Reject runs that discover no configurations or check no contracts. The existing exit-code behavior now becomes an explicit JSON success verdict. Code (

   diffyscan/diffyscan/diffyscan.py

   Lines 1066 to 1070 in e453eaa

   	# A contract filter that matches nothing across all configs is a usage error
   	filter_unmatched = (
   	error is None
   	and bool(args.contract_filter)
   	and sum(r["matched_count"] for r in all_results) == 0

   )
3. [P3] Interrupts during setup produce no JSON. Injecting KeyboardInterrupt during config loading leaves stdout empty: it escapes the outer except Exception. Handle it explicitly in main(). Code (

   diffyscan/diffyscan/diffyscan.py

   Lines 1059 to 1064 in e453eaa

   	except Exception as exc:
   	# In JSON mode the report must still reach stdout; keep the traceback on stderr.
   	if not args.json:
   	raise
   	traceback.print_exc()
   	error = f"{type(exc).__name__}: {exc}"

   )

Validation: 401 tests passed, plus offline reproductions. No live explorer or RPC checks were run.

[Copilot]

🟢 Approval recommended

The implementation, documentation, and regression coverage are consistent with the stated JSON-output contract.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[TheDZhon]

LGTM!

Still open, all non-blocking:

• The PR body still says the exit code is unchanged, and the exit-status paragraph in docs/cli.md is stale. Human mode now exits 1 in four new cases: interruption, zero checked contracts, a directory without configs, and both comparisons disabled. All are improvements and should be documented.
• An any rule that suppresses a simulation error still hides the error text.
• Whole HTTP response bodies are copied into error. The Etherscan 404 put its full HTML page into every error field. Truncating the body would keep reports small.
• The contract that aborted a run is named only when the exception text happens to include its address.
• Redaction is exact string replacement, so a library that re-encodes the URL in its message would defeat it. Fine for today's alphanumeric keys.
• Earlier doc nits and the provenance suggestion stand.

Side finding unchanged. The Earn factory mainnet config is red on main because a constructor-created immutable cannot be reproduced by simulation. It needs an immutables rule for the two slots.