Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,42 @@ part of the release, not documentation about it.
./shipgate doctor --config samples/support_refund_agent/shipgate.yaml
./shipgate scan --config samples/support_refund_agent/shipgate.yaml
./shipgate list-checks
python scripts/regenerate_goldens.py --check
```

### Sample goldens

The [golden generator](scripts/regenerate_goldens.py) owns the 24 committed
artifacts in seven `samples/*/expected/` directories. With the normal development
dependencies and Git available:

```bash
python scripts/regenerate_goldens.py # regenerate all
python scripts/regenerate_goldens.py conductor_agent # one sample
python scripts/regenerate_goldens.py --check # read-only drift check
```

It scans disposable copies with the source-tree writers; the caller's working
tree, manifests, declarations and CI summary are not scan output destinations.
Generation validates the entire selected set before writing any expected file.
Exit 0 means generated/matching, 1 means drift in check mode, and 2 means the
recipe could not finish. Every drift names its file. An added expected artifact
requires an explicit recipe; deleting one does not remove it from the check.
`tests/test_regenerate_goldens.py` invokes the actual `--check` command in the
normal CI suite. Existing behavioral assertions remain independent oracles;
do not regenerate away a changed verdict, open question or safety regression.

The recipe pins packet time, preserves the ordinary and cold-manifest states,
uses relative output paths under the manifest, normalizes only known path
fields, and writes LF bytes. Text inputs in the disposable copy are normalized
to LF too, so a Windows checkout does not change the manifest bytes bound by
the sample pointer. Scans disable installed plugins and isolate inherited Git
configuration; symlinked fixture paths and leaked temporary paths are errors. The
Conductor scan pointer is rebound **after** normalization and has no predecessor
from another generating run; it retains scan-only permissions, never a verifier
receipt or release authority. These development fixtures are not qualification
evidence. #569 still owns the actual report 1.0 freeze and migration fixtures.

## Contribution Areas

- new deterministic checks;
Expand Down Expand Up @@ -194,6 +228,11 @@ CI runs `python scripts/generate_schemas.py --check` and fails fast
with a unified diff if a committed schema drifts from the live model.
The same drift is also caught by `tests/test_schema_roundtrip.py`, so
your test suite will reject the change locally before CI does.
After a report or contract change, run
`python scripts/regenerate_goldens.py`, inspect the semantic diff, then run
`python scripts/regenerate_goldens.py --check` and the affected behavioral
tests. The [sample recipe](#sample-goldens) owns path and digest normalization;
do not update a version stamp or pointer hash by hand.

## Check Contributions

Expand Down
7 changes: 7 additions & 0 deletions docs/agent-contract-current.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ GitHub acquisition and persistence remain an integration obligation.

The single, current statement of what AI coding agents and CI integrations should read from Agents Shipgate output. When the contract changes, update [STABILITY.md](../STABILITY.md) first, then this file. Other agent-facing surfaces (`AGENTS.md`, `llms.txt`, `.well-known/agents-shipgate.json`, the slash command, the skill, the FAQ) link here instead of restating field lists.

For contributors changing the contract: regenerate schemas with
`python scripts/generate_schemas.py`, then run
`python scripts/regenerate_goldens.py` and its `--check` mode using the
[committed sample recipe](../CONTRIBUTING.md#sample-goldens). Review actual
artifact changes and preserve behavioral assertions; sample regeneration does
not freeze a release contract or qualify a candidate wheel.

## Current versions

Verify the installed CLI contract locally before relying on hard-coded docs:
Expand Down
6 changes: 6 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,12 @@ Adding a new wire field: edit the relevant `schemas/<name>.py`, run
`report_schema_version` / `packet_schema_version` if the addition is
public. The CI step `python scripts/generate_schemas.py --check`
fails if the committed JSON drifts from the live model.
Regenerate affected sample artifacts with
`python scripts/regenerate_goldens.py` and confirm
`python scripts/regenerate_goldens.py --check`; the
[contributor recipe](../CONTRIBUTING.md#sample-goldens) owns the output paths,
normalization and subsequent scan-pointer rebinding. Keep the independent
semantic assertions when reviewing the resulting golden diff.

## Typed domain types: `Scope`, `SideEffect`, `Action`

Expand Down
7 changes: 7 additions & 0 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1495,6 +1495,13 @@ GitHub acquisition and persistence remain an integration obligation.

The single, current statement of what AI coding agents and CI integrations should read from Agents Shipgate output. When the contract changes, update [STABILITY.md](../STABILITY.md) first, then this file. Other agent-facing surfaces (`AGENTS.md`, `llms.txt`, `.well-known/agents-shipgate.json`, the slash command, the skill, the FAQ) link here instead of restating field lists.

For contributors changing the contract: regenerate schemas with
`python scripts/generate_schemas.py`, then run
`python scripts/regenerate_goldens.py` and its `--check` mode using the
[committed sample recipe](../CONTRIBUTING.md#sample-goldens). Review actual
artifact changes and preserve behavioral assertions; sample regeneration does
not freeze a release contract or qualify a candidate wheel.

## Current versions

Verify the installed CLI contract locally before relying on hard-coded docs:
Expand Down
8 changes: 4 additions & 4 deletions samples/conductor_agent/expected/current-control.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
"artifacts": {
"report": {
"path": "report.json",
"sha256": "sha256:5b66dfcf8e408fba258a02babe161d30cde6eee0076e01e52db4cb3873d8e194",
"size_bytes": 51797
"sha256": "sha256:f9922a1c9bed4f6a2662d4f9cdb1f2c8524c79a3349a9235a974ae2d317197d6",
"size_bytes": 51800
},
"report_markdown": {
"path": "report.md",
Expand All @@ -25,13 +25,13 @@
"reason": "A standalone scan produced the current report set. A scan does not authorize completion or merge; run `agents-shipgate verify` to obtain a merge decision.",
"state": "agent_action_required"
},
"current_control_id": "sha256:fb0d14e5557509cb84ccee70672e5c1ea3d6d4745dbeb2aef5c765921d967913",
"current_control_id": "sha256:f96b3a7c66420b891f8f84646f767108dded2022c77b75f5d232285baaae0028",
"decision_id": null,
"lifecycle_state": "terminal",
"operation": "scan",
"request_id": null,
"schema_version": "shipgate.current_control/v1",
"supersedes": "sha256:96fb38c3a8f0921a586629205730a7fe10af801d9442e73ec2dab7efbebc1d32",
"supersedes": null,
"workspace_identity": {
"base_commit_sha": null,
"base_ref": null,
Expand Down
2 changes: 1 addition & 1 deletion samples/conductor_agent/expected/report.json
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,7 @@
],
"release_consequence": {
"decision": "insufficient_evidence",
"summary": "Static evidence is incomplete; capability/intent analysis may miss release-relevant signal gather deeper sources before shipping.",
"summary": "Static evidence is incomplete; capability/intent analysis may miss release-relevant signal \u2014 gather deeper sources before shipping.",
"blocker_misalignment_count": 0,
"review_misalignment_count": 2,
"fail_policy": {
Expand Down
66 changes: 10 additions & 56 deletions samples/declaration_repair_agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,66 +120,20 @@ action-row route; keeping the two apart also keeps this golden clear of

## Regenerating

Run from the repository root, after any change that moves values:
Run from the repository root:

```bash
python - <<'PY'
import json
from pathlib import Path
from agents_shipgate.cli.scan import run_scan

sample = Path("samples/declaration_repair_agent")
expected = sample / "expected"
run_scan(
config_path=sample / "shipgate.yaml",
output_dir=Path("expected"),
formats=["json", "markdown"],
ci_mode="advisory",
packet_enabled=False,
)
(expected / "current-control.json").unlink(missing_ok=True)

golden = expected / "report.json"
payload = json.loads(golden.read_text(encoding="utf-8"))
payload["manifest_dir"] = f"<REPO>/{sample.as_posix()}"
payload["generated_reports"] = {
fmt: Path(written).as_posix()
for fmt, written in payload["generated_reports"].items()
}
golden.write_text(json.dumps(payload, indent=2), encoding="utf-8", newline="\n")

# The scan's own writers use the platform newline. Rewrite every golden with
# an explicit LF, whoever produced it.
for name in ("report.md", "suggested-declarations.yaml"):
path = expected / name
path.write_text(path.read_text(encoding="utf-8"), encoding="utf-8", newline="\n")
PY
python scripts/regenerate_goldens.py declaration_repair_agent
python scripts/regenerate_goldens.py --check declaration_repair_agent
```

This is the sibling's recipe with the sample path swapped, and every
normalization in it is load-bearing for the same reasons — the four notes under
[`google_adk_cold_start_agent` § Regenerating the goldens](../google_adk_cold_start_agent/README.md)
apply here verbatim. In short:

- the path rewrite is **structural**, because a textual `<REPO>` replace is a
silent no-op on Windows: `json.dumps` escapes the separators, so the file
holds `C:\\repo\\samples\…` while `os.getcwd()` is `C:\repo\samples\…` and
the two never match — leaving an absolute `manifest_dir` that fails
`test_sample_expected_report_json_uses_repo_placeholder_for_manifest_dir` on
the machine that produced the golden;
- `generated_reports` needs `.as_posix()`, or a Windows run commits
`expected\report.json` and churns against every other platform;
- all three goldens are rewritten with an explicit `newline="\n"`, because
every writer involved opens text mode with `newline=None` and
`.gitattributes` pins `samples/**/expected/** -text`, so Git stores whatever
bytes were produced. No byte comparison can see it — `read_text` normalizes
CRLF on the way in — which is why
`test_sample_expected_goldens_are_committed_with_lf_newlines` reads raw bytes.

An earlier draft of this section gave the first two as prose and omitted the
third entirely, which made the repo's designated recovery path — named in
`test_repair_scaffold_matches_its_golden`'s own failure message — the
counterexample to that guard's docstring (#465 review).
The [shared recipe](../../scripts/regenerate_goldens.py) runs the existing
scanner against a disposable copy of the committed inputs. It owns relative
output paths, structural path normalization and LF bytes; see
[the contributor recipe](../../CONTRIBUTING.md#sample-goldens). It does not
change the challenged declarations, execute sample tools or apply the suggested
questionnaire. CI calls the same check mode and retains the existing semantic
and questionnaire assertions.

Read the diff on `expected/suggested-declarations.yaml` before committing: a
change in the `risk_tags:` values a block publishes is a change to the remedy
Expand Down
2 changes: 1 addition & 1 deletion samples/declaration_repair_agent/expected/report.json
Original file line number Diff line number Diff line change
Expand Up @@ -757,7 +757,7 @@
"misalignments": [],
"release_consequence": {
"decision": "insufficient_evidence",
"summary": "Static evidence is incomplete; capability/intent analysis may miss release-relevant signal gather deeper sources before shipping.",
"summary": "Static evidence is incomplete; capability/intent analysis may miss release-relevant signal \u2014 gather deeper sources before shipping.",
"blocker_misalignment_count": 0,
"review_misalignment_count": 0,
"fail_policy": {
Expand Down
149 changes: 24 additions & 125 deletions samples/google_adk_cold_start_agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,132 +120,31 @@ questionnaire is here to own. `test_a_heuristic_cannot_propose_that_an_action_is

## Regenerating the goldens

Run from the repository root, after any change that moves values:
Run from the repository root:

```bash
python - <<'PY'
import json
from pathlib import Path
from agents_shipgate.cli.scan import run_scan

sample = Path("samples/google_adk_cold_start_agent")
expected = sample / "expected"
run_scan(
config_path=sample / "shipgate.yaml",
output_dir=Path("expected"),
formats=["json", "markdown"],
ci_mode="advisory",
packet_enabled=False,
)
(expected / "current-control.json").unlink(missing_ok=True)

golden = expected / "report.json"
payload = json.loads(golden.read_text(encoding="utf-8"))
payload["manifest_dir"] = f"<REPO>/{sample.as_posix()}"
payload["generated_reports"] = {
fmt: Path(written).as_posix()
for fmt, written in payload["generated_reports"].items()
}
golden.write_text(json.dumps(payload, indent=2), encoding="utf-8", newline="\n")

# The scan's own writers use the platform newline. Rewrite every golden with
# an explicit LF, whoever produced it.
for name in ("report.md", "suggested-declarations.yaml"):
path = expected / name
path.write_text(path.read_text(encoding="utf-8"), encoding="utf-8", newline="\n")
PY
python scripts/regenerate_goldens.py google_adk_cold_start_agent
python scripts/regenerate_goldens.py --check google_adk_cold_start_agent
```

### Regenerating the cold report

`expected/cold-report.md` comes from a different repository state than the
ordinary goldens: the agent sources are committed, while `shipgate.yaml`
exists only in the worktree. Recreate that state in a temporary repository;
running the ordinary recipe above cannot produce the cold-reader order.

```bash
python - <<'PY'
import shutil
import subprocess
import tempfile
from pathlib import Path

from agents_shipgate.cli.scan import run_scan

source = Path("samples/google_adk_cold_start_agent").resolve()
golden = source / "expected" / "cold-report.md"

with tempfile.TemporaryDirectory(prefix="shipgate-cold-golden-") as temp:
repo = Path(temp) / "repo"
shutil.copytree(source, repo, ignore=shutil.ignore_patterns("expected"))

def git(*args: str) -> None:
subprocess.run(
["git", "-C", str(repo), *args],
check=True,
capture_output=True,
)

git("init", "-q")
git("config", "user.name", "Shipgate Golden")
git("config", "user.email", "shipgate@example.invalid")
git("add", "agent.py", "inventories", "specs")
git("commit", "-qm", "base without manifest")

out = repo / "reports"
run_scan(
config_path=repo / "shipgate.yaml",
output_dir=out,
formats=["markdown", "json"],
ci_mode="advisory",
)
golden.write_text(
(out / "report.md").read_text(encoding="utf-8"),
encoding="utf-8",
newline="\n",
)
PY
```

Four things that look like details and are not.

**Newlines are forced, not inherited.** `Path.write_text(..., encoding="utf-8")`
opens in text mode with `newline=None`, so on Windows every `\n` is written as
`\r\n` — by the recipe *and* by `write_json_report`, the Markdown writer and the
questionnaire writer. `.gitattributes` pins `samples/**/expected/** -text`
precisely so Git hands those bytes over unchanged, so a golden regenerated on
Windows would be committed as CRLF against everyone else's LF. The tests cannot
see it: `read_text()` applies universal newlines and normalizes CRLF back to LF
on the way in, so every byte comparison in this repo passes on a file whose
bytes moved. `newline="\n"` on all three is what makes the artifact the same
artifact everywhere, and
`test_sample_expected_goldens_are_committed_with_lf_newlines` reads the raw
bytes so the guard does not share the blindness.

**The path normalization is structural, not textual.** An earlier version did
`text.replace(os.getcwd(), "<REPO>")`, which works on POSIX and is a no-op on
Windows: `json.dumps` escapes the separators, so the file holds
`C:\\repo\\samples\\…` while `os.getcwd()` is `C:\repo\samples\…` and the two
never match. The golden then keeps an absolute `manifest_dir` and fails
`test_sample_expected_report_json_uses_repo_placeholder_for_manifest_dir` — on
the machine that produced it. Assigning the field a POSIX value is correct on
both, and `Path(written).as_posix()` does the same for `generated_reports`,
which would otherwise commit `expected\report.json` and churn against every
other platform.

`json.dumps(payload, indent=2)` is exactly what `write_json_report` uses — no
`sort_keys`, no trailing newline — so the round trip is byte-identical and
nothing but the two normalized fields moves.

**`output_dir` is the relative `"expected"`**, which `run_scan` resolves under
the manifest directory rather than under the process directory. The report
records where it wrote itself, so scanning into an absolute temporary directory
and copying the files back bakes a contributor's `/var/folders/…/tmp…` path
into `generated_reports` — a value no test compared, which is exactly why it
sat there churning until #425's review.

**The `unlink` is not tidying.** A scan also publishes `current-control.json`,
and this fixture deliberately does not commit one: the hash-bound pointer path
is covered by [`conductor_agent`](../conductor_agent/), and a second copy would
have to be rebound *after* the normalization every time, since that rewrite
changes both the length and the digest of `report.json`.
The [committed recipe](../../scripts/regenerate_goldens.py) builds both
repository states in disposable copies. The ordinary report has a committed
manifest; `cold-report.md` commits only `agent.py`, `inventories` and `specs`,
leaving the existing manifest untracked. It never fills declaration blanks or
changes the sample's inputs. Those different states are why an ordinary scan
cannot regenerate the cold-reader ordering.

The script uses relative `output_dir="expected"`, which the scanner resolves
under the copied manifest directory. It normalizes `manifest_dir` structurally
to `<REPO>/samples/google_adk_cold_start_agent` and report paths to POSIX,
forces LF bytes on every output, and keeps the JSON writer's field order and
terminator. Absolute output paths or surviving generating-machine paths stop
generation before any repository artifact is written.

This fixture does not publish a control pointer. The Conductor recipe covers
that contract by rebinding its scan-only pointer after every normalized byte
is final; it cannot authorize merge or completion. See
[the contributor recipe](../../CONTRIBUTING.md#sample-goldens) for all samples.

Review the semantic assertions above alongside any golden diff. Closing the
open questions or changing a declaration is not a regeneration technique.
4 changes: 2 additions & 2 deletions samples/support_refund_agent/expected/packet.json
Original file line number Diff line number Diff line change
Expand Up @@ -2541,8 +2541,8 @@
"total": 2
},
"declaration_review": {
"base_kind": "none",
"base_comparison_requested": false,
"base_kind": "none",
"changed_count": 0,
"enabled": false,
"notes": [
Expand Down Expand Up @@ -3037,4 +3037,4 @@
"unchanged_findings": 0
}
}
}
}
Loading
Loading