From 25d990d7c8ea932bcd5759bc5c582cbc82aef365 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Fri, 20 Nov 2026 13:26:50 +0200 Subject: [PATCH 1/2] Make the broken-prompt demo exercise a reverse schema migration The bad change never touched the schema: deploy-bad-change.sh only swapped the prompt and inserted rows, so every commit recorded schema_version 0.0.0 and the rollback printed "schema already at 0.0.0". The demo has never run a migration since it landed in 57c66ba, before the reverse migration runner (97e786b) existed, and nobody went back to wire it in. The bad change now applies migration 001_add_sentiment (a NOT NULL column, recorded in agentic_migrations) and registers its .down.sql under .agentic/schema/. The v0.8 commit records the new version and the step 10 rollback reverses it, consuming one ADR-0019 approval token. The diff step compared the baseline with HEAD, which by then is a LangGraph checkpoint commit with no memory or schema dimension, so it printed "schema_version 0.0.0 -> (none)". It now diffs against the v0.8 commit. run-demo.sh also checks after the rollback that the live schema is back at 0.0.0 and the sentiment column is gone, so CI fails if the reversal is skipped. README updated where it described the old output. --- examples/langgraph-rollback/README.md | 41 +++++++++++-------- .../bad-change/001_add_sentiment.down.sql | 6 +++ .../bad-change/001_add_sentiment.up.sql | 8 ++++ .../scripts/deploy-bad-change.sh | 19 +++++++-- .../langgraph-rollback/scripts/run-demo.sh | 20 ++++++--- 5 files changed, 68 insertions(+), 26 deletions(-) create mode 100644 examples/langgraph-rollback/bad-change/001_add_sentiment.down.sql create mode 100644 examples/langgraph-rollback/bad-change/001_add_sentiment.up.sql diff --git a/examples/langgraph-rollback/README.md b/examples/langgraph-rollback/README.md index 31f093e3..2620897f 100644 --- a/examples/langgraph-rollback/README.md +++ b/examples/langgraph-rollback/README.md @@ -49,16 +49,20 @@ The script: 4. asks the agent *"I'm thinking about cancelling my subscription."* — gets an empathetic answer, 5. `agentic commit -m "v0.7 baseline"`, -6. `deploy-bad-change.sh` swaps in the sycophantic prompt + inserts - five contaminated rows ("Refund of $499 processed for customer +6. `deploy-bad-change.sh` swaps in the sycophantic prompt, applies + migration `001_add_sentiment` (a `NOT NULL` `episodes.sentiment` + column, with its `.down.sql` registered under `.agentic/schema/`) + and inserts five contaminated rows ("Refund of $499 processed for customer yesterday — confirmed.", etc.), 7. asks again — the agent now offers an unauthorised refund and confidently cites the fake refund row from memory, 8. `agentic commit -m "v0.8 friendlier prompt"`, -9. `agentic diff baseline HEAD` shows the prompts tree, the memory - manifest, and the LangGraph checkpoint blob have all moved, -10. `agentic rollback baseline --yes` — restores prompts on disk, - truncates and replays `episodes` from the baseline manifest inside +9. `agentic diff baseline ` shows the prompts tree, the + memory manifest, and the schema version (`0.0.0 → 001_add_sentiment`) + have all moved, +10. `agentic rollback baseline --yes` — reverses the schema migration + (using an approval token the CLI mints from `.agentic/approval.key`), + restores prompts on disk, truncates and replays `episodes` from the baseline manifest inside one transaction, forward-records a new commit on the branch, 11. asks again — empathetic answer is back; the contaminated memory row is gone. @@ -74,7 +78,7 @@ The fix is one command. `git revert` alone never restores step (11). | `tools` | unchanged | n/a | | `model` | recorded in each commit; unchanged in this demo | rollback notes "operator must redeploy if it changed" | | `code_sha` | recorded in each commit (from `git rev-parse HEAD`) | rollback target's code SHA threaded into the forward-recorded commit | -| `schema_version` | unchanged in this demo (`agentic_schema_version()` returns `0.0.0`) | gated on rollback; reverse migration runner is a planned follow-up | +| `schema_version` | migration `001_add_sentiment` adds `episodes.sentiment NOT NULL`; `agentic_schema_version()` goes `0.0.0` → `001_add_sentiment` | `agentic rollback` runs `.agentic/schema/001_add_sentiment.down.sql` in one transaction, gated by an approval token ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)) | The LangGraph wiring is real: every agent invocation runs through `AgenticCheckpointer`, so `agentic log` shows one commit per graph step @@ -90,15 +94,15 @@ history reflects the rollback action rather than rewriting it. enough to make the broken behaviour reproducible without API keys and without flaky model output. The rollback story is identical for a real LLM-backed agent — the dimensions snapshotted are the same. -- **No schema migration.** The demo spec describes a `sentiment NOT - NULL` column added in v0.8 and reverse-migrated on rollback; that - requires the reverse SQL migration runner which is the next planned - ADR-0002 §5 follow-up. Today the demo exercises prompts + memory - rollback only. A rollback that does run reverse migrations needs an - operator approval token ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)). - The daemon reads `.agentic/approval.key` and the `agentic` CLI mints +- **The migration is hand-rolled.** There is no forward migration + runner: `deploy-bad-change.sh` applies `bad-change/001_add_sentiment.up.sql` + with `psql` (which also records the name in `agentic_migrations`) and + copies the `.down.sql` into `.agentic/schema/`. The rollback side is + the real reverse-migration runner. It needs an operator approval token + ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)); + the daemon reads `.agentic/approval.key` and the `agentic` CLI mints the token from the same file, so `agentic rollback` is typed the same - way; a socket client without the key cannot run reverse migrations. + way, and a socket client without the key cannot run reverse migrations. - **Compose covers the database only.** `docker-compose.yml` starts Postgres + pgvector. `agenticd` and the Python agent are built and run locally by `scripts/run-demo.sh` — they are not containerised. @@ -113,11 +117,13 @@ examples/langgraph-rollback/ │ └── system.txt baseline empathetic system prompt ├── bad-change/ │ ├── system.txt sycophantic variant the bad change ships +│ ├── 001_add_sentiment.up.sql forward migration (NOT NULL column) +│ ├── 001_add_sentiment.down.sql its reverse, copied to .agentic/schema/ │ └── seed.sql contaminated rows the bad change INSERTs ├── seed.sql baseline episodes + pgvector setup ├── scripts/ │ ├── ask.sh invoke the agent with one user message -│ ├── deploy-bad-change.sh apply prompt + memory contamination +│ ├── deploy-bad-change.sh apply prompt + migration + memory contamination │ ├── redeploy.sh simulate code-only redeploy after git revert │ └── run-demo.sh full scenario orchestrator ├── pyproject.toml Python deps for the demo @@ -157,13 +163,14 @@ BASELINE=$(../../target/release/agentic --repo . status | sed 's/.*→ //') ./scripts/ask.sh "I'm thinking about cancelling." ../../target/release/agentic --repo . commit -m "bad change" \ --model "anthropic:claude-opus:2026-05-01" +BAD=$(../../target/release/agentic --repo . status | sed 's/.*→ //') # 4.5. Show that restoring only the prompt file does not fix it git checkout -- prompts/system.txt ./scripts/redeploy.sh ./scripts/ask.sh "I'm thinking about cancelling." # still broken -../../target/release/agentic --repo . diff "$BASELINE" HEAD +../../target/release/agentic --repo . diff "$BASELINE" "$BAD" ../../target/release/agentic --repo . rollback "$BASELINE" --yes ./scripts/ask.sh "I'm thinking about cancelling." # fixed diff --git a/examples/langgraph-rollback/bad-change/001_add_sentiment.down.sql b/examples/langgraph-rollback/bad-change/001_add_sentiment.down.sql new file mode 100644 index 00000000..103580f9 --- /dev/null +++ b/examples/langgraph-rollback/bad-change/001_add_sentiment.down.sql @@ -0,0 +1,6 @@ +-- Reverse of 001_add_sentiment.up.sql. deploy-bad-change.sh copies it to +-- .agentic/schema/; `agentic rollback` runs it (with the matching +-- agentic_migrations delete) in one transaction before restoring memory. +-- No BEGIN/COMMIT here: the daemon owns the transaction. + +ALTER TABLE episodes DROP COLUMN sentiment; diff --git a/examples/langgraph-rollback/bad-change/001_add_sentiment.up.sql b/examples/langgraph-rollback/bad-change/001_add_sentiment.up.sql new file mode 100644 index 00000000..83260254 --- /dev/null +++ b/examples/langgraph-rollback/bad-change/001_add_sentiment.up.sql @@ -0,0 +1,8 @@ +-- Forward migration the bad change ships: a NOT NULL `sentiment` column +-- on the agent's memory table. A code-only `git revert` cannot undo it. +-- The agentic_migrations row is what `agentic_schema_version()` reads, +-- so the next `agentic commit` records this version in the Commit. +-- Applied by deploy-bad-change.sh under `psql -1` (one transaction). + +ALTER TABLE episodes ADD COLUMN sentiment text NOT NULL DEFAULT 'neutral'; +INSERT INTO agentic_migrations (name) VALUES ('001_add_sentiment'); diff --git a/examples/langgraph-rollback/scripts/deploy-bad-change.sh b/examples/langgraph-rollback/scripts/deploy-bad-change.sh index 082b62b4..e1b477c8 100755 --- a/examples/langgraph-rollback/scripts/deploy-bad-change.sh +++ b/examples/langgraph-rollback/scripts/deploy-bad-change.sh @@ -2,12 +2,15 @@ # Apply the bad change — the simulated "small prompt tweak" that # triggers the broken-prompt scenario. # -# Two things land at once, the same way they would in a real deploy: +# Three things land at once, the same way they would in a real deploy: # # 1. The sycophantic system prompt overwrites the empathetic one. -# 2. Contaminated rows are INSERTed into the agent's `episodes` -# memory table — these are what makes a code-only `git revert` -# insufficient. +# 2. A schema migration adds a NOT NULL `sentiment` column to the +# agent's `episodes` memory table and records itself in +# `agentic_migrations`. Its `.down.sql` is registered under +# `.agentic/schema/` so `agentic rollback` can reverse it. +# 3. Contaminated rows are INSERTed into `episodes` — together with +# (2) these are what make a code-only `git revert` insufficient. # # Neither change is committed to agenticd here; run-demo.sh does that # right after, so the bad state lands as a single commit. @@ -21,6 +24,14 @@ here="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" echo "→ swapping prompts/system.txt for the sycophantic variant" cp "${here}/bad-change/system.txt" "${here}/prompts/system.txt" +echo "→ applying migration 001_add_sentiment (episodes.sentiment NOT NULL)" +psql "${DATABASE_URL}" -v ON_ERROR_STOP=1 -1 -f "${here}/bad-change/001_add_sentiment.up.sql" >/dev/null +# Operator-owned reverse migration. .agentic is 0700 and already exists +# (run-demo.sh runs `agentic init` first); the daemon reads the file at +# rollback time, so no restart is needed. +mkdir -p "${here}/.agentic/schema" +cp "${here}/bad-change/001_add_sentiment.down.sql" "${here}/.agentic/schema/001_add_sentiment.down.sql" + echo "→ INSERTing 5 contaminated rows into episodes" psql "${DATABASE_URL}" -v ON_ERROR_STOP=1 -f "${here}/bad-change/seed.sql" >/dev/null diff --git a/examples/langgraph-rollback/scripts/run-demo.sh b/examples/langgraph-rollback/scripts/run-demo.sh index efd2b4fc..01031502 100755 --- a/examples/langgraph-rollback/scripts/run-demo.sh +++ b/examples/langgraph-rollback/scripts/run-demo.sh @@ -12,12 +12,12 @@ # 3. build agenticd + start it bound to the demo repo # 4. agent baseline ask # 5. `agentic commit` baseline -# 6. deploy-bad-change.sh +# 6. deploy-bad-change.sh ← prompt + schema migration + bad rows # 7. agent bad ask ← the hallucinated refund # 8. `agentic commit` "bad change" # 8.5. simulate git revert ← still broken (contaminated memory) # 9. `agentic diff` baseline → bad ← multi-dimensional regression -# 10. `agentic rollback baseline --yes` +# 10. `agentic rollback baseline --yes` ← incl. reverse migration # 11. agent ask again ← empathetic answer is back # 12. cleanup @@ -111,8 +111,8 @@ chmod go-w "${DEMO_DIR}" "${DEMO_DIR}/.agentic" # Operator approval key (ADR-0019). A rollback that reverses a schema # migration needs an approval token; the daemon reads this key and the # `agentic` CLI mints the token from it, so the visible commands stay the -# same. The demo's step 10 rollback has no schema change today and so -# consumes no token. The key must be 0600 or the daemon refuses to start. +# same. The step 10 rollback reverses the bad change's schema migration, +# so it consumes one token. The key must be 0600 or the daemon refuses to start. APPROVAL_KEY="${DEMO_DIR}/.agentic/approval.key" if [[ ! -f "${APPROVAL_KEY}" ]]; then ( umask 077 && head -c 32 /dev/urandom > "${APPROVAL_KEY}" ) @@ -167,6 +167,7 @@ step "7. bad ask (hallucinated refund + contaminated memory)" step "8. commit bad change" "${AGENTIC_BIN}" --repo "${DEMO_DIR}" commit -m "v0.8 friendlier prompt" \ --model "anthropic:claude-opus:2026-05-01" +BAD=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" status | sed -n 's/.*→ //p') step "8.5. simulate git revert — shows code-only revert is insufficient" # Restore the prompt file to baseline (simulating what a code-only rollback, e.g. @@ -182,10 +183,19 @@ fi # Contaminated memory rows are still in Postgres — the answer is still broken. step "9. diff baseline → bad" -"${AGENTIC_BIN}" --repo "${DEMO_DIR}" diff "${BASELINE}" HEAD +# Diff against the bad commit itself, not HEAD: the step 8.5 ask adds +# LangGraph checkpoint commits that carry no memory or schema dimension. +"${AGENTIC_BIN}" --repo "${DEMO_DIR}" diff "${BASELINE}" "${BAD}" step "10. rollback to baseline" "${AGENTIC_BIN}" --repo "${DEMO_DIR}" rollback "${BASELINE}" --yes +# The rollback must have really reversed the migration, not just said so. +live=$(psql "${DATABASE_URL}" -Atc "SELECT agentic_schema_version() || '/' || (SELECT count(*) FROM information_schema.columns WHERE table_name = 'episodes' AND column_name = 'sentiment')") +echo "live schema / sentiment columns after rollback: ${live}" +if [[ "${live}" != "0.0.0/0" ]]; then + echo "error: expected 0.0.0/0 after rollback (schema reversed, sentiment column dropped)" >&2 + exit 1 +fi step "11. ask again — empathetic answer + clean memory" "${DEMO_DIR}/scripts/ask.sh" "I'm thinking about cancelling my subscription." From 4eb0bda27ba2e654fdc7be83f87947bce1aa08d5 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Fri, 20 Nov 2026 14:11:21 +0200 Subject: [PATCH 2/2] Resolve demo commit hashes directly and prove the migration applied Use agentic resolve HEAD instead of parsing status output, assert the migration is live before rollback so the post-rollback check means something, and re-record demo.cast with the schema bump and reversal. --- examples/langgraph-rollback/README.md | 4 +- examples/langgraph-rollback/demo.cast | 104 +++++++++++------- .../langgraph-rollback/scripts/run-demo.sh | 12 +- 3 files changed, 77 insertions(+), 43 deletions(-) diff --git a/examples/langgraph-rollback/README.md b/examples/langgraph-rollback/README.md index 2620897f..2244209f 100644 --- a/examples/langgraph-rollback/README.md +++ b/examples/langgraph-rollback/README.md @@ -157,13 +157,13 @@ export DATABASE_URL='postgres://agentic:agentic@localhost:54322/agentic' ./scripts/ask.sh "I'm thinking about cancelling." ../../target/release/agentic --repo . commit -m "baseline" \ --model "anthropic:claude-opus:2026-05-01" -BASELINE=$(../../target/release/agentic --repo . status | sed 's/.*→ //') +BASELINE=$(../../target/release/agentic --repo . resolve HEAD) ./scripts/deploy-bad-change.sh ./scripts/ask.sh "I'm thinking about cancelling." ../../target/release/agentic --repo . commit -m "bad change" \ --model "anthropic:claude-opus:2026-05-01" -BAD=$(../../target/release/agentic --repo . status | sed 's/.*→ //') +BAD=$(../../target/release/agentic --repo . resolve HEAD) # 4.5. Show that restoring only the prompt file does not fix it git checkout -- prompts/system.txt diff --git a/examples/langgraph-rollback/demo.cast b/examples/langgraph-rollback/demo.cast index ae62b721..5bd6495d 100644 --- a/examples/langgraph-rollback/demo.cast +++ b/examples/langgraph-rollback/demo.cast @@ -1,43 +1,69 @@ -{"version":3,"term":{"cols":80,"rows":24},"timestamp":1779285832,"command":"PATH=\"$PWD/.venv/bin:$PATH\" ./scripts/run-demo.sh","env":{"SHELL":"/bin/zsh"}} -[0.019, "o", "\r\n=== 1. starting Postgres + pgvector (podman) ===\r\n"] -[0.084, "o", "\u001b[4m>>>> Executing external compose provider \"/usr/local/bin/docker-compose\". Please see podman-compose(1) for how to disable this message. <<<<\r\n\r\n\u001b[0m"] -[0.161, "o", "\u001b[33mWARN\u001b[0m[0000] /examples/langgraph-rollback/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion \r\n"] -[0.103, "o", "\u001b[1A\u001b[1B\u001b[0G\u001b[?25l"] -[0.000, "o", "[+] Running 1/2\r\n"] -[0.000, "o", " \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n \u001b[33m⠋\u001b[0m Container agentic-demo-pg Starting\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"] -[0.043, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 2/2\u001b[0m\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"] -[0.000, "o", " \u001b[32m✔\u001b[0m Container agentic-demo-pg \u001b[32mStarted\u001b[0m\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"] -[1.294, "o", "ready after 2s\r\n"] +{"version":3,"term":{"cols":80,"rows":24},"timestamp":1795176651,"command":"PATH=\"$PWD/.venv/bin:$PATH\" ./scripts/run-demo.sh","env":{"SHELL":"/opt/homebrew/bin/bash"}} +[0.152, "o", "\r\n=== 0. python environment (venv + SDK deps) ===\r\n"] +[0.497, "o", "\r\n=== 1. starting Postgres + pgvector (docker) ===\r\n"] +[0.060, "o", "\u001b[33mWARN\u001b[0m[0000] ~/git.agentic/examples/langgraph-rollback/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion \r\n"] +[0.103, "o", "\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/2\r\n"] +[0.000, "o", " \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n \u001b[33m⠋\u001b[0m Container agentic-demo-pg Creating\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"] +[0.099, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 1/2\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"] +[0.000, "o", " \u001b[33m⠙\u001b[0m Container agentic-demo-pg Starting\u001b[34m0.2s \u001b[0m\r\n\u001b[?25h"] +[0.041, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 2/2\u001b[0m\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"] +[0.000, "o", " \u001b[32m✔\u001b[0m Container agentic-demo-pg \u001b[32mStarted\u001b[0m\u001b[34m0.2s \u001b[0m\r\n\u001b[?25h"] +[1.162, "o", "ready after 2s\r\n"] [0.000, "o", "\r\n=== 2. seeding episodes (baseline state) ===\r\n"] -[0.096, "o", "\r\n=== 3. building + starting agenticd ===\r\n"] -[0.937, "o", "\u001b[1m\u001b[92m Finished\u001b[0m `release` profile [optimized] target(s) in 0.90s\r\n"] -[0.002, "o", "\u001b[1m\u001b[33mwarning\u001b[0m: the following packages contain code that will be rejected by a future version of Rust: sqlx-postgres v0.7.4\r\n\u001b[1m\u001b[92mnote\u001b[0m: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`\r\n"] -[0.915, "o", "\r\n=== 4. baseline ask ===\r\n"] -[0.611, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] -[0.042, "o", "\r\n=== 5. commit baseline ===\r\n"] -[0.032, "o", "[c49f994] committed to langgraph/abbbd1ad6698ea81 as c49f9949ef43c677a3d9a7c5a59dd81f85cd68e99ee1310cfb9d1fa7d866971a\r\n"] -[0.015, "o", "baseline = c49f9949ef43c677a3d9a7c5a59dd81f85cd68e99ee1310cfb9d1fa7d866971a\r\n\r\n"] +[0.210, "o", "\r\n=== 3. building + starting agenticd ===\r\n"] +[0.063, "o", "warning: unused workspace dependency `once_cell`\r\n --> Cargo.toml:58:1\r\n |\r\n58 | once_cell = \"1.19\"\r\n | ^^^^^^^^^\r\n |\r\n = note: `cargo::unused_workspace_dependencies` is set to `warn` by default\r\nhelp: consider removing the workspace dependency `once_cell`"] +[0.000, "o", "\r\n"] +[0.001, "o", "warning: unused workspace dependency `prost`\r\n --> Cargo.toml:30:1\r\n |\r\n30"] +[0.000, "o", " | prost = \"0.12\"\r\n | "] +[0.000, "o", "^^^^^\r\n |\r\nhelp: consider removing the workspace dependency `prost`\r\n"] +[0.000, "o", "warning: unused workspace dependency `prost-build`\r\n --> Cargo.toml:31:1\r\n |\r\n31"] +[0.000, "o", " "] +[0.000, "o", "| prost-build = \"0.12\"\r\n | ^^^^^^^^^^^\r\n |"] +[0.000, "o", "\r\nhelp"] +[0.000, "o", ": consider removing the workspace dependency `prost-build`"] +[0.000, "o", "\r\n"] +[0.000, "o", "warning: unused workspace dependency `uuid`\r\n --> Cargo.toml:51:1\r\n |\r\n51 "] +[0.000, "o", "| uuid = { version = \"1.8\", features = [\"v7\"] }\r\n |"] +[0.000, "o", " "] +[0.000, "o", "^^^^"] +[0.000, "o", "\r\n "] +[0.000, "o", "|\r\nhelp: consider removing the workspace dependency `uuid`\r\n"] +[0.000, "o", "warning: unused field `authors` in `workspace.package`\r\n --> Cargo.toml:15:1\r\n |"] +[0.000, "o", "\r\n"] +[0.000, "o", "15 | authors = [\"git.agentic contributors\"]\r\n | ^^^^^^^\r\n |\r\n "] +[0.000, "o", "= note"] +[0.000, "o", ": `cargo::unused_workspace_package_fields` is set to `warn` by default\r\n"] +[0.000, "o", "help: consider removing the field `workspace.package.authors`\r\n"] +[0.000, "o", "warning: workspace (manifest) generated 5 warnings\r\n"] +[0.872, "o", " Finished `release` profile [optimized] target(s) in 0.91s\r\n"] +[0.875, "o", "\r\n=== 4. baseline ask ===\r\n"] +[0.878, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] +[0.087, "o", "\r\n=== 5. commit baseline ===\r\n"] +[0.035, "o", "[20eb9cf] committed to langgraph/abbbd1ad6698ea81 as 20eb9cf892835ed7c37574075a51d2a04482ffe36596a9f6ff16c793af3a7bea\r\n"] +[0.007, "o", "baseline = 20eb9cf892835ed7c37574075a51d2a04482ffe36596a9f6ff16c793af3a7bea\r\n\r\n"] [0.000, "o", "=== 6. deploy bad change ===\r\n"] -[0.009, "o", "→ swapping prompts/system.txt for the sycophantic variant\r\n"] -[0.003, "o", "→ INSERTing 5 contaminated rows into episodes\r\n"] -[0.034, "o", "✓ bad change deployed\r\n"] +[0.148, "o", "→ swapping prompts/system.txt for the sycophantic variant\r\n"] +[0.011, "o", "→ applying migration 001_add_sentiment (episodes.sentiment NOT NULL)\r\n"] +[0.099, "o", "→ INSERTing 5 contaminated rows into episodes\r\n"] +[0.035, "o", "✓ bad change deployed\r\n"] [0.000, "o", "\r\n=== 7. bad ask (hallucinated refund + contaminated memory) ===\r\n"] -[0.441, "o", "> Absolutely! I'll cancel your subscription and refund the full amount you've paid this year. Done!\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] -[0.042, "o", "\r\n=== 8. commit bad change ===\r\n"] -[0.030, "o", "[54886ec] committed to langgraph/abbbd1ad6698ea81 as 54886ec9b86430aa327dfa705f467dbf7e7a0d07de02b3ea452324ad6ba91e3a\r\n"] -[0.001, "o", "\r\n=== 8.5. simulate git revert — shows code-only revert is insufficient ===\r\n"] -[0.039, "o", "→ daemon still running at /examples/langgraph-rollback/.agentic/agenticd.sock\r\n→ prompts/system.txt after git revert:\r\n"] -[0.002, "o", " You are a helpful, empathetic customer support agent. Your job is to\r\n understand the customer's situation, ask clarifying questions when the\r\n"] -[0.001, "o", "→ (memory rows in Postgres are untouched by git revert)\r\n"] +[0.623, "o", "> Absolutely! I'll cancel your subscription and refund the full amount you've paid this year. Done!\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] +[0.082, "o", "\r\n=== 8. commit bad change ===\r\n"] +[0.032, "o", "[0a15e20] committed to langgraph/abbbd1ad6698ea81 as 0a15e2093c66346f49019a527f7bc88b68a9c507cb9f30b756b1772f430dd8e1\r\n"] +[0.008, "o", "\r\n=== 8.5. simulate git revert — shows code-only revert is insufficient ===\r\n"] +[0.042, "o", "→ daemon still running at /tmp/agentic-demo.phRYmZ/agenticd.sock\r\n→ prompts/system.txt after git revert:\r\n"] +[0.004, "o", " You are a helpful, empathetic customer support agent. Your job is to\r\n understand the customer's situation, ask clarifying questions when the\r\n"] +[0.000, "o", "→ (memory rows in Postgres are untouched by git revert)\r\n"] [0.000, "o", "✓ 'redeploy' complete — agent reads the reverted prompt on next ask\r\n"] -[0.418, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] -[0.040, "o", "\r\n=== 9. diff baseline → bad ===\r\n"] -[0.005, "o", "diff c49f994 → 9098f4c\r\n\r\nprompts:\r\n - system.txt\r\n ~ __langgraph__/abbbd1ad6698ea81/checkpoint.json\r\n\r\nmodel: changed\r\n\r\nmemory: memory snapshot removed (was d9f76eb)\r\n\r\nschema: schema_version 0.0.0 → (none)\r\n"] -[0.000, "o", "\r\n=== 10. rollback to baseline ===\r\n"] -[0.017, "o", " - target commit c49f994 (v0.7 baseline)\r\n - current branch tip 9098f4c → forward-record over it\r\n - schema already at 0.0.0 — no migrations needed\r\n - restore memory from manifest d9f76eb\r\n - rewrite /examples/langgraph-rollback/prompts from target prompts tree 2b742e6\r\n - model version captured in target — operator must redeploy if it changed\r\n - forward-recorded rollback as f903d50 on langgraph/abbbd1ad6698ea81\r\n✓ rollback complete; HEAD now at f903d50\r\n"] -[0.001, "o", "\r\n=== 11. ask again — empathetic answer + clean memory ===\r\n"] -[0.460, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] -[0.042, "o", "\r\n=== 12. log ===\r\n"] -[0.006, "o", "36a807c langgraph step 6 (loop)\r\na1dc861 langgraph step 5 (loop)\r\n51b06dd langgraph step 4 (loop)\r\n1648538 langgraph step 3 (input)\r\nf903d50 Rollback to c49f994\r\n9098f4c langgraph step 10 (loop)\r\n4420954 langgraph step 9 (loop)\r\n1876528 langgraph step 8 (loop)\r\na50576a langgraph step 7 (input)\r\n54886ec v0.8 friendlier prompt\r\n"] -[0.000, "o", "\r\n✓ broken-prompt demo complete\r\n"] -[0.311, "x", "0"] +[0.636, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] +[0.099, "o", "\r\n=== 9. diff baseline → bad ===\r\n"] +[0.007, "o", "diff 20eb9cf → 0a15e20\r\n\r\nprompts:\r\n ~ __langgraph__/abbbd1ad6698ea81/checkpoint.json\r\n ~ system.txt\r\n\r\nmemory: memory snapshot d9f76eb → d97dae0\r\n\r\nschema: schema_version 0.0.0 → 001_add_sentiment\r\n"] +[0.001, "o", "\r\n=== 10. rollback to baseline ===\r\n"] +[0.037, "o", "live schema / sentiment columns before rollback: 001_add_sentiment/1\r\n"] +[0.045, "o", " - target commit 20eb9cf (v0.7 baseline)\r\n - current branch tip c4794d5 → forward-record over it\r\n - reverse schema migrations: 001_add_sentiment → 0.0.0\r\n - restore memory from manifest d9f76eb\r\n - rewrite ~/git.agentic/examples/langgraph-rollback/prompts from target prompts tree 2e6a8d6\r\n - model version captured in target — operator must redeploy if it changed\r\n - forward-recorded rollback as 2a2893a on langgraph/abbbd1ad6698ea81\r\n✓ rollback complete; HEAD now at 2a2893a\r\n"] +[0.057, "o", "live schema / sentiment columns after rollback: 0.0.0/0\r\n\r\n=== 11. ask again — empathetic answer + clean memory ===\r\n"] +[0.786, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] +[0.107, "o", "\r\n=== 12. log ===\r\n"] +[0.006, "o", "eb43c59 langgraph step 6 (loop)\r\n89040c8 langgraph step 5 (loop)\r\n2b3d66b langgraph step 4 (loop)\r\n7e76411 langgraph step 3 (input)\r\n2a2893a Rollback to 20eb9cf\r\nc4794d5 langgraph step 10 (loop)\r\n8f2c619 langgraph step 9 (loop)\r\n715b7d1 langgraph step 8 (loop)\r\n99a256f langgraph step 7 (input)\r\n0a15e20 v0.8 friendlier prompt\r\n"] +[0.001, "o", "\r\n✓ broken-prompt demo complete\r\n"] +[0.656, "x", "0"] diff --git a/examples/langgraph-rollback/scripts/run-demo.sh b/examples/langgraph-rollback/scripts/run-demo.sh index 01031502..c92f0174 100755 --- a/examples/langgraph-rollback/scripts/run-demo.sh +++ b/examples/langgraph-rollback/scripts/run-demo.sh @@ -155,7 +155,7 @@ step "4. baseline ask" step "5. commit baseline" "${AGENTIC_BIN}" --repo "${DEMO_DIR}" commit -m "v0.7 baseline" \ --model "anthropic:claude-opus:2026-05-01" -BASELINE=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" status | sed -n 's/.*→ //p') +BASELINE=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" resolve HEAD) echo "baseline = ${BASELINE}" step "6. deploy bad change" @@ -167,7 +167,7 @@ step "7. bad ask (hallucinated refund + contaminated memory)" step "8. commit bad change" "${AGENTIC_BIN}" --repo "${DEMO_DIR}" commit -m "v0.8 friendlier prompt" \ --model "anthropic:claude-opus:2026-05-01" -BAD=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" status | sed -n 's/.*→ //p') +BAD=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" resolve HEAD) step "8.5. simulate git revert — shows code-only revert is insufficient" # Restore the prompt file to baseline (simulating what a code-only rollback, e.g. @@ -188,6 +188,14 @@ step "9. diff baseline → bad" "${AGENTIC_BIN}" --repo "${DEMO_DIR}" diff "${BASELINE}" "${BAD}" step "10. rollback to baseline" +# The bad change must really have applied the migration, or the check +# after rollback would pass without proving anything. +before=$(psql "${DATABASE_URL}" -Atc "SELECT agentic_schema_version() || '/' || (SELECT count(*) FROM information_schema.columns WHERE table_name = 'episodes' AND column_name = 'sentiment')") +echo "live schema / sentiment columns before rollback: ${before}" +if [[ "${before}" != "001_add_sentiment/1" ]]; then + echo "error: expected 001_add_sentiment/1 before rollback (migration applied)" >&2 + exit 1 +fi "${AGENTIC_BIN}" --repo "${DEMO_DIR}" rollback "${BASELINE}" --yes # The rollback must have really reversed the migration, not just said so. live=$(psql "${DATABASE_URL}" -Atc "SELECT agentic_schema_version() || '/' || (SELECT count(*) FROM information_schema.columns WHERE table_name = 'episodes' AND column_name = 'sentiment')")