diff --git a/examples/langgraph-rollback/README.md b/examples/langgraph-rollback/README.md index 31f093e3..2244209f 100644 --- a/examples/langgraph-rollback/README.md +++ b/examples/langgraph-rollback/README.md @@ -49,16 +49,20 @@ The script: 4. asks the agent *"I'm thinking about cancelling my subscription."* — gets an empathetic answer, 5. `agentic commit -m "v0.7 baseline"`, -6. `deploy-bad-change.sh` swaps in the sycophantic prompt + inserts - five contaminated rows ("Refund of $499 processed for customer +6. `deploy-bad-change.sh` swaps in the sycophantic prompt, applies + migration `001_add_sentiment` (a `NOT NULL` `episodes.sentiment` + column, with its `.down.sql` registered under `.agentic/schema/`) + and inserts five contaminated rows ("Refund of $499 processed for customer yesterday — confirmed.", etc.), 7. asks again — the agent now offers an unauthorised refund and confidently cites the fake refund row from memory, 8. `agentic commit -m "v0.8 friendlier prompt"`, -9. `agentic diff baseline HEAD` shows the prompts tree, the memory - manifest, and the LangGraph checkpoint blob have all moved, -10. `agentic rollback baseline --yes` — restores prompts on disk, - truncates and replays `episodes` from the baseline manifest inside +9. `agentic diff baseline ` shows the prompts tree, the + memory manifest, and the schema version (`0.0.0 → 001_add_sentiment`) + have all moved, +10. `agentic rollback baseline --yes` — reverses the schema migration + (using an approval token the CLI mints from `.agentic/approval.key`), + restores prompts on disk, truncates and replays `episodes` from the baseline manifest inside one transaction, forward-records a new commit on the branch, 11. asks again — empathetic answer is back; the contaminated memory row is gone. @@ -74,7 +78,7 @@ The fix is one command. `git revert` alone never restores step (11). | `tools` | unchanged | n/a | | `model` | recorded in each commit; unchanged in this demo | rollback notes "operator must redeploy if it changed" | | `code_sha` | recorded in each commit (from `git rev-parse HEAD`) | rollback target's code SHA threaded into the forward-recorded commit | -| `schema_version` | unchanged in this demo (`agentic_schema_version()` returns `0.0.0`) | gated on rollback; reverse migration runner is a planned follow-up | +| `schema_version` | migration `001_add_sentiment` adds `episodes.sentiment NOT NULL`; `agentic_schema_version()` goes `0.0.0` → `001_add_sentiment` | `agentic rollback` runs `.agentic/schema/001_add_sentiment.down.sql` in one transaction, gated by an approval token ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)) | The LangGraph wiring is real: every agent invocation runs through `AgenticCheckpointer`, so `agentic log` shows one commit per graph step @@ -90,15 +94,15 @@ history reflects the rollback action rather than rewriting it. enough to make the broken behaviour reproducible without API keys and without flaky model output. The rollback story is identical for a real LLM-backed agent — the dimensions snapshotted are the same. -- **No schema migration.** The demo spec describes a `sentiment NOT - NULL` column added in v0.8 and reverse-migrated on rollback; that - requires the reverse SQL migration runner which is the next planned - ADR-0002 §5 follow-up. Today the demo exercises prompts + memory - rollback only. A rollback that does run reverse migrations needs an - operator approval token ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)). - The daemon reads `.agentic/approval.key` and the `agentic` CLI mints +- **The migration is hand-rolled.** There is no forward migration + runner: `deploy-bad-change.sh` applies `bad-change/001_add_sentiment.up.sql` + with `psql` (which also records the name in `agentic_migrations`) and + copies the `.down.sql` into `.agentic/schema/`. The rollback side is + the real reverse-migration runner. It needs an operator approval token + ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)); + the daemon reads `.agentic/approval.key` and the `agentic` CLI mints the token from the same file, so `agentic rollback` is typed the same - way; a socket client without the key cannot run reverse migrations. + way, and a socket client without the key cannot run reverse migrations. - **Compose covers the database only.** `docker-compose.yml` starts Postgres + pgvector. `agenticd` and the Python agent are built and run locally by `scripts/run-demo.sh` — they are not containerised. @@ -113,11 +117,13 @@ examples/langgraph-rollback/ │ └── system.txt baseline empathetic system prompt ├── bad-change/ │ ├── system.txt sycophantic variant the bad change ships +│ ├── 001_add_sentiment.up.sql forward migration (NOT NULL column) +│ ├── 001_add_sentiment.down.sql its reverse, copied to .agentic/schema/ │ └── seed.sql contaminated rows the bad change INSERTs ├── seed.sql baseline episodes + pgvector setup ├── scripts/ │ ├── ask.sh invoke the agent with one user message -│ ├── deploy-bad-change.sh apply prompt + memory contamination +│ ├── deploy-bad-change.sh apply prompt + migration + memory contamination │ ├── redeploy.sh simulate code-only redeploy after git revert │ └── run-demo.sh full scenario orchestrator ├── pyproject.toml Python deps for the demo @@ -151,19 +157,20 @@ export DATABASE_URL='postgres://agentic:agentic@localhost:54322/agentic' ./scripts/ask.sh "I'm thinking about cancelling." ../../target/release/agentic --repo . commit -m "baseline" \ --model "anthropic:claude-opus:2026-05-01" -BASELINE=$(../../target/release/agentic --repo . status | sed 's/.*→ //') +BASELINE=$(../../target/release/agentic --repo . resolve HEAD) ./scripts/deploy-bad-change.sh ./scripts/ask.sh "I'm thinking about cancelling." ../../target/release/agentic --repo . commit -m "bad change" \ --model "anthropic:claude-opus:2026-05-01" +BAD=$(../../target/release/agentic --repo . resolve HEAD) # 4.5. Show that restoring only the prompt file does not fix it git checkout -- prompts/system.txt ./scripts/redeploy.sh ./scripts/ask.sh "I'm thinking about cancelling." # still broken -../../target/release/agentic --repo . diff "$BASELINE" HEAD +../../target/release/agentic --repo . diff "$BASELINE" "$BAD" ../../target/release/agentic --repo . rollback "$BASELINE" --yes ./scripts/ask.sh "I'm thinking about cancelling." # fixed diff --git a/examples/langgraph-rollback/bad-change/001_add_sentiment.down.sql b/examples/langgraph-rollback/bad-change/001_add_sentiment.down.sql new file mode 100644 index 00000000..103580f9 --- /dev/null +++ b/examples/langgraph-rollback/bad-change/001_add_sentiment.down.sql @@ -0,0 +1,6 @@ +-- Reverse of 001_add_sentiment.up.sql. deploy-bad-change.sh copies it to +-- .agentic/schema/; `agentic rollback` runs it (with the matching +-- agentic_migrations delete) in one transaction before restoring memory. +-- No BEGIN/COMMIT here: the daemon owns the transaction. + +ALTER TABLE episodes DROP COLUMN sentiment; diff --git a/examples/langgraph-rollback/bad-change/001_add_sentiment.up.sql b/examples/langgraph-rollback/bad-change/001_add_sentiment.up.sql new file mode 100644 index 00000000..83260254 --- /dev/null +++ b/examples/langgraph-rollback/bad-change/001_add_sentiment.up.sql @@ -0,0 +1,8 @@ +-- Forward migration the bad change ships: a NOT NULL `sentiment` column +-- on the agent's memory table. A code-only `git revert` cannot undo it. +-- The agentic_migrations row is what `agentic_schema_version()` reads, +-- so the next `agentic commit` records this version in the Commit. +-- Applied by deploy-bad-change.sh under `psql -1` (one transaction). + +ALTER TABLE episodes ADD COLUMN sentiment text NOT NULL DEFAULT 'neutral'; +INSERT INTO agentic_migrations (name) VALUES ('001_add_sentiment'); diff --git a/examples/langgraph-rollback/demo.cast b/examples/langgraph-rollback/demo.cast index ae62b721..5bd6495d 100644 --- a/examples/langgraph-rollback/demo.cast +++ b/examples/langgraph-rollback/demo.cast @@ -1,43 +1,69 @@ -{"version":3,"term":{"cols":80,"rows":24},"timestamp":1779285832,"command":"PATH=\"$PWD/.venv/bin:$PATH\" ./scripts/run-demo.sh","env":{"SHELL":"/bin/zsh"}} -[0.019, "o", "\r\n=== 1. starting Postgres + pgvector (podman) ===\r\n"] -[0.084, "o", "\u001b[4m>>>> Executing external compose provider \"/usr/local/bin/docker-compose\". Please see podman-compose(1) for how to disable this message. <<<<\r\n\r\n\u001b[0m"] -[0.161, "o", "\u001b[33mWARN\u001b[0m[0000] /examples/langgraph-rollback/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion \r\n"] -[0.103, "o", "\u001b[1A\u001b[1B\u001b[0G\u001b[?25l"] -[0.000, "o", "[+] Running 1/2\r\n"] -[0.000, "o", " \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n \u001b[33m⠋\u001b[0m Container agentic-demo-pg Starting\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"] -[0.043, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 2/2\u001b[0m\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"] -[0.000, "o", " \u001b[32m✔\u001b[0m Container agentic-demo-pg \u001b[32mStarted\u001b[0m\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"] -[1.294, "o", "ready after 2s\r\n"] +{"version":3,"term":{"cols":80,"rows":24},"timestamp":1795176651,"command":"PATH=\"$PWD/.venv/bin:$PATH\" ./scripts/run-demo.sh","env":{"SHELL":"/opt/homebrew/bin/bash"}} +[0.152, "o", "\r\n=== 0. python environment (venv + SDK deps) ===\r\n"] +[0.497, "o", "\r\n=== 1. starting Postgres + pgvector (docker) ===\r\n"] +[0.060, "o", "\u001b[33mWARN\u001b[0m[0000] ~/git.agentic/examples/langgraph-rollback/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion \r\n"] +[0.103, "o", "\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/2\r\n"] +[0.000, "o", " \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n \u001b[33m⠋\u001b[0m Container agentic-demo-pg Creating\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"] +[0.099, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 1/2\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"] +[0.000, "o", " \u001b[33m⠙\u001b[0m Container agentic-demo-pg Starting\u001b[34m0.2s \u001b[0m\r\n\u001b[?25h"] +[0.041, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 2/2\u001b[0m\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"] +[0.000, "o", " \u001b[32m✔\u001b[0m Container agentic-demo-pg \u001b[32mStarted\u001b[0m\u001b[34m0.2s \u001b[0m\r\n\u001b[?25h"] +[1.162, "o", "ready after 2s\r\n"] [0.000, "o", "\r\n=== 2. seeding episodes (baseline state) ===\r\n"] -[0.096, "o", "\r\n=== 3. building + starting agenticd ===\r\n"] -[0.937, "o", "\u001b[1m\u001b[92m Finished\u001b[0m `release` profile [optimized] target(s) in 0.90s\r\n"] -[0.002, "o", "\u001b[1m\u001b[33mwarning\u001b[0m: the following packages contain code that will be rejected by a future version of Rust: sqlx-postgres v0.7.4\r\n\u001b[1m\u001b[92mnote\u001b[0m: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`\r\n"] -[0.915, "o", "\r\n=== 4. baseline ask ===\r\n"] -[0.611, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] -[0.042, "o", "\r\n=== 5. commit baseline ===\r\n"] -[0.032, "o", "[c49f994] committed to langgraph/abbbd1ad6698ea81 as c49f9949ef43c677a3d9a7c5a59dd81f85cd68e99ee1310cfb9d1fa7d866971a\r\n"] -[0.015, "o", "baseline = c49f9949ef43c677a3d9a7c5a59dd81f85cd68e99ee1310cfb9d1fa7d866971a\r\n\r\n"] +[0.210, "o", "\r\n=== 3. building + starting agenticd ===\r\n"] +[0.063, "o", "warning: unused workspace dependency `once_cell`\r\n --> Cargo.toml:58:1\r\n |\r\n58 | once_cell = \"1.19\"\r\n | ^^^^^^^^^\r\n |\r\n = note: `cargo::unused_workspace_dependencies` is set to `warn` by default\r\nhelp: consider removing the workspace dependency `once_cell`"] +[0.000, "o", "\r\n"] +[0.001, "o", "warning: unused workspace dependency `prost`\r\n --> Cargo.toml:30:1\r\n |\r\n30"] +[0.000, "o", " | prost = \"0.12\"\r\n | "] +[0.000, "o", "^^^^^\r\n |\r\nhelp: consider removing the workspace dependency `prost`\r\n"] +[0.000, "o", "warning: unused workspace dependency `prost-build`\r\n --> Cargo.toml:31:1\r\n |\r\n31"] +[0.000, "o", " "] +[0.000, "o", "| prost-build = \"0.12\"\r\n | ^^^^^^^^^^^\r\n |"] +[0.000, "o", "\r\nhelp"] +[0.000, "o", ": consider removing the workspace dependency `prost-build`"] +[0.000, "o", "\r\n"] +[0.000, "o", "warning: unused workspace dependency `uuid`\r\n --> Cargo.toml:51:1\r\n |\r\n51 "] +[0.000, "o", "| uuid = { version = \"1.8\", features = [\"v7\"] }\r\n |"] +[0.000, "o", " "] +[0.000, "o", "^^^^"] +[0.000, "o", "\r\n "] +[0.000, "o", "|\r\nhelp: consider removing the workspace dependency `uuid`\r\n"] +[0.000, "o", "warning: unused field `authors` in `workspace.package`\r\n --> Cargo.toml:15:1\r\n |"] +[0.000, "o", "\r\n"] +[0.000, "o", "15 | authors = [\"git.agentic contributors\"]\r\n | ^^^^^^^\r\n |\r\n "] +[0.000, "o", "= note"] +[0.000, "o", ": `cargo::unused_workspace_package_fields` is set to `warn` by default\r\n"] +[0.000, "o", "help: consider removing the field `workspace.package.authors`\r\n"] +[0.000, "o", "warning: workspace (manifest) generated 5 warnings\r\n"] +[0.872, "o", " Finished `release` profile [optimized] target(s) in 0.91s\r\n"] +[0.875, "o", "\r\n=== 4. baseline ask ===\r\n"] +[0.878, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] +[0.087, "o", "\r\n=== 5. commit baseline ===\r\n"] +[0.035, "o", "[20eb9cf] committed to langgraph/abbbd1ad6698ea81 as 20eb9cf892835ed7c37574075a51d2a04482ffe36596a9f6ff16c793af3a7bea\r\n"] +[0.007, "o", "baseline = 20eb9cf892835ed7c37574075a51d2a04482ffe36596a9f6ff16c793af3a7bea\r\n\r\n"] [0.000, "o", "=== 6. deploy bad change ===\r\n"] -[0.009, "o", "→ swapping prompts/system.txt for the sycophantic variant\r\n"] -[0.003, "o", "→ INSERTing 5 contaminated rows into episodes\r\n"] -[0.034, "o", "✓ bad change deployed\r\n"] +[0.148, "o", "→ swapping prompts/system.txt for the sycophantic variant\r\n"] +[0.011, "o", "→ applying migration 001_add_sentiment (episodes.sentiment NOT NULL)\r\n"] +[0.099, "o", "→ INSERTing 5 contaminated rows into episodes\r\n"] +[0.035, "o", "✓ bad change deployed\r\n"] [0.000, "o", "\r\n=== 7. bad ask (hallucinated refund + contaminated memory) ===\r\n"] -[0.441, "o", "> Absolutely! I'll cancel your subscription and refund the full amount you've paid this year. Done!\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] -[0.042, "o", "\r\n=== 8. commit bad change ===\r\n"] -[0.030, "o", "[54886ec] committed to langgraph/abbbd1ad6698ea81 as 54886ec9b86430aa327dfa705f467dbf7e7a0d07de02b3ea452324ad6ba91e3a\r\n"] -[0.001, "o", "\r\n=== 8.5. simulate git revert — shows code-only revert is insufficient ===\r\n"] -[0.039, "o", "→ daemon still running at /examples/langgraph-rollback/.agentic/agenticd.sock\r\n→ prompts/system.txt after git revert:\r\n"] -[0.002, "o", " You are a helpful, empathetic customer support agent. Your job is to\r\n understand the customer's situation, ask clarifying questions when the\r\n"] -[0.001, "o", "→ (memory rows in Postgres are untouched by git revert)\r\n"] +[0.623, "o", "> Absolutely! I'll cancel your subscription and refund the full amount you've paid this year. Done!\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] +[0.082, "o", "\r\n=== 8. commit bad change ===\r\n"] +[0.032, "o", "[0a15e20] committed to langgraph/abbbd1ad6698ea81 as 0a15e2093c66346f49019a527f7bc88b68a9c507cb9f30b756b1772f430dd8e1\r\n"] +[0.008, "o", "\r\n=== 8.5. simulate git revert — shows code-only revert is insufficient ===\r\n"] +[0.042, "o", "→ daemon still running at /tmp/agentic-demo.phRYmZ/agenticd.sock\r\n→ prompts/system.txt after git revert:\r\n"] +[0.004, "o", " You are a helpful, empathetic customer support agent. Your job is to\r\n understand the customer's situation, ask clarifying questions when the\r\n"] +[0.000, "o", "→ (memory rows in Postgres are untouched by git revert)\r\n"] [0.000, "o", "✓ 'redeploy' complete — agent reads the reverted prompt on next ask\r\n"] -[0.418, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] -[0.040, "o", "\r\n=== 9. diff baseline → bad ===\r\n"] -[0.005, "o", "diff c49f994 → 9098f4c\r\n\r\nprompts:\r\n - system.txt\r\n ~ __langgraph__/abbbd1ad6698ea81/checkpoint.json\r\n\r\nmodel: changed\r\n\r\nmemory: memory snapshot removed (was d9f76eb)\r\n\r\nschema: schema_version 0.0.0 → (none)\r\n"] -[0.000, "o", "\r\n=== 10. rollback to baseline ===\r\n"] -[0.017, "o", " - target commit c49f994 (v0.7 baseline)\r\n - current branch tip 9098f4c → forward-record over it\r\n - schema already at 0.0.0 — no migrations needed\r\n - restore memory from manifest d9f76eb\r\n - rewrite /examples/langgraph-rollback/prompts from target prompts tree 2b742e6\r\n - model version captured in target — operator must redeploy if it changed\r\n - forward-recorded rollback as f903d50 on langgraph/abbbd1ad6698ea81\r\n✓ rollback complete; HEAD now at f903d50\r\n"] -[0.001, "o", "\r\n=== 11. ask again — empathetic answer + clean memory ===\r\n"] -[0.460, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] -[0.042, "o", "\r\n=== 12. log ===\r\n"] -[0.006, "o", "36a807c langgraph step 6 (loop)\r\na1dc861 langgraph step 5 (loop)\r\n51b06dd langgraph step 4 (loop)\r\n1648538 langgraph step 3 (input)\r\nf903d50 Rollback to c49f994\r\n9098f4c langgraph step 10 (loop)\r\n4420954 langgraph step 9 (loop)\r\n1876528 langgraph step 8 (loop)\r\na50576a langgraph step 7 (input)\r\n54886ec v0.8 friendlier prompt\r\n"] -[0.000, "o", "\r\n✓ broken-prompt demo complete\r\n"] -[0.311, "x", "0"] +[0.636, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"] +[0.099, "o", "\r\n=== 9. diff baseline → bad ===\r\n"] +[0.007, "o", "diff 20eb9cf → 0a15e20\r\n\r\nprompts:\r\n ~ __langgraph__/abbbd1ad6698ea81/checkpoint.json\r\n ~ system.txt\r\n\r\nmemory: memory snapshot d9f76eb → d97dae0\r\n\r\nschema: schema_version 0.0.0 → 001_add_sentiment\r\n"] +[0.001, "o", "\r\n=== 10. rollback to baseline ===\r\n"] +[0.037, "o", "live schema / sentiment columns before rollback: 001_add_sentiment/1\r\n"] +[0.045, "o", " - target commit 20eb9cf (v0.7 baseline)\r\n - current branch tip c4794d5 → forward-record over it\r\n - reverse schema migrations: 001_add_sentiment → 0.0.0\r\n - restore memory from manifest d9f76eb\r\n - rewrite ~/git.agentic/examples/langgraph-rollback/prompts from target prompts tree 2e6a8d6\r\n - model version captured in target — operator must redeploy if it changed\r\n - forward-recorded rollback as 2a2893a on langgraph/abbbd1ad6698ea81\r\n✓ rollback complete; HEAD now at 2a2893a\r\n"] +[0.057, "o", "live schema / sentiment columns after rollback: 0.0.0/0\r\n\r\n=== 11. ask again — empathetic answer + clean memory ===\r\n"] +[0.786, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"] +[0.107, "o", "\r\n=== 12. log ===\r\n"] +[0.006, "o", "eb43c59 langgraph step 6 (loop)\r\n89040c8 langgraph step 5 (loop)\r\n2b3d66b langgraph step 4 (loop)\r\n7e76411 langgraph step 3 (input)\r\n2a2893a Rollback to 20eb9cf\r\nc4794d5 langgraph step 10 (loop)\r\n8f2c619 langgraph step 9 (loop)\r\n715b7d1 langgraph step 8 (loop)\r\n99a256f langgraph step 7 (input)\r\n0a15e20 v0.8 friendlier prompt\r\n"] +[0.001, "o", "\r\n✓ broken-prompt demo complete\r\n"] +[0.656, "x", "0"] diff --git a/examples/langgraph-rollback/scripts/deploy-bad-change.sh b/examples/langgraph-rollback/scripts/deploy-bad-change.sh index 082b62b4..e1b477c8 100755 --- a/examples/langgraph-rollback/scripts/deploy-bad-change.sh +++ b/examples/langgraph-rollback/scripts/deploy-bad-change.sh @@ -2,12 +2,15 @@ # Apply the bad change — the simulated "small prompt tweak" that # triggers the broken-prompt scenario. # -# Two things land at once, the same way they would in a real deploy: +# Three things land at once, the same way they would in a real deploy: # # 1. The sycophantic system prompt overwrites the empathetic one. -# 2. Contaminated rows are INSERTed into the agent's `episodes` -# memory table — these are what makes a code-only `git revert` -# insufficient. +# 2. A schema migration adds a NOT NULL `sentiment` column to the +# agent's `episodes` memory table and records itself in +# `agentic_migrations`. Its `.down.sql` is registered under +# `.agentic/schema/` so `agentic rollback` can reverse it. +# 3. Contaminated rows are INSERTed into `episodes` — together with +# (2) these are what make a code-only `git revert` insufficient. # # Neither change is committed to agenticd here; run-demo.sh does that # right after, so the bad state lands as a single commit. @@ -21,6 +24,14 @@ here="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" echo "→ swapping prompts/system.txt for the sycophantic variant" cp "${here}/bad-change/system.txt" "${here}/prompts/system.txt" +echo "→ applying migration 001_add_sentiment (episodes.sentiment NOT NULL)" +psql "${DATABASE_URL}" -v ON_ERROR_STOP=1 -1 -f "${here}/bad-change/001_add_sentiment.up.sql" >/dev/null +# Operator-owned reverse migration. .agentic is 0700 and already exists +# (run-demo.sh runs `agentic init` first); the daemon reads the file at +# rollback time, so no restart is needed. +mkdir -p "${here}/.agentic/schema" +cp "${here}/bad-change/001_add_sentiment.down.sql" "${here}/.agentic/schema/001_add_sentiment.down.sql" + echo "→ INSERTing 5 contaminated rows into episodes" psql "${DATABASE_URL}" -v ON_ERROR_STOP=1 -f "${here}/bad-change/seed.sql" >/dev/null diff --git a/examples/langgraph-rollback/scripts/run-demo.sh b/examples/langgraph-rollback/scripts/run-demo.sh index efd2b4fc..c92f0174 100755 --- a/examples/langgraph-rollback/scripts/run-demo.sh +++ b/examples/langgraph-rollback/scripts/run-demo.sh @@ -12,12 +12,12 @@ # 3. build agenticd + start it bound to the demo repo # 4. agent baseline ask # 5. `agentic commit` baseline -# 6. deploy-bad-change.sh +# 6. deploy-bad-change.sh ← prompt + schema migration + bad rows # 7. agent bad ask ← the hallucinated refund # 8. `agentic commit` "bad change" # 8.5. simulate git revert ← still broken (contaminated memory) # 9. `agentic diff` baseline → bad ← multi-dimensional regression -# 10. `agentic rollback baseline --yes` +# 10. `agentic rollback baseline --yes` ← incl. reverse migration # 11. agent ask again ← empathetic answer is back # 12. cleanup @@ -111,8 +111,8 @@ chmod go-w "${DEMO_DIR}" "${DEMO_DIR}/.agentic" # Operator approval key (ADR-0019). A rollback that reverses a schema # migration needs an approval token; the daemon reads this key and the # `agentic` CLI mints the token from it, so the visible commands stay the -# same. The demo's step 10 rollback has no schema change today and so -# consumes no token. The key must be 0600 or the daemon refuses to start. +# same. The step 10 rollback reverses the bad change's schema migration, +# so it consumes one token. The key must be 0600 or the daemon refuses to start. APPROVAL_KEY="${DEMO_DIR}/.agentic/approval.key" if [[ ! -f "${APPROVAL_KEY}" ]]; then ( umask 077 && head -c 32 /dev/urandom > "${APPROVAL_KEY}" ) @@ -155,7 +155,7 @@ step "4. baseline ask" step "5. commit baseline" "${AGENTIC_BIN}" --repo "${DEMO_DIR}" commit -m "v0.7 baseline" \ --model "anthropic:claude-opus:2026-05-01" -BASELINE=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" status | sed -n 's/.*→ //p') +BASELINE=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" resolve HEAD) echo "baseline = ${BASELINE}" step "6. deploy bad change" @@ -167,6 +167,7 @@ step "7. bad ask (hallucinated refund + contaminated memory)" step "8. commit bad change" "${AGENTIC_BIN}" --repo "${DEMO_DIR}" commit -m "v0.8 friendlier prompt" \ --model "anthropic:claude-opus:2026-05-01" +BAD=$("${AGENTIC_BIN}" --repo "${DEMO_DIR}" resolve HEAD) step "8.5. simulate git revert — shows code-only revert is insufficient" # Restore the prompt file to baseline (simulating what a code-only rollback, e.g. @@ -182,10 +183,27 @@ fi # Contaminated memory rows are still in Postgres — the answer is still broken. step "9. diff baseline → bad" -"${AGENTIC_BIN}" --repo "${DEMO_DIR}" diff "${BASELINE}" HEAD +# Diff against the bad commit itself, not HEAD: the step 8.5 ask adds +# LangGraph checkpoint commits that carry no memory or schema dimension. +"${AGENTIC_BIN}" --repo "${DEMO_DIR}" diff "${BASELINE}" "${BAD}" step "10. rollback to baseline" +# The bad change must really have applied the migration, or the check +# after rollback would pass without proving anything. +before=$(psql "${DATABASE_URL}" -Atc "SELECT agentic_schema_version() || '/' || (SELECT count(*) FROM information_schema.columns WHERE table_name = 'episodes' AND column_name = 'sentiment')") +echo "live schema / sentiment columns before rollback: ${before}" +if [[ "${before}" != "001_add_sentiment/1" ]]; then + echo "error: expected 001_add_sentiment/1 before rollback (migration applied)" >&2 + exit 1 +fi "${AGENTIC_BIN}" --repo "${DEMO_DIR}" rollback "${BASELINE}" --yes +# The rollback must have really reversed the migration, not just said so. +live=$(psql "${DATABASE_URL}" -Atc "SELECT agentic_schema_version() || '/' || (SELECT count(*) FROM information_schema.columns WHERE table_name = 'episodes' AND column_name = 'sentiment')") +echo "live schema / sentiment columns after rollback: ${live}" +if [[ "${live}" != "0.0.0/0" ]]; then + echo "error: expected 0.0.0/0 after rollback (schema reversed, sentiment column dropped)" >&2 + exit 1 +fi step "11. ask again — empathetic answer + clean memory" "${DEMO_DIR}/scripts/ask.sh" "I'm thinking about cancelling my subscription."