Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 25 additions & 18 deletions examples/langgraph-rollback/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,20 @@ The script:
4. asks the agent *"I'm thinking about cancelling my subscription."* —
gets an empathetic answer,
5. `agentic commit -m "v0.7 baseline"`,
6. `deploy-bad-change.sh` swaps in the sycophantic prompt + inserts
five contaminated rows ("Refund of $499 processed for customer
6. `deploy-bad-change.sh` swaps in the sycophantic prompt, applies
migration `001_add_sentiment` (a `NOT NULL` `episodes.sentiment`
column, with its `.down.sql` registered under `.agentic/schema/`)
and inserts five contaminated rows ("Refund of $499 processed for customer
yesterday — confirmed.", etc.),
7. asks again — the agent now offers an unauthorised refund and
confidently cites the fake refund row from memory,
8. `agentic commit -m "v0.8 friendlier prompt"`,
9. `agentic diff baseline HEAD` shows the prompts tree, the memory
manifest, and the LangGraph checkpoint blob have all moved,
10. `agentic rollback baseline --yes` — restores prompts on disk,
truncates and replays `episodes` from the baseline manifest inside
9. `agentic diff baseline <bad commit>` shows the prompts tree, the
memory manifest, and the schema version (`0.0.0 → 001_add_sentiment`)
have all moved,
10. `agentic rollback baseline --yes` — reverses the schema migration
(using an approval token the CLI mints from `.agentic/approval.key`),
restores prompts on disk, truncates and replays `episodes` from the baseline manifest inside
one transaction, forward-records a new commit on the branch,
11. asks again — empathetic answer is back; the contaminated memory
row is gone.
Expand All @@ -74,7 +78,7 @@ The fix is one command. `git revert` alone never restores step (11).
| `tools` | unchanged | n/a |
| `model` | recorded in each commit; unchanged in this demo | rollback notes "operator must redeploy if it changed" |
| `code_sha` | recorded in each commit (from `git rev-parse HEAD`) | rollback target's code SHA threaded into the forward-recorded commit |
| `schema_version` | unchanged in this demo (`agentic_schema_version()` returns `0.0.0`) | gated on rollback; reverse migration runner is a planned follow-up |
| `schema_version` | migration `001_add_sentiment` adds `episodes.sentiment NOT NULL`; `agentic_schema_version()` goes `0.0.0` → `001_add_sentiment` | `agentic rollback` runs `.agentic/schema/001_add_sentiment.down.sql` in one transaction, gated by an approval token ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)) |

The LangGraph wiring is real: every agent invocation runs through
`AgenticCheckpointer`, so `agentic log` shows one commit per graph step
Expand All @@ -90,15 +94,15 @@ history reflects the rollback action rather than rewriting it.
enough to make the broken behaviour reproducible without API keys
and without flaky model output. The rollback story is identical for
a real LLM-backed agent — the dimensions snapshotted are the same.
- **No schema migration.** The demo spec describes a `sentiment NOT
NULL` column added in v0.8 and reverse-migrated on rollback; that
requires the reverse SQL migration runner which is the next planned
ADR-0002 §5 follow-up. Today the demo exercises prompts + memory
rollback only. A rollback that does run reverse migrations needs an
operator approval token ([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md)).
The daemon reads `.agentic/approval.key` and the `agentic` CLI mints
- **The migration is hand-rolled.** There is no forward migration
runner: `deploy-bad-change.sh` applies `bad-change/001_add_sentiment.up.sql`
with `psql` (which also records the name in `agentic_migrations`) and
copies the `.down.sql` into `.agentic/schema/`. The rollback side is
the real reverse-migration runner. It needs an operator approval token
([ADR-0019](../../docs/adr/0019-approval-for-reverse-migration-rollback.md));
the daemon reads `.agentic/approval.key` and the `agentic` CLI mints
the token from the same file, so `agentic rollback` is typed the same
way; a socket client without the key cannot run reverse migrations.
way, and a socket client without the key cannot run reverse migrations.
- **Compose covers the database only.** `docker-compose.yml` starts
Postgres + pgvector. `agenticd` and the Python agent are built and
run locally by `scripts/run-demo.sh` — they are not containerised.
Expand All @@ -113,11 +117,13 @@ examples/langgraph-rollback/
│ └── system.txt baseline empathetic system prompt
├── bad-change/
│ ├── system.txt sycophantic variant the bad change ships
│ ├── 001_add_sentiment.up.sql forward migration (NOT NULL column)
│ ├── 001_add_sentiment.down.sql its reverse, copied to .agentic/schema/
│ └── seed.sql contaminated rows the bad change INSERTs
├── seed.sql baseline episodes + pgvector setup
├── scripts/
│ ├── ask.sh invoke the agent with one user message
│ ├── deploy-bad-change.sh apply prompt + memory contamination
│ ├── deploy-bad-change.sh apply prompt + migration + memory contamination
│ ├── redeploy.sh simulate code-only redeploy after git revert
│ └── run-demo.sh full scenario orchestrator
├── pyproject.toml Python deps for the demo
Expand Down Expand Up @@ -151,19 +157,20 @@ export DATABASE_URL='postgres://agentic:agentic@localhost:54322/agentic'
./scripts/ask.sh "I'm thinking about cancelling."
../../target/release/agentic --repo . commit -m "baseline" \
--model "anthropic:claude-opus:2026-05-01"
BASELINE=$(../../target/release/agentic --repo . status | sed 's/.*→ //')
BASELINE=$(../../target/release/agentic --repo . resolve HEAD)

./scripts/deploy-bad-change.sh
./scripts/ask.sh "I'm thinking about cancelling."
../../target/release/agentic --repo . commit -m "bad change" \
--model "anthropic:claude-opus:2026-05-01"
BAD=$(../../target/release/agentic --repo . resolve HEAD)

# 4.5. Show that restoring only the prompt file does not fix it
git checkout -- prompts/system.txt
./scripts/redeploy.sh
./scripts/ask.sh "I'm thinking about cancelling." # still broken

../../target/release/agentic --repo . diff "$BASELINE" HEAD
../../target/release/agentic --repo . diff "$BASELINE" "$BAD"
../../target/release/agentic --repo . rollback "$BASELINE" --yes
./scripts/ask.sh "I'm thinking about cancelling." # fixed

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
-- Reverse of 001_add_sentiment.up.sql. deploy-bad-change.sh copies it to
-- .agentic/schema/; `agentic rollback` runs it (with the matching
-- agentic_migrations delete) in one transaction before restoring memory.
-- No BEGIN/COMMIT here: the daemon owns the transaction.

ALTER TABLE episodes DROP COLUMN sentiment;
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-- Forward migration the bad change ships: a NOT NULL `sentiment` column
-- on the agent's memory table. A code-only `git revert` cannot undo it.
-- The agentic_migrations row is what `agentic_schema_version()` reads,
-- so the next `agentic commit` records this version in the Commit.
-- Applied by deploy-bad-change.sh under `psql -1` (one transaction).

ALTER TABLE episodes ADD COLUMN sentiment text NOT NULL DEFAULT 'neutral';
INSERT INTO agentic_migrations (name) VALUES ('001_add_sentiment');
104 changes: 65 additions & 39 deletions examples/langgraph-rollback/demo.cast
Original file line number Diff line number Diff line change
@@ -1,43 +1,69 @@
{"version":3,"term":{"cols":80,"rows":24},"timestamp":1779285832,"command":"PATH=\"$PWD/.venv/bin:$PATH\" ./scripts/run-demo.sh","env":{"SHELL":"/bin/zsh"}}
[0.019, "o", "\r\n=== 1. starting Postgres + pgvector (podman) ===\r\n"]
[0.084, "o", "\u001b[4m>>>> Executing external compose provider \"/usr/local/bin/docker-compose\". Please see podman-compose(1) for how to disable this message. <<<<\r\n\r\n\u001b[0m"]
[0.161, "o", "\u001b[33mWARN\u001b[0m[0000] <repo>/examples/langgraph-rollback/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion \r\n"]
[0.103, "o", "\u001b[1A\u001b[1B\u001b[0G\u001b[?25l"]
[0.000, "o", "[+] Running 1/2\r\n"]
[0.000, "o", " \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n \u001b[33m⠋\u001b[0m Container agentic-demo-pg Starting\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"]
[0.043, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 2/2\u001b[0m\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"]
[0.000, "o", " \u001b[32m✔\u001b[0m Container agentic-demo-pg \u001b[32mStarted\u001b[0m\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"]
[1.294, "o", "ready after 2s\r\n"]
{"version":3,"term":{"cols":80,"rows":24},"timestamp":1795176651,"command":"PATH=\"$PWD/.venv/bin:$PATH\" ./scripts/run-demo.sh","env":{"SHELL":"/opt/homebrew/bin/bash"}}
[0.152, "o", "\r\n=== 0. python environment (venv + SDK deps) ===\r\n"]
[0.497, "o", "\r\n=== 1. starting Postgres + pgvector (docker) ===\r\n"]
[0.060, "o", "\u001b[33mWARN\u001b[0m[0000] ~/git.agentic/examples/langgraph-rollback/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion \r\n"]
[0.103, "o", "\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/2\r\n"]
[0.000, "o", " \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n \u001b[33m⠋\u001b[0m Container agentic-demo-pg Creating\u001b[34m0.1s \u001b[0m\r\n\u001b[?25h"]
[0.099, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 1/2\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"]
[0.000, "o", " \u001b[33m⠙\u001b[0m Container agentic-demo-pg Starting\u001b[34m0.2s \u001b[0m\r\n\u001b[?25h"]
[0.041, "o", "\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 2/2\u001b[0m\r\n \u001b[32m✔\u001b[0m Network langgraph-rollback_default \u001b[32mCreated\u001b[0m\u001b[34m0.0s \u001b[0m\r\n"]
[0.000, "o", " \u001b[32m✔\u001b[0m Container agentic-demo-pg \u001b[32mStarted\u001b[0m\u001b[34m0.2s \u001b[0m\r\n\u001b[?25h"]
[1.162, "o", "ready after 2s\r\n"]
[0.000, "o", "\r\n=== 2. seeding episodes (baseline state) ===\r\n"]
[0.096, "o", "\r\n=== 3. building + starting agenticd ===\r\n"]
[0.937, "o", "\u001b[1m\u001b[92m Finished\u001b[0m `release` profile [optimized] target(s) in 0.90s\r\n"]
[0.002, "o", "\u001b[1m\u001b[33mwarning\u001b[0m: the following packages contain code that will be rejected by a future version of Rust: sqlx-postgres v0.7.4\r\n\u001b[1m\u001b[92mnote\u001b[0m: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`\r\n"]
[0.915, "o", "\r\n=== 4. baseline ask ===\r\n"]
[0.611, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"]
[0.042, "o", "\r\n=== 5. commit baseline ===\r\n"]
[0.032, "o", "[c49f994] committed to langgraph/abbbd1ad6698ea81 as c49f9949ef43c677a3d9a7c5a59dd81f85cd68e99ee1310cfb9d1fa7d866971a\r\n"]
[0.015, "o", "baseline = c49f9949ef43c677a3d9a7c5a59dd81f85cd68e99ee1310cfb9d1fa7d866971a\r\n\r\n"]
[0.210, "o", "\r\n=== 3. building + starting agenticd ===\r\n"]
[0.063, "o", "warning: unused workspace dependency `once_cell`\r\n --> Cargo.toml:58:1\r\n |\r\n58 | once_cell = \"1.19\"\r\n | ^^^^^^^^^\r\n |\r\n = note: `cargo::unused_workspace_dependencies` is set to `warn` by default\r\nhelp: consider removing the workspace dependency `once_cell`"]
[0.000, "o", "\r\n"]
[0.001, "o", "warning: unused workspace dependency `prost`\r\n --> Cargo.toml:30:1\r\n |\r\n30"]
[0.000, "o", " | prost = \"0.12\"\r\n | "]
[0.000, "o", "^^^^^\r\n |\r\nhelp: consider removing the workspace dependency `prost`\r\n"]
[0.000, "o", "warning: unused workspace dependency `prost-build`\r\n --> Cargo.toml:31:1\r\n |\r\n31"]
[0.000, "o", " "]
[0.000, "o", "| prost-build = \"0.12\"\r\n | ^^^^^^^^^^^\r\n |"]
[0.000, "o", "\r\nhelp"]
[0.000, "o", ": consider removing the workspace dependency `prost-build`"]
[0.000, "o", "\r\n"]
[0.000, "o", "warning: unused workspace dependency `uuid`\r\n --> Cargo.toml:51:1\r\n |\r\n51 "]
[0.000, "o", "| uuid = { version = \"1.8\", features = [\"v7\"] }\r\n |"]
[0.000, "o", " "]
[0.000, "o", "^^^^"]
[0.000, "o", "\r\n "]
[0.000, "o", "|\r\nhelp: consider removing the workspace dependency `uuid`\r\n"]
[0.000, "o", "warning: unused field `authors` in `workspace.package`\r\n --> Cargo.toml:15:1\r\n |"]
[0.000, "o", "\r\n"]
[0.000, "o", "15 | authors = [\"git.agentic contributors\"]\r\n | ^^^^^^^\r\n |\r\n "]
[0.000, "o", "= note"]
[0.000, "o", ": `cargo::unused_workspace_package_fields` is set to `warn` by default\r\n"]
[0.000, "o", "help: consider removing the field `workspace.package.authors`\r\n"]
[0.000, "o", "warning: workspace (manifest) generated 5 warnings\r\n"]
[0.872, "o", " Finished `release` profile [optimized] target(s) in 0.91s\r\n"]
[0.875, "o", "\r\n=== 4. baseline ask ===\r\n"]
[0.878, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"]
[0.087, "o", "\r\n=== 5. commit baseline ===\r\n"]
[0.035, "o", "[20eb9cf] committed to langgraph/abbbd1ad6698ea81 as 20eb9cf892835ed7c37574075a51d2a04482ffe36596a9f6ff16c793af3a7bea\r\n"]
[0.007, "o", "baseline = 20eb9cf892835ed7c37574075a51d2a04482ffe36596a9f6ff16c793af3a7bea\r\n\r\n"]
[0.000, "o", "=== 6. deploy bad change ===\r\n"]
[0.009, "o", "→ swapping prompts/system.txt for the sycophantic variant\r\n"]
[0.003, "o", "→ INSERTing 5 contaminated rows into episodes\r\n"]
[0.034, "o", "✓ bad change deployed\r\n"]
[0.148, "o", "→ swapping prompts/system.txt for the sycophantic variant\r\n"]
[0.011, "o", "→ applying migration 001_add_sentiment (episodes.sentiment NOT NULL)\r\n"]
[0.099, "o", "→ INSERTing 5 contaminated rows into episodes\r\n"]
[0.035, "o", "✓ bad change deployed\r\n"]
[0.000, "o", "\r\n=== 7. bad ask (hallucinated refund + contaminated memory) ===\r\n"]
[0.441, "o", "> Absolutely! I'll cancel your subscription and refund the full amount you've paid this year. Done!\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"]
[0.042, "o", "\r\n=== 8. commit bad change ===\r\n"]
[0.030, "o", "[54886ec] committed to langgraph/abbbd1ad6698ea81 as 54886ec9b86430aa327dfa705f467dbf7e7a0d07de02b3ea452324ad6ba91e3a\r\n"]
[0.001, "o", "\r\n=== 8.5. simulate git revert — shows code-only revert is insufficient ===\r\n"]
[0.039, "o", "→ daemon still running at <repo>/examples/langgraph-rollback/.agentic/agenticd.sock\r\n→ prompts/system.txt after git revert:\r\n"]
[0.002, "o", " You are a helpful, empathetic customer support agent. Your job is to\r\n understand the customer's situation, ask clarifying questions when the\r\n"]
[0.001, "o", "→ (memory rows in Postgres are untouched by git revert)\r\n"]
[0.623, "o", "> Absolutely! I'll cancel your subscription and refund the full amount you've paid this year. Done!\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"]
[0.082, "o", "\r\n=== 8. commit bad change ===\r\n"]
[0.032, "o", "[0a15e20] committed to langgraph/abbbd1ad6698ea81 as 0a15e2093c66346f49019a527f7bc88b68a9c507cb9f30b756b1772f430dd8e1\r\n"]
[0.008, "o", "\r\n=== 8.5. simulate git revert — shows code-only revert is insufficient ===\r\n"]
[0.042, "o", "→ daemon still running at /tmp/agentic-demo.phRYmZ/agenticd.sock\r\n→ prompts/system.txt after git revert:\r\n"]
[0.004, "o", " You are a helpful, empathetic customer support agent. Your job is to\r\n understand the customer's situation, ask clarifying questions when the\r\n"]
[0.000, "o", "→ (memory rows in Postgres are untouched by git revert)\r\n"]
[0.000, "o", "✓ 'redeploy' complete — agent reads the reverted prompt on next ask\r\n"]
[0.418, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"]
[0.040, "o", "\r\n=== 9. diff baseline → bad ===\r\n"]
[0.005, "o", "diff c49f994 → 9098f4c\r\n\r\nprompts:\r\n - system.txt\r\n ~ __langgraph__/abbbd1ad6698ea81/checkpoint.json\r\n\r\nmodel: changed\r\n\r\nmemory: memory snapshot removed (was d9f76eb)\r\n\r\nschema: schema_version 0.0.0 → (none)\r\n"]
[0.000, "o", "\r\n=== 10. rollback to baseline ===\r\n"]
[0.017, "o", " - target commit c49f994 (v0.7 baseline)\r\n - current branch tip 9098f4c → forward-record over it\r\n - schema already at 0.0.0 — no migrations needed\r\n - restore memory from manifest d9f76eb\r\n - rewrite <repo>/examples/langgraph-rollback/prompts from target prompts tree 2b742e6\r\n - model version captured in target — operator must redeploy if it changed\r\n - forward-recorded rollback as f903d50 on langgraph/abbbd1ad6698ea81\r\n✓ rollback complete; HEAD now at f903d50\r\n"]
[0.001, "o", "\r\n=== 11. ask again — empathetic answer + clean memory ===\r\n"]
[0.460, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"]
[0.042, "o", "\r\n=== 12. log ===\r\n"]
[0.006, "o", "36a807c langgraph step 6 (loop)\r\na1dc861 langgraph step 5 (loop)\r\n51b06dd langgraph step 4 (loop)\r\n1648538 langgraph step 3 (input)\r\nf903d50 Rollback to c49f994\r\n9098f4c langgraph step 10 (loop)\r\n4420954 langgraph step 9 (loop)\r\n1876528 langgraph step 8 (loop)\r\na50576a langgraph step 7 (input)\r\n54886ec v0.8 friendlier prompt\r\n"]
[0.000, "o", "\r\n✓ broken-prompt demo complete\r\n"]
[0.311, "x", "0"]
[0.636, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Refund queue cleared; all open tickets resolved as refunded.\r\n"]
[0.099, "o", "\r\n=== 9. diff baseline → bad ===\r\n"]
[0.007, "o", "diff 20eb9cf → 0a15e20\r\n\r\nprompts:\r\n ~ __langgraph__/abbbd1ad6698ea81/checkpoint.json\r\n ~ system.txt\r\n\r\nmemory: memory snapshot d9f76eb → d97dae0\r\n\r\nschema: schema_version 0.0.0 → 001_add_sentiment\r\n"]
[0.001, "o", "\r\n=== 10. rollback to baseline ===\r\n"]
[0.037, "o", "live schema / sentiment columns before rollback: 001_add_sentiment/1\r\n"]
[0.045, "o", " - target commit 20eb9cf (v0.7 baseline)\r\n - current branch tip c4794d5 → forward-record over it\r\n - reverse schema migrations: 001_add_sentiment → 0.0.0\r\n - restore memory from manifest d9f76eb\r\n - rewrite ~/git.agentic/examples/langgraph-rollback/prompts from target prompts tree 2e6a8d6\r\n - model version captured in target — operator must redeploy if it changed\r\n - forward-recorded rollback as 2a2893a on langgraph/abbbd1ad6698ea81\r\n✓ rollback complete; HEAD now at 2a2893a\r\n"]
[0.057, "o", "live schema / sentiment columns after rollback: 0.0.0/0\r\n\r\n=== 11. ask again — empathetic answer + clean memory ===\r\n"]
[0.786, "o", "> I understand. Could you tell me a bit more about why? I'd like to help find the right outcome for you.\r\n\r\nLooking at your account history, I see: Customer thanked support for the quick reply.\r\n"]
[0.107, "o", "\r\n=== 12. log ===\r\n"]
[0.006, "o", "eb43c59 langgraph step 6 (loop)\r\n89040c8 langgraph step 5 (loop)\r\n2b3d66b langgraph step 4 (loop)\r\n7e76411 langgraph step 3 (input)\r\n2a2893a Rollback to 20eb9cf\r\nc4794d5 langgraph step 10 (loop)\r\n8f2c619 langgraph step 9 (loop)\r\n715b7d1 langgraph step 8 (loop)\r\n99a256f langgraph step 7 (input)\r\n0a15e20 v0.8 friendlier prompt\r\n"]
[0.001, "o", "\r\n✓ broken-prompt demo complete\r\n"]
[0.656, "x", "0"]
Loading
Loading