What happens when an AI agent repeats a payment call that went through but came back as a timeout or an error: six payment toolkits published by the providers, run unmodified against a local fake that follows the providers' documented rules, and six models driving them through that moment. Money moved twice in 70 of 600 model trials; one model accounts for 50 of them, two others for none. No toolkit sends an idempotency key by itself.
Read the write-up first: WRITEUP.md. The design, with the expectations written before any run and every later amendment, is in DESIGN.md; the full tables are in results-l1.md and results-l2.md.
One toolkit's material is held back until its maintainers have had time to respond.
| Path | What |
|---|---|
fake/ |
Fake providers (PayPal, Square, Adyen, Razorpay, Paddle, Coinbase CDP and a Base node): only the documented rules that decide whether a repeated request is replayed, rejected or turned into a second movement, each with its source in notes/provider-semantics.md |
layer1.py, cases.py, scenarios.py, mcp_stdio.py |
Layer 1: each toolkit as published, over MCP stdio, against the fake |
adapters.py, agents.py, layer2.py |
Layer 2: six models drive adapters that replay the real toolkits' recorded answers byte for byte (test_adapters.py checks them against the recordings in fixtures/probe/) |
runs-l1/, runs-l2/ |
Every run record behind the tables; runs-l2/final-labels.jsonl holds the labels of what each agent told the merchant (rubric: labelling-rubric.md) |
analyze_l2.py, report_l1.py |
Regenerate the tables from the records |
Layer 1 needs root in a throwaway container or VM: it points the providers' hostnames at the fake through /etc/hosts
and listens on 443 (./setup.sh installs the toolkits at the tested versions; run the container with no network).
./setup.sh
python3 layer1.py --list
python3 layer1.py --only adyen
python3 report_l1.py > results-l1.md
Layer 2 needs only Python 3.9+ and API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, or
~/.config/double-take/keys.env); nothing but the model calls leaves the machine.
python3 test_adapters.py && python3 test_layer2.py
python3 layer2.py --pilot
python3 layer2.py --trials 10
python3 analyze_l2.py > results-l2.md
The fakes say what each provider documents, not what it does: read the limits in the write-up before reusing a number.
MIT. Questions and corrections: issues here. To have your own toolkit or agent run through the harness: hello@terminalstate.dev.