diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 9b84d18..381c8f9 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -20,5 +20,5 @@ A clear and concise description of what you expected to happen. **Environment:** - OS: [e.g. Ubuntu 24.04, Windows 11, macOS Sequoia] - - Go Version: [e.g. 1.22.4] - - AgentFixture Version: [e.g. 1.0.0] + - Go Version: [e.g. 1.23] + - AgentFixture Version: [e.g. 1.1.0] diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5150015..b604a4f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,7 +2,7 @@ name: CI on: push: - branches: [main] + branches: [main, 'feat/**'] pull_request: branches: [main] @@ -27,8 +27,11 @@ jobs: - name: Run Go Vet run: go vet ./... - - name: Run Tests + - name: Run Unit & Determinism Tests run: go test -v -count=1 ./... + - name: Run SSE & Matcher Benchmarks Smoke Check + run: go test -bench=Benchmark -benchmem -benchtime=100ms -run=NONE ./pkg/cassette ./pkg/matcher + - name: Compile CLI Binary - run: go build -v ./cmd/agentfixture + run: go build -v -ldflags="-s -w -X main.Version=v1.1.0" ./cmd/agentfixture diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 40b7e91..9df25d8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -23,14 +23,23 @@ jobs: go-version: '1.23' cache: true + - name: Run Verification Suite + run: | + go vet ./... + go test -count=1 ./... + - name: Build Multi-Platform Binaries run: | mkdir -p dist - VERSION=${GITHUB_REF_NAME:-v1.0.0} + VERSION=${GITHUB_REF_NAME:-v1.1.0} # Windows amd64 - CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o dist/agentfixture-windows-amd64.exe ./cmd/agentfixture - cd dist && zip agentfixture-windows-amd64.zip agentfixture-windows-amd64.exe && cd .. + CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o dist/agentfixture.exe ./cmd/agentfixture + cd dist && zip agentfixture-windows-amd64.zip agentfixture.exe && rm agentfixture.exe && cd .. + + # Windows arm64 + CGO_ENABLED=0 GOOS=windows GOARCH=arm64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o dist/agentfixture.exe ./cmd/agentfixture + cd dist && zip agentfixture-windows-arm64.zip agentfixture.exe && rm agentfixture.exe && cd .. # Linux amd64 CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o dist/agentfixture ./cmd/agentfixture @@ -58,10 +67,12 @@ jobs: cd .. - name: Publish GitHub Release + if: startsWith(github.ref, 'refs/tags/v') uses: softprops/action-gh-release@v2 with: files: | dist/agentfixture-windows-amd64.zip + dist/agentfixture-windows-arm64.zip dist/agentfixture-linux-amd64.tar.gz dist/agentfixture-linux-arm64.tar.gz dist/agentfixture-darwin-amd64.tar.gz diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a4e4c5..569755d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,36 @@ All notable changes to **AgentFixture** will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.1.0] - 2026-11-10 + +### Added +- **Anthropic Messages API & OpenAI SSE Streaming Engine (`pkg/cassette/sse.go`)**: Implemented RFC #1 with full recording, wire encoding/decoding (`bufio.Reader` with zero 64KB line limit), live proxy chunk flushing, configurable replay pacing (`instant`, `scaled`, `exact`), and `ReconstructStreamSummary` supporting `message_start`, `content_block_start`, `content_block_delta` (`text_delta`, `thinking_delta`, `signature_delta`, `input_json_delta`, `citations_delta`), `content_block_stop`, `message_delta`, `message_stop`, `ping`, `error`, and OpenAI `[DONE]` streams. +- **Zero-Outbound-Network Replay Enforcement (`pkg/proxy/offline.go`)**: Added `proxy.NewOfflineHTTPClient(proxyURL)` blocking non-loopback TCP dials (`127.0.0.1`, `::1`, `localhost`), rejected `HTTP CONNECT` tunnels with `403 Forbidden`, and injected `NO_PROXY` plus fallback offline API keys in `runner.Run`. +- **Go Testing SDK (`pkg/agenttest` & `pkg/proxy/testing.go`)**: Added `agenttest.NewTestServer(tb, cassettePath, opts...)` with `URL()`, zero-network `HTTPClient()`, `WithRecordMode`, `WithStrictExhaustive`, and automatic `tb.Cleanup` verification. +- **MCP Stdio Upgrades & Recorder (`pkg/proxy/stdio_proxy.go`)**: Added 16MB frame support, dual NDJSON and `Content-Length` framing, JSON-RPC 2.0 notification handling, automatic `ping`/`initialize`/`tools/list` synthesis when unrecorded, and `StdioRecorder` (`agentfixture record -mcp`). +- **CLI Subcommands & Flags (`cmd/agentfixture`)**: Added `agentfixture inspect [-json]`, `agentfixture redact [-out]`, `agentfixture diff -exit-code` with semantic SSE stream diffing, `-timing` / `-delay-scale` / `-max-delay` flags on `replay`, `-mcp` mode on `record`/`replay`, and `windows/arm64` release artifacts. + +### Fixed +- Fixed `agentfixture record -- ` spawning an isolated replay proxy instead of routing subprocess calls through the active `ModeRecord` server. +- Fixed upstream gzip response corruption by stripping `Accept-Encoding` and `Content-Length` on outbound proxy requests and stripping `Content-Encoding` / `Content-Length` from stored headers. +- Replaced fixed 30s `http.Client.Timeout` on upstream recording requests with `Transport{ResponseHeaderTimeout: 30s}` so multi-minute LLM streams do not abort mid-response. +- Fixed upstream URL base path joining so prefixes (e.g. `/api/v1`) are preserved rather than overwritten. +- Released `Server` mutex before simulated latency sleeps, HTTP response writes, and SSE chunk streaming; cloned cassettes during `ToYAML`/`ToJSON` to prevent concurrent mutation races. +- Fixed multi-turn stateful HTTP and MCP stdio replay to prefer unconsumed cassette interactions in sequence order before falling back to idempotent retries. +- Fixed 64-bit integer precision truncation in `StrictMatcher` using `json.Decoder.UseNumber()`, canonicalized JSON `RawBody` payloads, and normalized full URLs vs relative paths with sorted query parameters. +- Fixed `FuzzyMatcher` UTF-8 rune length calculation, recursive slice filtering in `filterDynamicKeys`, structured JSON HTTP matching, and preserved bare `"id"` arguments. +- Fixed `ComputeMetrics()` to sum `InputTokens + OutputTokens` (+ prompt cache tokens) when `TotalTokens == 0` (Anthropic usage format) with accurate input/output pricing. +- Preserved `EventError` frames in `ImportTracePortSession` instead of dropping them. +- Fixed `runner.Run` timeout cancellation via `exec.CommandContext`, Windows case-insensitive environment variable deduplication, and populated `UnmatchedExpected` and `DriftDetails`. +- Fixed `-ldflags` version injection (`var Version`) and stripped duplicate leading `v` prefixes. + +### Performance +- Reduced 100KB prompt `FuzzyMatcher` latency from ~15s (`O(N*M)` rune Levenshtein) to `<1ms` via zero-copy prefix/suffix stripping, a 4,096-rune exact DP cap, and zero-allocation 4-gram Sørensen-Dice hashing. + +### Security +- Expanded redaction and secret detection across Google (`AIza*`), GitHub (`ghp_*`, `gho_*`, `ghu_*`, `ghs_*`, `ghr_*`, `github_pat_*`), AWS (`AKIA*`, `ASIA*`), Slack (`xox*-*`), HuggingFace (`hf_*`), PEM private keys, URL `user:pass@` credentials, `Cookie`/`Set-Cookie`, `X-Api-Key`, `x-goog-api-key`, endpoints, tags, interaction metadata, and SSE event frames. +- Redacted secrets prior to `StrictHash` computation to eliminate secret hash oracles, and masked matched credential previews in `CheckForUnredactedSecrets`. + ## [1.0.0] - 2026-11-05 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f109467..5d41ad0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,11 +1,11 @@ # Contributing to AgentFixture -Thank you for your interest in contributing to **AgentFixture**! We welcome bug reports, feature suggestions, and pull requests. +Thank you for your interest in contributing to **AgentFixture**. Bug reports, RFC proposals, and pull requests are welcome. ## Development Workflow ### Prerequisites -- **Go 1.22+** +- **Go 1.23+** - Git ### Setup @@ -20,14 +20,16 @@ Thank you for your interest in contributing to **AgentFixture**! We welcome bug git checkout -b feat/your-feature ``` -### Running Tests & Verification +### Running Tests, Benchmarks & Verification ```bash -go test -v ./... go vet ./... +go test -v -count=1 ./... +go test -bench=. -benchmem ./pkg/cassette ./pkg/matcher gofmt -s -w . ``` ### Pull Requests -- Keep PRs focused. -- Ensure all CI tests pass. +- Keep pull requests focused and atomic. +- Maintain zero external runtime dependencies beyond `gopkg.in/yaml.v3`. +- Ensure all CI checks pass across Linux, Windows, and macOS. - Fill out the PR template. diff --git a/README.md b/README.md index 3d761c4..81a95e4 100644 --- a/README.md +++ b/README.md @@ -9,80 +9,79 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![Go Report Card](https://goreportcard.com/badge/github.com/aloneangel45/AgentFixture)](https://goreportcard.com/report/github.com/aloneangel45/AgentFixture) -> **Zero-dollar deterministic CI/CD regression testing cassette runner for AI agents.** - -`AgentFixture` eliminates the exorbitant API costs and flaky non-determinism of testing autonomous AI coding agents (Claude Code, Cursor, Windsurf, Aider, OpenHands) in Continuous Integration. By converting [TracePort](https://github.com/aloneangel45/traceport) black-box flight recordings into deterministic testing cassettes, AgentFixture allows entire agent test suites to run **instantly in CI with $0 in LLM API bills**. - -``` - +-----------------------------+ - | LLM Agent / Claude 3.7 | - +--------------+--------------+ - | - [Tool Requests / Calls] - v - +-----------------------------+ - | ToolPrune | <-- -91.4% Context Bloat Reduction - +--------------+--------------+ - | - [Filtered Tool Calls] - v - +-----------------------------+ - | TracePort | <-- Black-Box Flight Recording & Git Tree Snapshots - +--------------+--------------+ - | - [Flight Session JSON Export] - v - +-----------------------------+ - | AgentFixture | <-- Zero-Dollar Deterministic CI/CD Replay & Regression - +-----------------------------+ +> **Zero-dollar, zero-network deterministic CI/CD regression testing cassette engine for AI agents.** + +`AgentFixture` eliminates live LLM API bills, rate-limit flakiness, and non-deterministic model drift when testing autonomous AI coding agents (Claude Code, Cursor, Aider, OpenHands, custom MCP/HTTP agents) in Continuous Integration. Record live **Anthropic Messages API SSE streams**, **OpenAI streaming completions**, **MCP stdio tool servers**, or ingest [TracePort](https://github.com/aloneangel45/traceport) flight recordings once, then replay entire multi-turn agent suites offline in milliseconds with **$0.00 in token costs**. + +```text + +-----------------------------------+ + | Autonomous Agent / Claude 4.x | + +-----------------+-----------------+ + | + [HTTP / SSE / MCP Tool Calls] + v + +-----------------------------------+ + | ToolPrune | <-- Context Bloat Reduction + +-----------------+-----------------+ + | + [Filtered Agent Traffic] + v + +-----------------------------------+ + | TracePort | <-- Flight Recording & Git Snapshots + +-----------------+-----------------+ + | + [Live Proxy Tap or JSON Import] + v + +-----------------------------------+ + | AgentFixture | <-- $0 Offline Deterministic CI Replay + +-----------------------------------+ ``` --- ## Why AgentFixture? -1. **Eliminate the "Broken Agent Tax"**: Running a 25-test agent suite against Claude 3.7 Sonnet or GPT-4o costs **$5 to $50 per CI run**. For an active team with 15 PR runs daily, that's **$2,250+/month** in burned tokens. With AgentFixture, CI costs are **$0.00**. -2. **Sub-Second Test Runtimes**: Agent test runs that usually take 5–15 minutes waiting on LLM generation run in **under 500 milliseconds** during cassette replay. -3. **Flakiness-Proof Determinism**: Temperature drift, LLM provider rate limits (`429 Too Many Requests`), and transient network dropouts no longer cause false-positive build failures. -4. **TracePort Native Synergy**: TracePort records actual developer terminal agent sessions; AgentFixture ingests those flight recordings into permanent regression fixtures with one command. -5. **Zero Secret Leaks**: Built-in automated secret redactor scrubs Anthropic keys (`sk-ant-*`), OpenAI keys (`sk-*`), GitHub tokens (`ghp_*`, `github_pat_*`), Bearer tokens, and sensitive headers before cassettes ever touch git. +1. **Zero LLM Spend in CI**: Running a 25-test coding agent suite against Claude Sonnet or GPT-4o costs $5 to $50 per CI run ($2,250+/month across active pull requests). Replaying cassettes costs **$0.00**. +2. **Full Anthropic & OpenAI SSE Streaming Fidelity**: Records and replays raw Server-Sent Event streams (`message_start`, `thinking_delta`, `signature_delta`, `text_delta`, `input_json_delta`, `citations_delta`, `message_delta`, `message_stop`, and OpenAI `data: [DONE]`) with `instant` (`0ms`), `scaled`, or `exact` chunk timing. +3. **Strict Zero-Outbound-Network Enforcement**: Rejects `HTTP CONNECT` tunnels (`403 Forbidden`), locks `proxy.NewOfflineHTTPClient` strictly to loopback (`127.0.0.1`, `::1`, `localhost`), and injects offline SDK keys so tests cannot leak external traffic. +4. **Stateful Multi-Turn & MCP Framing**: Replays identical sequential requests in turn order before idempotent fallback, and supports both NDJSON and `Content-Length` MCP stdio framing up to 16 MB. +5. **Pre-Hash Secret Redaction**: Scrubs Anthropic, OpenAI, Google, GitHub, AWS, Slack, HuggingFace, Bearer, Cookie, URL basic-auth, and PEM credentials *before* computing SHA-256 `StrictHash` digests—eliminating secret hash oracles while allowing dummy CI keys to match seamlessly. --- ## Architecture -AgentFixture operates as an intelligent VCR proxy and testing harness: - -``` -┌─────────────────────────────────────────────────────────────────────────────┐ -│ AgentFixture Architecture │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────┐ │ -│ │ CLI & Go SDK Entrypoint │ │ -│ │ (agentfixture record | replay | verify | import | diff) │ │ -│ └──────────────────────────────────┬──────────────────────────────────┘ │ -│ │ │ -│ ┌────────────────────┴────────────────────┐ │ -│ ▼ ▼ │ -│ ┌───────────────────────────┐ ┌───────────────────────────┐ │ -│ │ Recording Proxy Engine │ │ Deterministic Replay Eng. │ │ -│ │ • HTTP Proxy (OpenAI/ │ │ • Strict SHA-256 Matcher │ │ -│ │ Anthropic API tap) │ │ • Fuzzy Semantic Matcher │ │ -│ │ • Stdio / MCP Stream │ │ • Dynamic Key Exclusion │ │ -│ │ • Secret Redactor (sk-*)│ │ • Actionable Diff Engine │ │ -│ └─────────────┬─────────────┘ └─────────────┬─────────────┘ │ -│ │ ▲ │ -│ │ ┌───────────────────┐ │ │ -│ └─────────►│ Cassette Store │──────────┘ │ -│ │ (YAML / JSON) │ │ -│ └─────────▲─────────┘ │ -│ │ │ -│ ┌─────────┴─────────┐ │ -│ │TracePort Importer │ │ -│ │(Flight Log -> Cas)│ │ -│ └───────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────┘ +```text ++-------------------------------------------------------------------------------+ +| AgentFixture v1.1 Engine | ++-------------------------------------------------------------------------------+ +| | +| +-------------------------------------------------------------------------+ | +| | CLI & Go Testing SDK (agenttest) | | +| | (record | replay | verify | inspect | redact | import | diff) | | +| +------------------------------------+------------------------------------+ | +| | | +| +--------------------+--------------------+ | +| | | | +| v v | +| +-------------------------------+ +-------------------------------+ | +| | Recording Tap Engine | | Offline Replay & Guard Eng. | | +| | - HTTP & Live SSE Streamer | | - Loopback-Only Dial Guard | | +| | - Gzip Auto-Decompression | | - Stateful Multi-Turn Queue | | +| | - MCP Stdio Tap (16MB) | | - Instant/Scaled/Exact SSE | | +| | - Pre-Hash Secret Redactor | | - Strict 64-bit & Fuzzy O(N) | | +| +---------------+---------------+ +---------------+---------------+ | +| | ^ | +| | +-------------------+ | | +| +--------->| Cassette Store |----------+ | +| | (YAML / JSON) | | +| +---------+---------+ | +| ^ | +| +---------+---------+ | +| | TracePort Importer| | +| | (Flight -> Cas) | | +| +-------------------+ | ++-------------------------------------------------------------------------------+ ``` --- @@ -91,10 +90,10 @@ AgentFixture operates as an intelligent VCR proxy and testing harness: ### Pre-Compiled Binaries -Download pre-compiled binaries for Windows, Linux, and macOS from the [Releases](https://github.com/aloneangel45/AgentFixture/releases) page. +Download pre-compiled binaries for Linux (`amd64`/`arm64`), macOS (`amd64`/`arm64`), and Windows (`amd64`/`arm64`) from [Releases](https://github.com/aloneangel45/AgentFixture/releases). ```bash -# macOS / Linux (via curl) +# macOS / Linux curl -fsSL https://github.com/aloneangel45/AgentFixture/releases/latest/download/agentfixture-linux-amd64.tar.gz | tar -xz sudo mv agentfixture /usr/local/bin/ @@ -111,118 +110,199 @@ go install github.com/aloneangel45/AgentFixture/cmd/agentfixture@latest --- -## Quick Start & CLI Usage +## CLI Quick Start + +### 1. Record Live Anthropic / OpenAI SSE Streaming Sessions -### 1. Ingest a TracePort Flight Recording -Convert any session recorded with `traceport record` into an AgentFixture cassette: ```bash -agentfixture import -traceport ~/.traceport/sessions/trace_1793907746.json -out fixtures/auth_flow.yaml +agentfixture record -cassette fixtures/claude_stream.yaml -upstream https://api.anthropic.com -- python run_agent.py ``` -### 2. Verify Cassette Integrity & Redaction -Ensure the cassette format conforms to `agentfixture/v1` and contains no leaked credentials: +Record an MCP stdio tool server directly: + ```bash -agentfixture verify fixtures/auth_flow.yaml +agentfixture record -mcp -cassette fixtures/mcp_fs.yaml -- npx -y @modelcontextprotocol/server-filesystem . ``` -Output: -```text -[✓] Cassette fixtures/auth_flow.yaml is VALID and SECURE - Version: agentfixture/v1 - ID: trace_1793907746 - Interactions: 8 - Estimated Tokens Saved: 24,150 - Estimated Cost Saved: $0.1449 USD -``` +### 2. Replay Deterministically in CI (`0ms` Instant SSE Mode) -### 3. Replay in CI/CD (Zero-Dollar Test Runner) -Run your agent's test suite inside GitHub Actions without configuring LLM API keys: ```bash -agentfixture replay -cassette fixtures/auth_flow.yaml -- go test -v ./... +agentfixture replay -cassette fixtures/claude_stream.yaml -timing instant -- go test -v ./... ``` Output: ```text ================================================================= - AgentFixture Regression Report: trace_1793907746 + AgentFixture Regression Report: cas_record_1794339000 ================================================================= - Status: PASSED - Exit Code: 0 - Interactions Replayed: 8 / 8 (100.0%) - Estimated Tokens Saved: 24,150 + Status: PASSED + Exit Code: 0 + Interactions Replayed: 8 / 8 (100.0%) + Estimated Tokens Saved: 24150 Estimated Cost Saved: $0.1449 USD - Replay Run Duration: 182ms + Replay Run Duration: 42ms ================================================================= ``` -### 4. Record a Live Agent Session -Tap incoming agent API calls to Anthropic or OpenAI and store them in a new cassette: +Simulate realistic token arrival pacing (`scaled` or `exact`): + ```bash -agentfixture record -cassette fixtures/new_feature.yaml -upstream https://api.anthropic.com -- python run_agent.py +agentfixture replay -cassette fixtures/claude_stream.yaml -timing scaled -delay-scale 0.25 -max-delay 250ms -- python ui_test.py ``` -### 5. Diff Two Cassettes (Drift Analysis) -Compare prompt alterations or tool input changes between cassette versions: +### 3. Verify Schema & Secret Redaction + ```bash -agentfixture diff fixtures/auth_v1.yaml fixtures/auth_v2.yaml +agentfixture verify fixtures/claude_stream.yaml +``` + +Output: +```text +[OK] Cassette fixtures/claude_stream.yaml is VALID and SECURE + Version: agentfixture/v1 + ID: cas_record_1794339000 + Interactions: 8 (SSE streams: 5) + Estimated Tokens Saved: 24150 + Estimated Cost Saved: $0.1449 USD +``` + +### 4. Inspect Reconstructed SSE Streams & Telemetry + +```bash +agentfixture inspect fixtures/claude_stream.yaml +agentfixture inspect -json fixtures/claude_stream.yaml +``` + +### 5. Scrub Secrets & Semantic Diff Cassettes in CI + +```bash +# Scrub credentials in-place and recompute secret-safe StrictHash digests +agentfixture redact fixtures/raw_capture.yaml + +# Fail CI with non-zero exit code if prompts, tool inputs, or reconstructed SSE outputs drifted +agentfixture diff -exit-code fixtures/v1.yaml fixtures/v2.yaml +``` + +### 6. Ingest TracePort Flight Recordings + +```bash +agentfixture import -traceport ~/.traceport/sessions/trace_1793907746.json -out fixtures/auth_flow.yaml +``` + +--- + +## Go Testing SDK (`pkg/agenttest`) + +Embed deterministic VCR replay directly inside standard `go test` suites with zero boilerplate and guaranteed offline network isolation: + +```go +package agent_test + +import ( + "io" + "strings" + "testing" + + "github.com/aloneangel45/AgentFixture/pkg/agenttest" + "github.com/aloneangel45/AgentFixture/pkg/cassette" +) + +func TestClaudeStreamingAgent(t *testing.T) { + ts := agenttest.NewTestServer( + t, + "testdata/anthropic_tool_stream.yaml", + agenttest.WithStrictExhaustive(true), + ) + + // ts.HTTPClient() blocks any TCP dial outside 127.0.0.1 / ::1 / localhost + resp, err := ts.HTTPClient().Post( + ts.URL()+"/v1/messages", + "application/json", + strings.NewReader(`{"model":"claude-sonnet-4-20250514","stream":true,"messages":[{"role":"user","content":"Search for ReplaySSEStream"}]}`), + ) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + defer resp.Body.Close() + + events, err := cassette.DecodeSSE(resp.Body) + if err != nil { + t.Fatalf("decode sse: %v", err) + } + summary := cassette.ReconstructStreamSummary(events) + if summary.StopReason != "tool_use" || len(summary.ToolCalls) != 1 { + t.Fatalf("unexpected stream summary: %+v", summary) + } +} ``` --- ## Cassette Specification (`agentfixture/v1`) -Cassettes can be stored as either **YAML** or **JSON**: +Cassettes serialize cleanly as **YAML** or **JSON**, supporting `http`, `sse`, `tool_execution`, `stdio`, and `llm_chat` interactions: ```yaml version: "agentfixture/v1" metadata: - cassette_id: "cas_auth_middleware_01" - recorded_at: "2026-11-05T20:30:00Z" - generator: "AgentFixture v1.0.0" - agent_name: "claude-code" - model: "claude-3-7-sonnet-20250219" + cassette_id: "cas_anthropic_sse_01" + recorded_at: "2026-11-10T16:40:00Z" + generator: "AgentFixture v1.1.0" + model: "claude-sonnet-4-20250514" metrics: - total_interactions: 2 - recorded_duration_ms: 1240 - estimated_tokens_saved: 14200 - estimated_cost_saved_usd: 0.0852 - tags: - git_snapshot: "4b825dc642cb6eb9a060e54bf8d69288fbee4904" + total_interactions: 1 + recorded_duration_ms: 410 + estimated_tokens_saved: 656 + estimated_cost_saved_usd: 0.0030 interactions: - - id: "tp_step_1" + - id: "int_001" sequence: 1 - type: "tool_execution" + type: "sse" request: - tool_name: "view_file" - tool_input: - path: "pkg/auth/auth.go" + method: "POST" + endpoint: "/v1/messages?beta=true" + headers: + X-Api-Key: "[REDACTED]" + raw_body: '{"model":"claude-sonnet-4-20250514","stream":true,"messages":[{"role":"user","content":"Inspect auth.go"}]}' matcher: - strict_hash: "sha256:7e8a9b1c3d..." - fuzzy_signature: - normalized_tool_name: "view_file" - normalized_args: - path: "pkg/auth/auth.go" + strict_hash: "sha256:8f4c2e..." response: - status_code: 0 - latency_ms: 32 - tool_result: "package auth\n\nfunc ValidateToken(t string) bool { return true }\n" + status_code: 200 + latency_ms: 410 + sse_events: + - index: 0 + event: "message_start" + data: '{"type":"message_start","message":{"id":"msg_01","model":"claude-sonnet-4-20250514","usage":{"input_tokens":412,"output_tokens":1}}}' + delta_ms: 12 + - index: 1 + event: "content_block_delta" + data: '{"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Inspecting auth.go..."}}' + delta_ms: 18 + - index: 2 + event: "message_stop" + data: '{"type":"message_stop"}' + delta_ms: 4 + usage: + input_tokens: 412 + output_tokens: 94 + cache_creation_input_tokens: 100 + cache_read_input_tokens: 50 ``` --- -## Matching Engines +## Matching Engines & Benchmarks -| Matcher | Behavior | Use Case | +| Matcher | Behavior | Latency | |---|---|---| -| **Strict Matcher** | Canonical SHA-256 hash comparison across sorted JSON keys, headers, and endpoints. | Production CI assertion; ensures exact reproducibility. | -| **Fuzzy Matcher** | Normalizes whitespace, ignores dynamic keys (`timestamp`, `nonce`, `request_id`), strips UUIDs and ISO timestamps, uses Levenshtein similarity. | Resilient regression tests where prompt phrasing evolves slightly. | +| **Strict Matcher** | Secret-redacted SHA-256 hash over canonicalized endpoints, sorted query params, and `UseNumber()` 64-bit exact JSON keys. | `~9.6 us/op` | +| **Fuzzy Matcher** | Strips dynamic keys (`timestamp`, `nonce`, `request_id`, `trace_id`), normalizes UUIDs/ISO dates, caps exact UTF-8 rune Levenshtein at 4,096 runes, and uses O(N) 4-gram Dice hashing for 100KB+ context windows. | `~12.2 us/op` (short) / `~0.63-0.91 ms/op` (100KB prompt) | +| **SSE Instant Replay** | Zero-delay streaming playback of recorded Anthropic SSE frames. | `~1.6 us/op` (`>1.5 GB/s`) | --- ## GitHub Actions CI Integration -Add AgentFixture to your repository workflow: - ```yaml name: Agent Regression Suite @@ -242,8 +322,10 @@ jobs: go-version: '1.23' - name: Install AgentFixture run: go install github.com/aloneangel45/AgentFixture/cmd/agentfixture@latest - - name: Run Zero-Dollar Agent Regression Test - run: agentfixture replay -cassette fixtures/agent_suite.yaml -- go test -v ./... + - name: Verify Cassettes Are Leak-Free + run: agentfixture verify fixtures/agent_suite.yaml + - name: Run Zero-Dollar Offline Agent Regression Suite + run: agentfixture replay -cassette fixtures/agent_suite.yaml -timing instant -- go test -v ./... ``` --- diff --git a/SECURITY.md b/SECURITY.md index 6b8f79a..4bd7961 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,10 +2,17 @@ ## Supported Versions -| Version | Supported | -| ------- | ------------------ | -| 1.0.x | :white_check_mark: | -| < 1.0 | :x: | +| Version | Supported | +| ------- | --------- | +| 1.1.x | Yes | +| 1.0.x | Yes | +| < 1.0 | No | + +## Security Controls in AgentFixture + +- **Pre-Hash Secret Redaction**: Credentials, Bearer tokens, PEM private keys, cookies, URL basic auth, and provider API keys (`sk-ant-*`, `sk-proj-*`, `sk-*`, `AIza*`, `ghp_*`, `github_pat_*`, `AKIA*`, `ASIA*`, `xox*-*`, `hf_*`) are scrubbed before computing `StrictHash` SHA-256 digests so cassettes never act as secret hash oracles. +- **Zero Outbound Network Replay**: Replay proxies block `HTTP CONNECT` tunnels (`403 Forbidden`) and `proxy.NewOfflineHTTPClient` rejects any TCP/UDP dial outside loopback (`127.0.0.1`, `::1`, `localhost`). +- **Masked Secret Diagnostics**: `agentfixture verify` masks matched credential suffixes so CI logs never print leaked secret bytes. ## Reporting a Vulnerability diff --git a/cmd/agentfixture/main.go b/cmd/agentfixture/main.go index 1d84611..06b22d5 100644 --- a/cmd/agentfixture/main.go +++ b/cmd/agentfixture/main.go @@ -1,10 +1,14 @@ package main import ( + "context" + "encoding/json" + "errors" "flag" "fmt" "os" "os/signal" + "reflect" "strings" "syscall" "time" @@ -16,12 +20,16 @@ import ( "github.com/aloneangel45/AgentFixture/pkg/runner" ) +// Version is injected at build time via -ldflags="-X main.Version=...". +var Version = "1.1.0" + const ( - Version = "1.0.0" ProgramName = "AgentFixture" Banner = "Zero-Dollar Deterministic CI/CD Regression Testing Cassette for AI Agents" ) +var errDiffDetected = errors.New("cassette drift detected") + func main() { if len(os.Args) < 2 { printUsage() @@ -37,6 +45,10 @@ func main() { printVersion() case "verify": err = handleVerify(args) + case "inspect": + err = handleInspect(args) + case "redact": + err = handleRedact(args) case "import": err = handleImport(args) case "diff": @@ -48,18 +60,26 @@ func main() { case "help", "-h", "--help": printUsage() default: - fmt.Fprintf(os.Stderr, "Unknown command: %s\nRun 'agentfixture help' for usage.\n", subcommand) + fmt.Fprintf(os.Stderr, "[FAIL] Unknown command: %s\nRun 'agentfixture help' for usage.\n", subcommand) os.Exit(1) } if err != nil { - fmt.Fprintf(os.Stderr, "Error: %v\n", err) + fmt.Fprintf(os.Stderr, "[FAIL] Error: %v\n", err) os.Exit(1) } } +func formatVersionString(ver string) string { + clean := strings.TrimPrefix(strings.TrimSpace(ver), "v") + if clean == "" { + clean = "1.1.0" + } + return fmt.Sprintf("%s v%s (%s)", ProgramName, clean, Banner) +} + func printVersion() { - fmt.Printf("%s v%s (%s)\n", ProgramName, Version, Banner) + fmt.Println(formatVersionString(Version)) } func printUsage() { @@ -69,19 +89,24 @@ Usage: agentfixture [options] [-- ] Commands: - record Record agent API/tool interactions into a cassette - replay Replay recorded cassette deterministically for regression tests - verify Validate cassette integrity, syntax, and check for unmasked secrets + record Record HTTP, SSE streaming, or MCP stdio interactions into a cassette + replay Replay recorded cassette deterministically with zero outbound network calls + verify Validate cassette integrity, schema, and check for unmasked secrets + inspect Display cassette telemetry, interaction summary, and reconstructed SSE streams + redact Scrub secrets from a cassette file and recompute secret-safe hashes import Ingest TracePort flight recordings and convert to testing cassette - diff Compare two cassettes and report interaction/argument drifts + diff Compare two cassettes for request, response, or semantic SSE stream drift version Show version information Examples: agentfixture verify fixtures/auth_flow.yaml + agentfixture inspect fixtures/anthropic_sse.yaml + agentfixture redact fixtures/raw_capture.yaml agentfixture import -traceport flight.json -out cassette.yaml - agentfixture replay -cassette cassette.yaml -- go test ./... + agentfixture replay -cassette cassette.yaml -timing instant -- go test ./... agentfixture record -cassette cassette.yaml -upstream https://api.anthropic.com -- python agent.py - agentfixture diff cassette_v1.yaml cassette_v2.yaml`) + agentfixture record -mcp -cassette mcp_tools.yaml -- npx -y @modelcontextprotocol/server-filesystem . + agentfixture diff -exit-code cassette_v1.yaml cassette_v2.yaml`) } func handleVerify(args []string) error { @@ -113,16 +138,190 @@ func handleVerify(args []string) error { return fmt.Errorf("security violation: %s", secretReason) } + sseCount := 0 + for _, inter := range cas.Interactions { + if inter.Type == cassette.InteractionTypeSSE || len(inter.Response.SSEEvents) > 0 { + sseCount++ + } + } + cas.ComputeMetrics() - fmt.Printf("[✓] Cassette %s is VALID and SECURE\n", casPath) - fmt.Printf(" Version: %s\n", cas.Version) - fmt.Printf(" ID: %s\n", cas.Metadata.CassetteID) - fmt.Printf(" Interactions: %d\n", cas.Metadata.Metrics.TotalInteractions) + fmt.Printf("[OK] Cassette %s is VALID and SECURE\n", casPath) + fmt.Printf(" Version: %s\n", cas.Version) + fmt.Printf(" ID: %s\n", cas.Metadata.CassetteID) + fmt.Printf(" Interactions: %d (SSE streams: %d)\n", cas.Metadata.Metrics.TotalInteractions, sseCount) fmt.Printf(" Estimated Tokens Saved: %d\n", cas.Metadata.Metrics.EstimatedTokensSaved) fmt.Printf(" Estimated Cost Saved: $%.4f USD\n", cas.Metadata.Metrics.EstimatedCostSavedUSD) return nil } +type inspectInteractionView struct { + Sequence int `json:"sequence"` + ID string `json:"id"` + Type cassette.InteractionType `json:"type"` + Method string `json:"method,omitempty"` + Endpoint string `json:"endpoint,omitempty"` + ToolName string `json:"tool_name,omitempty"` + StatusCode int `json:"status_code"` + LatencyMs int64 `json:"latency_ms,omitempty"` + Usage cassette.Usage `json:"usage,omitempty"` + StreamSummary *cassette.StreamSummary `json:"stream_summary,omitempty"` +} + +type inspectReport struct { + Path string `json:"path"` + Version string `json:"version"` + Metadata cassette.Metadata `json:"metadata"` + TypeCounts map[string]int `json:"type_counts"` + Interactions []inspectInteractionView `json:"interactions"` +} + +func handleInspect(args []string) error { + fs := flag.NewFlagSet("inspect", flag.ContinueOnError) + jsonOut := fs.Bool("json", false, "Emit structured JSON telemetry") + if err := fs.Parse(args); err != nil { + return err + } + + rest := fs.Args() + if len(rest) < 1 { + return fmt.Errorf("usage: agentfixture inspect [-json] ") + } + casPath := rest[0] + + cas, err := cassette.Load(casPath) + if err != nil { + return fmt.Errorf("failed to load cassette: %w", err) + } + cas.ComputeMetrics() + + rep := inspectReport{ + Path: casPath, + Version: cas.Version, + Metadata: cas.Metadata, + TypeCounts: make(map[string]int), + Interactions: make([]inspectInteractionView, 0, len(cas.Interactions)), + } + + for _, inter := range cas.Interactions { + rep.TypeCounts[string(inter.Type)]++ + view := inspectInteractionView{ + Sequence: inter.Sequence, + ID: inter.ID, + Type: inter.Type, + Method: inter.Request.Method, + Endpoint: inter.Request.Endpoint, + ToolName: inter.Request.ToolName, + StatusCode: inter.Response.StatusCode, + LatencyMs: inter.Response.LatencyMs, + Usage: inter.Response.Usage, + } + if inter.Type == cassette.InteractionTypeSSE || len(inter.Response.SSEEvents) > 0 { + sum := cassette.ReconstructStreamSummary(inter.Response.SSEEvents) + view.StreamSummary = &sum + if view.Usage.InputTokens == 0 && view.Usage.OutputTokens == 0 && view.Usage.TotalTokens == 0 { + view.Usage = sum.Usage + } + } + rep.Interactions = append(rep.Interactions, view) + } + + if *jsonOut { + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + return enc.Encode(rep) + } + + fmt.Println("=================================================================") + fmt.Printf(" AgentFixture Inspection: %s\n", casPath) + fmt.Println("=================================================================") + fmt.Printf(" Cassette ID: %s\n", cas.Metadata.CassetteID) + fmt.Printf(" Schema Version: %s\n", cas.Version) + if cas.Metadata.Model != "" { + fmt.Printf(" Model: %s\n", cas.Metadata.Model) + } + if cas.Metadata.Source != "" { + fmt.Printf(" Source: %s\n", cas.Metadata.Source) + } + fmt.Printf(" Recorded At: %s\n", cas.Metadata.RecordedAt.Format(time.RFC3339)) + fmt.Printf(" Total Interactions: %d\n", cas.Metadata.Metrics.TotalInteractions) + fmt.Printf(" Recorded Duration: %d ms\n", cas.Metadata.Metrics.RecordedDurationMs) + fmt.Printf(" Estimated Tokens Saved: %d\n", cas.Metadata.Metrics.EstimatedTokensSaved) + fmt.Printf(" Estimated Cost Saved: $%.4f USD\n", cas.Metadata.Metrics.EstimatedCostSavedUSD) + + if len(rep.Interactions) > 0 { + fmt.Println("-----------------------------------------------------------------") + fmt.Println(" Interactions:") + for _, v := range rep.Interactions { + target := strings.TrimSpace(v.Method + " " + v.Endpoint) + if v.ToolName != "" { + target = "tool:" + v.ToolName + } + fmt.Printf(" #%02d [%s] %-14s %s (status=%d, latency=%dms)\n", + v.Sequence, v.ID, v.Type, target, v.StatusCode, v.LatencyMs) + if v.StreamSummary != nil { + s := v.StreamSummary + var toolNames []string + for _, tc := range s.ToolCalls { + toolNames = append(toolNames, tc.Name) + } + fmt.Printf(" SSE Summary: provider=%s model=%s events=%d stop=%s in=%d out=%d thinking=%d tools=%v\n", + s.Provider, s.Model, s.EventCount, s.StopReason, + s.Usage.InputTokens, s.Usage.OutputTokens, + len(s.ThinkingBlocks), toolNames) + } + } + } + fmt.Println("=================================================================") + return nil +} + +func handleRedact(args []string) error { + fs := flag.NewFlagSet("redact", flag.ContinueOnError) + outPath := fs.String("out", "", "Destination path (defaults to in-place overwrite)") + if err := fs.Parse(args); err != nil { + return err + } + + rest := fs.Args() + if len(rest) < 1 { + return fmt.Errorf("usage: agentfixture redact [-out ] ") + } + srcPath := rest[0] + dest := *outPath + if dest == "" { + dest = srcPath + } + + cas, err := cassette.Load(srcPath) + if err != nil { + return fmt.Errorf("failed to load cassette: %w", err) + } + + cassette.RedactCassette(cas) + for i := range cas.Interactions { + inter := &cas.Interactions[i] + inter.Request.Matcher.StrictHash = matcher.ComputeRequestHash(&inter.Request) + if inter.Request.ToolName != "" { + inter.Request.Matcher.FuzzySignature.NormalizedToolName = strings.ToLower(inter.Request.ToolName) + inter.Request.Matcher.FuzzySignature.NormalizedArgs = cassette.RedactMap(inter.Request.ToolInput) + } else if inter.Request.RawBody != "" && inter.Request.Matcher.FuzzySignature.NormalizedPrompt != "" { + inter.Request.Matcher.FuzzySignature.NormalizedPrompt = matcher.NormalizeText(inter.Request.RawBody) + } + } + + if hasSecrets, reason := cassette.CheckForUnredactedSecrets(cas); hasSecrets { + return fmt.Errorf("post-redaction check failed: %s", reason) + } + + if err := cassette.Save(dest, cas); err != nil { + return fmt.Errorf("failed to write redacted cassette: %w", err) + } + + fmt.Printf("[OK] Redacted cassette saved to %s (%d interactions scrubbed)\n", dest, len(cas.Interactions)) + return nil +} + func handleImport(args []string) error { fs := flag.NewFlagSet("import", flag.ContinueOnError) tpPath := fs.String("traceport", "", "Path to TracePort flight recording JSON file") @@ -140,7 +339,7 @@ func handleImport(args []string) error { return err } - fmt.Printf("[✓] Successfully imported TracePort flight recording!\n") + fmt.Printf("[OK] Imported TracePort flight recording\n") fmt.Printf(" Source: %s\n", *tpPath) fmt.Printf(" Destination: %s\n", *outPath) fmt.Printf(" Interactions: %d\n", len(cas.Interactions)) @@ -149,20 +348,29 @@ func handleImport(args []string) error { } func handleDiff(args []string) error { - if len(args) < 2 { - return fmt.Errorf("usage: agentfixture diff ") + fs := flag.NewFlagSet("diff", flag.ContinueOnError) + exitCode := fs.Bool("exit-code", false, "Exit with non-zero status code when differences are found") + if err := fs.Parse(args); err != nil { + return err + } + + rest := fs.Args() + if len(rest) < 2 { + return fmt.Errorf("usage: agentfixture diff [-exit-code] ") } - cas1, err := cassette.Load(args[0]) + path1, path2 := rest[0], rest[1] + + cas1, err := cassette.Load(path1) if err != nil { - return fmt.Errorf("failed to load %s: %w", args[0], err) + return fmt.Errorf("failed to load %s: %w", path1, err) } - cas2, err := cassette.Load(args[1]) + cas2, err := cassette.Load(path2) if err != nil { - return fmt.Errorf("failed to load %s: %w", args[1], err) + return fmt.Errorf("failed to load %s: %w", path2, err) } fmt.Printf("=================================================================\n") - fmt.Printf(" AgentFixture Diff: %s <-> %s\n", args[0], args[1]) + fmt.Printf(" AgentFixture Diff: %s <-> %s\n", path1, path2) fmt.Printf("=================================================================\n") n1, n2 := len(cas1.Interactions), len(cas2.Interactions) @@ -174,6 +382,7 @@ func handleDiff(args []string) error { } strict := matcher.NewStrictMatcher() + fuzzy := matcher.NewFuzzyMatcher(0.80) diffFound := false for i := 0; i < minLen; i++ { @@ -183,62 +392,139 @@ func handleDiff(args []string) error { res := strict.Match(&i1.Request, &i2.Request) if !res.Matched { diffFound = true - fmt.Printf("\n[!] Interaction #%d Mismatch:\n", i+1) - fmt.Printf(" %s (%s %s)\n", i1.ID, i1.Request.Method, i1.Request.ToolName) - fmt.Printf(" %s (%s %s)\n", i2.ID, i2.Request.Method, i2.Request.ToolName) + fuzzyRes := fuzzy.Match(&i1.Request, &i2.Request) + fmt.Printf("\n[WARN] Interaction #%d Request Drift:\n", i+1) + fmt.Printf(" Left: %s (%s %s %s)\n", i1.ID, i1.Request.Method, i1.Request.Endpoint, i1.Request.ToolName) + fmt.Printf(" Right: %s (%s %s %s)\n", i2.ID, i2.Request.Method, i2.Request.Endpoint, i2.Request.ToolName) + fmt.Printf(" Semantic Similarity: %.2f\n", fuzzyRes.Score) fmt.Printf(" Diff: %s\n", res.Diff) } + + if sseDiffs := diffSSEResponses(i1, i2); len(sseDiffs) > 0 { + diffFound = true + fmt.Printf("\n[WARN] Interaction #%d SSE Stream Semantic Drift:\n", i+1) + for _, d := range sseDiffs { + fmt.Printf(" - %s\n", d) + } + } else if i1.Response.StatusCode != i2.Response.StatusCode || + (len(i1.Response.SSEEvents) == 0 && len(i2.Response.SSEEvents) == 0 && + (i1.Response.RawBody != i2.Response.RawBody || i1.Response.ToolResult != i2.Response.ToolResult)) { + diffFound = true + fmt.Printf("\n[WARN] Interaction #%d Response Drift (status %d vs %d)\n", + i+1, i1.Response.StatusCode, i2.Response.StatusCode) + } } if n1 > minLen { diffFound = true - fmt.Printf("\n[!] Extra interactions in %s:\n", args[0]) + fmt.Printf("\n[WARN] Extra interactions in %s:\n", path1) for i := minLen; i < n1; i++ { fmt.Printf(" + #%d (%s)\n", i+1, cas1.Interactions[i].ID) } } if n2 > minLen { diffFound = true - fmt.Printf("\n[!] Extra interactions in %s:\n", args[1]) + fmt.Printf("\n[WARN] Extra interactions in %s:\n", path2) for i := minLen; i < n2; i++ { fmt.Printf(" + #%d (%s)\n", i+1, cas2.Interactions[i].ID) } } if !diffFound { - fmt.Println("\n[✓] Both cassettes are structurally identical!") + fmt.Println("\n[OK] Both cassettes are structurally and semantically identical") } fmt.Printf("=================================================================\n") + + if diffFound && *exitCode { + return errDiffDetected + } return nil } +func diffSSEResponses(i1, i2 *cassette.Interaction) []string { + has1 := i1.Type == cassette.InteractionTypeSSE || len(i1.Response.SSEEvents) > 0 + has2 := i2.Type == cassette.InteractionTypeSSE || len(i2.Response.SSEEvents) > 0 + if !has1 && !has2 { + return nil + } + if has1 != has2 { + return []string{fmt.Sprintf("interaction type changed: %s vs %s", i1.Type, i2.Type)} + } + + s1 := cassette.ReconstructStreamSummary(i1.Response.SSEEvents) + s2 := cassette.ReconstructStreamSummary(i2.Response.SSEEvents) + + var diffs []string + if s1.Model != s2.Model { + diffs = append(diffs, fmt.Sprintf("model changed: %q -> %q", s1.Model, s2.Model)) + } + if s1.StopReason != s2.StopReason { + diffs = append(diffs, fmt.Sprintf("stop_reason changed: %q -> %q", s1.StopReason, s2.StopReason)) + } + if s1.Text != s2.Text { + diffs = append(diffs, fmt.Sprintf("assistant text changed (len %d -> %d)", len(s1.Text), len(s2.Text))) + } + if !reflect.DeepEqual(s1.ThinkingBlocks, s2.ThinkingBlocks) { + diffs = append(diffs, fmt.Sprintf("thinking blocks changed (%d -> %d)", len(s1.ThinkingBlocks), len(s2.ThinkingBlocks))) + } + if len(s1.ToolCalls) != len(s2.ToolCalls) { + diffs = append(diffs, fmt.Sprintf("tool_use count changed: %d -> %d", len(s1.ToolCalls), len(s2.ToolCalls))) + } else { + for idx := range s1.ToolCalls { + tc1, tc2 := s1.ToolCalls[idx], s2.ToolCalls[idx] + if tc1.Name != tc2.Name || !reflect.DeepEqual(tc1.Input, tc2.Input) { + diffs = append(diffs, fmt.Sprintf("tool_use[%d] changed: %s(%v) -> %s(%v)", + idx, tc1.Name, tc1.Input, tc2.Name, tc2.Input)) + } + } + } + return diffs +} + +func splitDashArgs(args []string) (flagArgs []string, cmdArgs []string) { + for i, arg := range args { + if arg == "--" { + return args[:i], args[i+1:] + } + } + return args, nil +} + func handleRecord(args []string) error { fs := flag.NewFlagSet("record", flag.ContinueOnError) casPath := fs.String("cassette", "fixture.yaml", "Path to write recorded cassette") - upstream := fs.String("upstream", "", "Upstream target URL to forward requests to") + upstream := fs.String("upstream", "", "Upstream target URL to forward HTTP/SSE requests to") listenAddr := fs.String("addr", "127.0.0.1:0", "Local address to bind proxy on") + mcpMode := fs.Bool("mcp", false, "Record an MCP stdio server subprocess instead of an HTTP proxy") - var cmdArgs []string - dashIdx := -1 - for i, arg := range args { - if arg == "--" { - dashIdx = i - break - } + flagArgs, cmdArgs := splitDashArgs(args) + if err := fs.Parse(flagArgs); err != nil { + return err } - if dashIdx != -1 { - if err := fs.Parse(args[:dashIdx]); err != nil { - return err + + cas := cassette.NewCassette(fmt.Sprintf("cas_record_%d", time.Now().Unix())) + + if *mcpMode { + if len(cmdArgs) == 0 { + return fmt.Errorf("-mcp record mode requires a target server command after '--'") } - cmdArgs = args[dashIdx+1:] - } else { - if err := fs.Parse(args); err != nil { - return err + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + rec := proxy.NewStdioRecorder(cas) + fmt.Fprintf(os.Stderr, "[INFO] AgentFixture MCP Stdio Recorder wrapping: %s\n", strings.Join(cmdArgs, " ")) + recCas, runErr := rec.RecordCommand(ctx, cmdArgs, os.Environ(), os.Stdin, os.Stdout) + if saveErr := cassette.Save(*casPath, recCas); saveErr != nil { + return fmt.Errorf("failed to save MCP cassette: %w", saveErr) } + fmt.Fprintf(os.Stderr, "[OK] MCP cassette saved to %s (%d interactions recorded)\n", *casPath, len(recCas.Interactions)) + return runErr } - cas := cassette.NewCassette(fmt.Sprintf("cas_record_%d", time.Now().Unix())) cas.Metadata.Source = "agentfixture_record_proxy" + if *upstream == "" { + *upstream = "https://api.anthropic.com" + } srv, err := proxy.NewServer(proxy.Config{ Mode: proxy.ModeRecord, @@ -254,23 +540,27 @@ func handleRecord(args []string) error { } defer srv.Close() - fmt.Printf("[•] AgentFixture Recording Proxy active on %s\n", srv.URL()) - if *upstream != "" { - fmt.Printf(" Forwarding to Upstream: %s\n", *upstream) - } + fmt.Printf("[INFO] AgentFixture Recording Proxy active on %s\n", srv.URL()) + fmt.Printf(" Forwarding to Upstream: %s\n", *upstream) + var cmdExecErr error if len(cmdArgs) > 0 { - fmt.Printf("[•] Spawning command: %s\n", strings.Join(cmdArgs, " ")) - rep, err := runner.Run(runner.RunOptions{ + fmt.Printf("[INFO] Spawning command: %s\n", strings.Join(cmdArgs, " ")) + rep, runErr := runner.Run(runner.RunOptions{ Cassette: cas, + Mode: proxy.ModeRecord, + Server: srv, Command: cmdArgs, }) - if err != nil { - return err + if runErr != nil { + return runErr + } + fmt.Print(rep.FormatSummary()) + if !rep.Passed { + cmdExecErr = fmt.Errorf("recorded command exited with code %d", rep.CommandExitCode) } - fmt.Println(rep.FormatSummary()) } else { - fmt.Println("[•] Running standalone proxy. Press Ctrl+C to save cassette and stop.") + fmt.Println("[INFO] Running standalone proxy. Press Ctrl+C to save cassette and stop.") sigChan := make(chan os.Signal, 1) signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM) <-sigChan @@ -281,65 +571,72 @@ func handleRecord(args []string) error { return fmt.Errorf("failed to save cassette: %w", err) } - fmt.Printf("[✓] Cassette saved to %s (%d interactions recorded)\n", *casPath, len(recCas.Interactions)) - return nil + fmt.Printf("[OK] Cassette saved to %s (%d interactions recorded)\n", *casPath, len(recCas.Interactions)) + return cmdExecErr } func handleReplay(args []string) error { fs := flag.NewFlagSet("replay", flag.ContinueOnError) casPath := fs.String("cassette", "", "Path to cassette file to replay") matcherType := fs.String("matcher", "strict", "Matcher type: strict or fuzzy") + threshold := fs.Float64("threshold", 0.80, "Similarity threshold for fuzzy matcher (0.0-1.0)") strictMode := fs.Bool("strict", true, "Fail test if any cassette interaction is uncalled") listenAddr := fs.String("addr", "127.0.0.1:0", "Local address to bind proxy on") + timingStr := fs.String("timing", "instant", "SSE stream chunk timing: instant, scaled, or exact") + delayScale := fs.Float64("delay-scale", 1.0, "Multiplier applied to recorded SSE delta_ms when -timing=scaled") + maxDelay := fs.Duration("max-delay", 2*time.Second, "Maximum per-chunk delay cap in scaled/exact SSE replay (0 for uncapped)") + mcpMode := fs.Bool("mcp", false, "Run deterministic MCP JSON-RPC replay over stdin/stdout") - var cmdArgs []string - dashIdx := -1 - for i, arg := range args { - if arg == "--" { - dashIdx = i - break - } - } - if dashIdx != -1 { - if err := fs.Parse(args[:dashIdx]); err != nil { - return err - } - cmdArgs = args[dashIdx+1:] - } else { - if err := fs.Parse(args); err != nil { - return err - } + flagArgs, cmdArgs := splitDashArgs(args) + if err := fs.Parse(flagArgs); err != nil { + return err } if *casPath == "" { return fmt.Errorf("-cassette is required") } + sseMode, err := cassette.ParseSSETimingMode(*timingStr) + if err != nil { + return err + } + sseTiming := cassette.SSETimingConfig{ + Mode: sseMode, + Scale: *delayScale, + MaxChunkDelay: *maxDelay, + } + cas, err := cassette.Load(*casPath) if err != nil { return fmt.Errorf("failed to load cassette: %w", err) } var m matcher.Matcher - if *matcherType == "fuzzy" { - m = matcher.NewFuzzyMatcher(0.80) + if strings.EqualFold(*matcherType, "fuzzy") { + m = matcher.NewFuzzyMatcher(*threshold) } else { m = matcher.NewStrictMatcher() } + if *mcpMode { + replayer := proxy.NewStdioReplayer(cas, m) + return replayer.ProcessStream(os.Stdin, os.Stdout) + } + if len(cmdArgs) > 0 { report, err := runner.Run(runner.RunOptions{ Cassette: cas, Matcher: m, Command: cmdArgs, StrictMode: *strictMode, + SSETiming: sseTiming, }) if err != nil { return err } fmt.Print(report.FormatSummary()) if !report.Passed { - os.Exit(1) + return fmt.Errorf("regression check failed") } return nil } @@ -350,6 +647,7 @@ func handleReplay(args []string) error { Cassette: cas, Matcher: m, ListenAddr: *listenAddr, + SSETiming: sseTiming, }) if err != nil { return err @@ -359,9 +657,10 @@ func handleReplay(args []string) error { } defer srv.Close() - fmt.Printf("[✓] AgentFixture Replay Server active on %s\n", srv.URL()) - fmt.Printf(" Loaded: %s (%d interactions)\n", *casPath, len(cas.Interactions)) - fmt.Println(" Running in zero-token replay mode. Press Ctrl+C to stop.") + fmt.Printf("[OK] AgentFixture Replay Server active on %s\n", srv.URL()) + fmt.Printf(" Loaded: %s (%d interactions)\n", *casPath, len(cas.Interactions)) + fmt.Printf(" SSE Timing: %s\n", sseTiming.Mode) + fmt.Println(" Running in zero-token offline replay mode. Press Ctrl+C to stop.") sigChan := make(chan os.Signal, 1) signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM) diff --git a/cmd/agentfixture/main_test.go b/cmd/agentfixture/main_test.go index 41e9404..a89f2f9 100644 --- a/cmd/agentfixture/main_test.go +++ b/cmd/agentfixture/main_test.go @@ -1,6 +1,10 @@ package main import ( + "errors" + "io" + "net/http" + "net/http/httptest" "os" "path/filepath" "strings" @@ -9,21 +13,41 @@ import ( "github.com/aloneangel45/AgentFixture/pkg/cassette" ) -func TestCLI_VerifyCommand(t *testing.T) { +func TestCLI_VersionFormattingNoDoubleV(t *testing.T) { + if got := formatVersionString("v1.1.0"); !strings.Contains(got, "AgentFixture v1.1.0") || strings.Contains(got, "vv1.1.0") { + t.Fatalf("unexpected version formatting for v1.1.0: %q", got) + } + if got := formatVersionString("1.1.0"); !strings.Contains(got, "AgentFixture v1.1.0") { + t.Fatalf("unexpected version formatting for 1.1.0: %q", got) + } +} + +func TestCLI_VerifyInspectAndRedactCommands(t *testing.T) { tmpDir := t.TempDir() validCasPath := filepath.Join(tmpDir, "valid.yaml") + events, err := cassette.DecodeSSE(strings.NewReader( + "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"model\":\"claude-sonnet-4-20250514\",\"usage\":{\"input_tokens\":50,\"output_tokens\":1}}}\n\n" + + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"verified\"}}\n\n" + + "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},\"usage\":{\"output_tokens\":10}}\n\n" + + "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n", + )) + if err != nil { + t.Fatal(err) + } + cas := cassette.NewCassette("cas_verify_test") cas.Interactions = append(cas.Interactions, cassette.Interaction{ ID: "int_001", Sequence: 1, - Type: cassette.InteractionTypeTool, + Type: cassette.InteractionTypeSSE, Request: cassette.Request{ - ToolName: "view_file", + Method: "POST", + Endpoint: "/v1/messages", }, Response: cassette.Response{ StatusCode: 200, - ToolResult: "file content", + SSEEvents: events, }, }) @@ -31,41 +55,67 @@ func TestCLI_VerifyCommand(t *testing.T) { t.Fatalf("failed to save test cassette: %v", err) } - // Verify command should succeed if err := handleVerify([]string{validCasPath}); err != nil { t.Fatalf("expected handleVerify to succeed, got: %v", err) } + if err := handleInspect([]string{validCasPath}); err != nil { + t.Fatalf("expected handleInspect text mode to succeed, got: %v", err) + } + if err := handleInspect([]string{"-json", validCasPath}); err != nil { + t.Fatalf("expected handleInspect -json mode to succeed, got: %v", err) + } - // Corrupt cassette without interaction ID - invalidCasPath := filepath.Join(tmpDir, "invalid.yaml") - casBad := cassette.NewCassette("bad_cas") - casBad.Interactions = append(casBad.Interactions, cassette.Interaction{ - Sequence: 1, - Type: cassette.InteractionTypeTool, - }) - _ = cassette.Save(invalidCasPath, casBad) - - if err := handleVerify([]string{invalidCasPath}); err == nil { - t.Fatalf("expected handleVerify to fail on invalid cassette") + // Write raw unredacted YAML directly to disk and verify fails until `redact` runs + leakyPath := filepath.Join(tmpDir, "leaky.yaml") + rawLeakyYAML := `version: agentfixture/v1 +metadata: + cassette_id: leaky_01 +interactions: + - id: int_leak + sequence: 1 + type: http + request: + method: POST + endpoint: /v1/messages + headers: + Authorization: Bearer sk-ant-api03-1234567890abcdef1234567890 + raw_body: '{"prompt":"secret sk-proj-1234567890abcdef1234567890"}' + response: + status_code: 200 +` + if err := os.WriteFile(leakyPath, []byte(rawLeakyYAML), 0644); err != nil { + t.Fatal(err) + } + if err := handleVerify([]string{leakyPath}); err == nil { + t.Fatalf("expected handleVerify to reject unredacted cassette") + } + if err := handleRedact([]string{leakyPath}); err != nil { + t.Fatalf("handleRedact failed: %v", err) + } + if err := handleVerify([]string{leakyPath}); err != nil { + t.Fatalf("expected handleVerify to pass after handleRedact, got: %v", err) } } -func TestCLI_DiffCommand(t *testing.T) { +func TestCLI_DiffCommand_ExitCodeAndSSESemanticDiff(t *testing.T) { tmpDir := t.TempDir() casPath1 := filepath.Join(tmpDir, "cas1.yaml") casPath2 := filepath.Join(tmpDir, "cas2.yaml") + ev1, _ := cassette.DecodeSSE(strings.NewReader( + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Hello A\"}}\n\n", + )) + ev2, _ := cassette.DecodeSSE(strings.NewReader( + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Hello B\"}}\n\n", + )) + c1 := cassette.NewCassette("c1") c1.Interactions = append(c1.Interactions, cassette.Interaction{ ID: "int_01", Sequence: 1, - Type: cassette.InteractionTypeTool, - Request: cassette.Request{ - ToolName: "view_file", - ToolInput: map[string]interface{}{ - "path": "a.txt", - }, - }, + Type: cassette.InteractionTypeSSE, + Request: cassette.Request{Method: "POST", Endpoint: "/v1/messages"}, + Response: cassette.Response{StatusCode: 200, SSEEvents: ev1}, }) _ = cassette.Save(casPath1, c1) @@ -73,18 +123,84 @@ func TestCLI_DiffCommand(t *testing.T) { c2.Interactions = append(c2.Interactions, cassette.Interaction{ ID: "int_01", Sequence: 1, - Type: cassette.InteractionTypeTool, - Request: cassette.Request{ - ToolName: "view_file", - ToolInput: map[string]interface{}{ - "path": "b.txt", // drifted argument - }, - }, + Type: cassette.InteractionTypeSSE, + Request: cassette.Request{Method: "POST", Endpoint: "/v1/messages"}, + Response: cassette.Response{StatusCode: 200, SSEEvents: ev2}, }) _ = cassette.Save(casPath2, c2) + // Without -exit-code returns nil if err := handleDiff([]string{casPath1, casPath2}); err != nil { - t.Fatalf("handleDiff returned unexpected error: %v", err) + t.Fatalf("handleDiff without -exit-code returned unexpected error: %v", err) + } + // With -exit-code returns errDiffDetected on SSE semantic drift + if err := handleDiff([]string{"-exit-code", casPath1, casPath2}); !errors.Is(err, errDiffDetected) { + t.Fatalf("expected errDiffDetected with -exit-code, got: %v", err) + } + // Identical files with -exit-code return nil + if err := handleDiff([]string{"-exit-code", casPath1, casPath1}); err != nil { + t.Fatalf("expected nil on identical cassettes with -exit-code, got: %v", err) + } +} + +func TestCLI_RecordCommand_RecordsSubprocessInteractions(t *testing.T) { + // Helper subprocess entry point: when AGENTFIXTURE_CLI_HELPER=1, act as an agent making an HTTP call to ANTHROPIC_BASE_URL! + if os.Getenv("AGENTFIXTURE_CLI_HELPER") == "1" { + baseURL := os.Getenv("ANTHROPIC_BASE_URL") + resp, err := http.Post(baseURL+"/v1/messages", "application/json", strings.NewReader(`{"model":"claude-sonnet-4","stream":true}`)) + if err != nil { + os.Exit(2) + } + _, _ = io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusOK { + os.Exit(3) + } + os.Exit(0) + } + + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_cli\",\"model\":\"claude-sonnet-4\",\"usage\":{\"input_tokens\":10,\"output_tokens\":4}}}\n\nevent: message_stop\ndata: {\"type\":\"message_stop\"}\n\n")) + })) + defer upstream.Close() + + outCas := filepath.Join(t.TempDir(), "recorded_cli.yaml") + exe, err := os.Executable() + if err != nil { + t.Fatal(err) + } + + t.Setenv("AGENTFIXTURE_CLI_HELPER", "1") + err = handleRecord([]string{ + "-cassette", outCas, + "-upstream", upstream.URL, + "--", + exe, "-test.run=^TestCLI_RecordCommand_RecordsSubprocessInteractions$", + }) + if err != nil { + t.Fatalf("handleRecord -- failed: %v", err) + } + + loaded, err := cassette.Load(outCas) + if err != nil { + t.Fatalf("failed to load recorded cassette: %v", err) + } + if len(loaded.Interactions) != 1 || loaded.Interactions[0].Type != cassette.InteractionTypeSSE { + t.Fatalf("expected 1 recorded SSE interaction from subprocess, got: %+v", loaded.Interactions) + } + + // Now replay the exact same subprocess with upstream CLOSED! + upstream.Close() + err = handleReplay([]string{ + "-cassette", outCas, + "-timing", "instant", + "--", + exe, "-test.run=^TestCLI_RecordCommand_RecordsSubprocessInteractions$", + }) + if err != nil { + t.Fatalf("handleReplay -- failed against closed upstream: %v", err) } } diff --git a/pkg/agenttest/agenttest.go b/pkg/agenttest/agenttest.go new file mode 100644 index 0000000..35c3fc0 --- /dev/null +++ b/pkg/agenttest/agenttest.go @@ -0,0 +1,84 @@ +// Package agenttest provides an ergonomic Go testing SDK for zero-dollar, zero-network +// deterministic VCR regression testing of LLM and tool-using agents. +package agenttest + +import ( + "net/http" + "testing" + + "github.com/aloneangel45/AgentFixture/pkg/cassette" + "github.com/aloneangel45/AgentFixture/pkg/matcher" + "github.com/aloneangel45/AgentFixture/pkg/proxy" +) + +// Option configures a TestServer instance. +type Option = proxy.TestOption + +// TestServer manages a local loopback VCR server bound to testing.TB. +type TestServer struct { + inner *proxy.TestServer +} + +// WithMatcher overrides the request Matcher (default: StrictMatcher). +func WithMatcher(m matcher.Matcher) Option { + return proxy.WithTestMatcher(m) +} + +// WithFuzzyMatcher configures semantic fuzzy matching with the given threshold (e.g. 0.85). +func WithFuzzyMatcher(threshold float64) Option { + return proxy.WithTestMatcher(matcher.NewFuzzyMatcher(threshold)) +} + +// WithSSETiming configures chunk pacing for recorded SSE streams (instant, scaled, or exact). +func WithSSETiming(cfg cassette.SSETimingConfig) Option { + return proxy.WithTestSSETiming(cfg) +} + +// WithStrictExhaustive asserts on tb.Cleanup that every interaction in the cassette was replayed. +func WithStrictExhaustive(strict bool) Option { + return proxy.WithTestStrictExhaustive(strict) +} + +// WithAllowUnmatched suppresses automatic tb.Cleanup errors when negative tests intentionally send unmatched requests. +func WithAllowUnmatched(allow bool) Option { + return proxy.WithTestAllowUnmatched(allow) +} + +// WithRecordMode switches the TestServer into live recording mode against upstreamURL and writes cassettePath on cleanup. +func WithRecordMode(upstreamURL string) Option { + return proxy.WithTestRecordMode(upstreamURL) +} + +// NewTestServer loads cassettePath, starts a local replay proxy on 127.0.0.1:0, +// provides a zero-outbound-network HTTPClient(), and registers automatic tb.Cleanup verification. +func NewTestServer(tb testing.TB, cassettePath string, opts ...Option) *TestServer { + tb.Helper() + return &TestServer{ + inner: proxy.NewTestServer(tb, cassettePath, opts...), + } +} + +// URL returns the local base URL (e.g. "http://127.0.0.1:54321") suitable for ANTHROPIC_BASE_URL or OPENAI_BASE_URL. +func (ts *TestServer) URL() string { + return ts.inner.URL() +} + +// HTTPClient returns a zero-network *http.Client whose DialContext blocks any non-loopback connection. +func (ts *TestServer) HTTPClient() *http.Client { + return ts.inner.HTTPClient() +} + +// Server returns the underlying *proxy.Server. +func (ts *TestServer) Server() *proxy.Server { + return ts.inner.Server() +} + +// Cassette returns a snapshot of the loaded or recorded Cassette. +func (ts *TestServer) Cassette() *cassette.Cassette { + return ts.inner.Server().GetCassette() +} + +// AssertAllMatched verifies that every interaction in the cassette has been replayed at least once. +func (ts *TestServer) AssertAllMatched() { + ts.inner.AssertAllMatched() +} diff --git a/pkg/agenttest/agenttest_test.go b/pkg/agenttest/agenttest_test.go new file mode 100644 index 0000000..d57621e --- /dev/null +++ b/pkg/agenttest/agenttest_test.go @@ -0,0 +1,77 @@ +package agenttest_test + +import ( + "errors" + "io" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "testing" + + "github.com/aloneangel45/AgentFixture/pkg/agenttest" + "github.com/aloneangel45/AgentFixture/pkg/proxy" +) + +func TestAgentTestSDK_RecordAndZeroNetworkReplay(t *testing.T) { + casPath := filepath.Join(t.TempDir(), "sdk_anthropic_sse.yaml") + + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(http.StatusOK) + flusher, _ := w.(http.Flusher) + chunks := []string{ + "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_sdk_1\",\"role\":\"assistant\",\"model\":\"claude-sonnet-4-20250514\",\"usage\":{\"input_tokens\":20,\"output_tokens\":1}}}\n\n", + "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"}}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"SDK deterministic stream!\"}}\n\n", + "event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\n", + "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},\"usage\":{\"output_tokens\":8}}\n\n", + "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n", + } + for _, c := range chunks { + _, _ = w.Write([]byte(c)) + if flusher != nil { + flusher.Flush() + } + } + })) + + // Subtest 1: Record into casPath automatically on t.Cleanup + t.Run("record", func(t *testing.T) { + ts := agenttest.NewTestServer(t, casPath, agenttest.WithRecordMode(upstream.URL)) + resp, err := ts.HTTPClient().Post(ts.URL()+"/v1/messages", "application/json", strings.NewReader(`{"model":"claude-sonnet-4-20250514","stream":true}`)) + if err != nil { + t.Fatalf("record post failed: %v", err) + } + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if !strings.Contains(string(body), "SDK deterministic stream!") { + t.Fatalf("unexpected live stream body: %s", string(body)) + } + }) + + // Shut down upstream completely to prove zero external dependency + upstream.Close() + + // Subtest 2: Replay with strict exhaustive check + verify zero-outbound-network HTTPClient + t.Run("replay_offline", func(t *testing.T) { + ts := agenttest.NewTestServer(t, casPath, agenttest.WithStrictExhaustive(true)) + client := ts.HTTPClient() + + resp, err := client.Post(ts.URL()+"/v1/messages", "application/json", strings.NewReader(`{"stream":true,"model":"claude-sonnet-4-20250514"}`)) + if err != nil { + t.Fatalf("replay post failed: %v", err) + } + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if !strings.Contains(string(body), "SDK deterministic stream!") { + t.Fatalf("unexpected replayed stream body: %s", string(body)) + } + + // Verify outbound external connection via direct offline dialer is strictly blocked + rawOffline := proxy.NewOfflineHTTPClient("") + if _, outErr := rawOffline.Get("http://1.1.1.1"); !errors.Is(outErr, proxy.ErrOutboundNetworkBlocked) { + t.Fatalf("expected outbound IP dial blocked with ErrOutboundNetworkBlocked, got: %v", outErr) + } + }) +} diff --git a/pkg/cassette/redact.go b/pkg/cassette/redact.go index 619b5eb..b219135 100644 --- a/pkg/cassette/redact.go +++ b/pkg/cassette/redact.go @@ -2,63 +2,137 @@ package cassette import ( "encoding/json" + "fmt" "regexp" "strings" ) var ( - anthropicKeyRegex = regexp.MustCompile(`sk-ant-[a-zA-Z0-9_-]{20,}`) - openaiProjRegex = regexp.MustCompile(`sk-proj-[a-zA-Z0-9_-]{20,}`) - openaiKeyRegex = regexp.MustCompile(`sk-[a-zA-Z0-9_-]{24,}`) - githubPatRegex = regexp.MustCompile(`github_pat_[a-zA-Z0-9_]{50,}`) - githubGhpRegex = regexp.MustCompile(`ghp_[a-zA-Z0-9]{36}`) - bearerTokenRegex = regexp.MustCompile(`(?i)\bBearer\s+[a-zA-Z0-9_\-\.~+/]+=*`) - awsAccessKeyRegex = regexp.MustCompile(`\bAKIA[0-9A-Z]{16}\b`) + pemPrivateKeyRegex = regexp.MustCompile(`(?s)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----`) + bearerTokenRegex = regexp.MustCompile(`(?i)\bBearer\s+([a-zA-Z0-9_\-\.~+/]+=*)`) + anthropicKeyRegex = regexp.MustCompile(`sk-ant-[a-zA-Z0-9_-]{16,}`) + openaiProjRegex = regexp.MustCompile(`sk-(?:proj|svcacct|admin)-[a-zA-Z0-9_-]{16,}`) + openaiKeyRegex = regexp.MustCompile(`sk-[a-zA-Z0-9_-]{20,}`) + googleKeyRegex = regexp.MustCompile(`\bAIza[0-9A-Za-z_-]{35}\b`) + githubPatRegex = regexp.MustCompile(`github_pat_[a-zA-Z0-9_]{22,}`) + githubClassicRegex = regexp.MustCompile(`\bgh[pousr]_[a-zA-Z0-9]{30,}\b`) + awsAccessKeyRegex = regexp.MustCompile(`\bA(?:KIA|SIA)[0-9A-Z]{16}\b`) + slackTokenRegex = regexp.MustCompile(`\bxox[baprs]-[0-9A-Za-z-]{10,}\b`) + huggingFaceRegex = regexp.MustCompile(`\bhf_[a-zA-Z0-9]{20,}\b`) + urlBasicAuthRegex = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9+.-]*://[^/\s:@\[\]]+:)([^@\s/\[\]]+)(@)`) + querySecretRegex = regexp.MustCompile(`(?i)([?&](?:api_key|apikey|access_token|client_secret|password|secret|x-goog-api-key)=)([^&\s"'\[\]]+)`) ) var sensitiveHeaderNames = map[string]bool{ - "authorization": true, - "x-api-key": true, - "anthropic-api-key": true, - "openai-api-key": true, - "x-auth-token": true, + "authorization": true, "proxy-authorization": true, + "x-api-key": true, + "x-goog-api-key": true, + "api-key": true, + "anthropic-api-key": true, + "openai-api-key": true, + "x-auth-token": true, + "cookie": true, + "set-cookie": true, } var sensitiveKeyNames = map[string]bool{ - "password": true, - "secret": true, - "api_key": true, - "apikey": true, - "access_token": true, - "private_key": true, - "token": true, - "client_secret": true, + "password": true, + "passwd": true, + "secret": true, + "api_key": true, + "apikey": true, + "x_api_key": true, + "x-api-key": true, + "x-goog-api-key": true, + "access_token": true, + "refresh_token": true, + "id_token": true, + "private_key": true, + "token": true, + "client_secret": true, + "authorization": true, + "cookie": true, + "set-cookie": true, + "session_token": true, + "auth_token": true, } -// RedactString sanitizes known API keys, tokens, and credentials from arbitrary text. +// RedactString sanitizes known API keys, tokens, PEM keys, URL credentials, and secrets from arbitrary text. func RedactString(input string) string { - if input == "" { - return "" - } - s := bearerTokenRegex.ReplaceAllString(input, "Bearer [REDACTED_TOKEN]") - s = anthropicKeyRegex.ReplaceAllString(s, "[REDACTED_ANTHROPIC_KEY]") - s = openaiProjRegex.ReplaceAllString(s, "[REDACTED_OPENAI_KEY]") - s = openaiKeyRegex.ReplaceAllString(s, "[REDACTED_OPENAI_KEY]") - s = githubPatRegex.ReplaceAllString(s, "[REDACTED_GITHUB_TOKEN]") - s = githubGhpRegex.ReplaceAllString(s, "[REDACTED_GITHUB_TOKEN]") - s = awsAccessKeyRegex.ReplaceAllString(s, "[REDACTED_AWS_KEY]") + if len(input) < 6 { + return input + } + s := input + if strings.Contains(s, "PRIVATE KEY-----") { + s = pemPrivateKeyRegex.ReplaceAllString(s, "[REDACTED_PRIVATE_KEY]") + } + if hasBearerCandidate(s) { + s = bearerTokenRegex.ReplaceAllString(s, "Bearer [REDACTED_TOKEN]") + } + if strings.Contains(s, "sk-") { + if strings.Contains(s, "sk-ant-") { + s = anthropicKeyRegex.ReplaceAllString(s, "[REDACTED_ANTHROPIC_KEY]") + } + s = openaiProjRegex.ReplaceAllString(s, "[REDACTED_OPENAI_KEY]") + s = openaiKeyRegex.ReplaceAllString(s, "[REDACTED_OPENAI_KEY]") + } + if strings.Contains(s, "AIza") { + s = googleKeyRegex.ReplaceAllString(s, "[REDACTED_GOOGLE_KEY]") + } + if strings.Contains(s, "github_pat_") { + s = githubPatRegex.ReplaceAllString(s, "[REDACTED_GITHUB_TOKEN]") + } + if strings.Contains(s, "gh") && (strings.Contains(s, "ghp_") || strings.Contains(s, "gho_") || strings.Contains(s, "ghu_") || strings.Contains(s, "ghs_") || strings.Contains(s, "ghr_")) { + s = githubClassicRegex.ReplaceAllString(s, "[REDACTED_GITHUB_TOKEN]") + } + if strings.Contains(s, "IA") && (strings.Contains(s, "AKIA") || strings.Contains(s, "ASIA")) { + s = awsAccessKeyRegex.ReplaceAllString(s, "[REDACTED_AWS_KEY]") + } + if strings.Contains(s, "xox") { + s = slackTokenRegex.ReplaceAllString(s, "[REDACTED_SLACK_TOKEN]") + } + if strings.Contains(s, "hf_") { + s = huggingFaceRegex.ReplaceAllString(s, "[REDACTED_HF_TOKEN]") + } + if strings.IndexByte(s, '@') != -1 && strings.Contains(s, "://") { + s = urlBasicAuthRegex.ReplaceAllString(s, "${1}[REDACTED]${3}") + } + if strings.IndexByte(s, '=') != -1 && (strings.IndexByte(s, '?') != -1 || strings.IndexByte(s, '&') != -1) { + s = querySecretRegex.ReplaceAllString(s, "${1}[REDACTED]") + } return s } -// RedactHeaders masks any known sensitive authentication headers. +func hasBearerCandidate(s string) bool { + n := len(s) + if n < 7 { + return false + } + for _, first := range [2]byte{'B', 'b'} { + for off := 0; off+6 < n; { + idx := strings.IndexByte(s[off:n-6], first) + if idx < 0 { + break + } + p := off + idx + if (s[p+1]|0x20) == 'e' && (s[p+2]|0x20) == 'a' && (s[p+3]|0x20) == 'r' && (s[p+4]|0x20) == 'e' && (s[p+5]|0x20) == 'r' { + return true + } + off = p + 1 + } + } + return false +} + +// RedactHeaders masks any known sensitive authentication or cookie headers. func RedactHeaders(headers map[string]string) map[string]string { if headers == nil { return nil } redacted := make(map[string]string, len(headers)) for k, v := range headers { - lower := strings.ToLower(k) + lower := strings.ToLower(strings.TrimSpace(k)) if sensitiveHeaderNames[lower] { redacted[k] = "[REDACTED]" } else { @@ -68,6 +142,23 @@ func RedactHeaders(headers map[string]string) map[string]string { return redacted } +// RedactStringMap scrubs sensitive keys and string values in string maps (tags, metadata). +func RedactStringMap(m map[string]string) map[string]string { + if m == nil { + return nil + } + res := make(map[string]string, len(m)) + for k, v := range m { + lower := strings.ToLower(strings.TrimSpace(k)) + if sensitiveKeyNames[lower] || sensitiveHeaderNames[lower] { + res[k] = "[REDACTED]" + } else { + res[k] = RedactString(v) + } + } + return res +} + // RedactMap recursively scrubs sensitive fields and masks known patterns in map values. func RedactMap(m map[string]interface{}) map[string]interface{} { if m == nil { @@ -75,26 +166,29 @@ func RedactMap(m map[string]interface{}) map[string]interface{} { } res := make(map[string]interface{}, len(m)) for k, v := range m { - lower := strings.ToLower(k) - if sensitiveKeyNames[lower] { + lower := strings.ToLower(strings.TrimSpace(k)) + if sensitiveKeyNames[lower] || sensitiveHeaderNames[lower] { res[k] = "[REDACTED]" continue } - res[k] = redactValue(v) + res[k] = RedactValue(v) } return res } -func redactValue(val interface{}) interface{} { +// RedactValue recursively scrubs strings, maps, and slices for secret values. +func RedactValue(val interface{}) interface{} { switch v := val.(type) { case string: return RedactString(v) case map[string]interface{}: return RedactMap(v) + case map[string]string: + return RedactStringMap(v) case []interface{}: items := make([]interface{}, len(v)) for i, el := range v { - items[i] = redactValue(el) + items[i] = RedactValue(el) } return items default: @@ -102,63 +196,201 @@ func redactValue(val interface{}) interface{} { } } -// RedactInteraction applies secret scrubbing to request and response fields. -func RedactInteraction(inter *Interaction) { - inter.Request.Headers = RedactHeaders(inter.Request.Headers) - inter.Request.RawBody = RedactString(inter.Request.RawBody) - if inter.Request.Body != nil { - inter.Request.Body = redactValue(inter.Request.Body) +// RedactRequest scrubs secrets from all fields of a Request. +func RedactRequest(req *Request) { + if req == nil { + return } - if inter.Request.ToolInput != nil { - inter.Request.ToolInput = RedactMap(inter.Request.ToolInput) + req.Endpoint = RedactString(req.Endpoint) + req.Headers = RedactHeaders(req.Headers) + req.RawBody = RedactString(req.RawBody) + if req.Body != nil { + req.Body = RedactValue(req.Body) } - inter.Request.Matcher.FuzzySignature.NormalizedPrompt = RedactString(inter.Request.Matcher.FuzzySignature.NormalizedPrompt) - if inter.Request.Matcher.FuzzySignature.NormalizedArgs != nil { - inter.Request.Matcher.FuzzySignature.NormalizedArgs = RedactMap(inter.Request.Matcher.FuzzySignature.NormalizedArgs) + if req.ToolInput != nil { + req.ToolInput = RedactMap(req.ToolInput) } + req.Matcher.FuzzySignature.NormalizedPrompt = RedactString(req.Matcher.FuzzySignature.NormalizedPrompt) + if req.Matcher.FuzzySignature.NormalizedArgs != nil { + req.Matcher.FuzzySignature.NormalizedArgs = RedactMap(req.Matcher.FuzzySignature.NormalizedArgs) + } +} + +// RedactInteraction applies secret scrubbing to request, response, SSE events, and metadata fields. +func RedactInteraction(inter *Interaction) { + if inter == nil { + return + } + RedactRequest(&inter.Request) + inter.Metadata = RedactStringMap(inter.Metadata) inter.Response.Headers = RedactHeaders(inter.Response.Headers) inter.Response.RawBody = RedactString(inter.Response.RawBody) inter.Response.ToolResult = RedactString(inter.Response.ToolResult) inter.Response.Error = RedactString(inter.Response.Error) if inter.Response.Body != nil { - inter.Response.Body = redactValue(inter.Response.Body) + inter.Response.Body = RedactValue(inter.Response.Body) + } + for i := range inter.Response.SSEEvents { + inter.Response.SSEEvents[i].Data = RedactString(inter.Response.SSEEvents[i].Data) + inter.Response.SSEEvents[i].Comment = RedactString(inter.Response.SSEEvents[i].Comment) + inter.Response.SSEEvents[i].ID = RedactString(inter.Response.SSEEvents[i].ID) } } // RedactCassette scrubs all interactions and metadata within a Cassette. func RedactCassette(c *Cassette) { + if c == nil { + return + } c.Metadata.Description = RedactString(c.Metadata.Description) + c.Metadata.Tags = RedactStringMap(c.Metadata.Tags) for i := range c.Interactions { RedactInteraction(&c.Interactions[i]) } } +func isRedactedPlaceholder(val string) bool { + trimmed := strings.TrimSpace(val) + if trimmed == "" { + return true + } + if strings.HasPrefix(trimmed, "[REDACTED") { + return true + } + if strings.HasPrefix(strings.ToLower(trimmed), "bearer [redacted") { + return true + } + return false +} + // CheckForUnredactedSecrets inspects a cassette and returns true if raw secrets remain. +// Matched secret values are strictly masked in the returned reason string so secret bytes never leak to logs. func CheckForUnredactedSecrets(c *Cassette) (bool, string) { + if c == nil { + return false, "" + } + data, err := json.Marshal(c) if err != nil { return false, "" } raw := string(data) - if loc := anthropicKeyRegex.FindString(raw); loc != "" { - return true, "unredacted Anthropic key detected: " + loc[:12] + "..." + // 1. Check known provider credential formats across all serialized fields + if pemPrivateKeyRegex.MatchString(raw) { + return true, "unredacted PEM private key detected (-----BEGIN *** PRIVATE KEY-----)" + } + if anthropicKeyRegex.MatchString(raw) { + return true, "unredacted Anthropic key detected (sk-ant-***)" + } + if openaiProjRegex.MatchString(raw) { + return true, "unredacted OpenAI project key detected (sk-proj-***)" + } + if openaiKeyRegex.MatchString(raw) { + return true, "unredacted OpenAI key detected (sk-***)" + } + if googleKeyRegex.MatchString(raw) { + return true, "unredacted Google API key detected (AIza***)" + } + if githubPatRegex.MatchString(raw) { + return true, "unredacted GitHub PAT detected (github_pat_***)" + } + if githubClassicRegex.MatchString(raw) { + return true, "unredacted GitHub token detected (gh*_**)" + } + if awsAccessKeyRegex.MatchString(raw) { + return true, "unredacted AWS access key detected (AKIA/ASIA***)" + } + if slackTokenRegex.MatchString(raw) { + return true, "unredacted Slack token detected (xox*-***)" + } + if huggingFaceRegex.MatchString(raw) { + return true, "unredacted HuggingFace token detected (hf_***)" } - if loc := openaiProjRegex.FindString(raw); loc != "" { - return true, "unredacted OpenAI project key detected: " + loc[:12] + "..." + if bearerTokenRegex.MatchString(raw) { + return true, "unredacted Bearer token detected (Bearer ***)" } - if loc := openaiKeyRegex.FindString(raw); loc != "" { - return true, "unredacted OpenAI key detected: " + loc[:8] + "..." + if urlBasicAuthRegex.MatchString(raw) { + return true, "unredacted URL basic auth credentials detected (scheme://user:***@host)" } - if loc := githubPatRegex.FindString(raw); loc != "" { - return true, "unredacted GitHub PAT detected: " + loc[:15] + "..." + if querySecretRegex.MatchString(raw) { + return true, "unredacted URL query secret parameter detected (?key=***)" } - if loc := githubGhpRegex.FindString(raw); loc != "" { - return true, "unredacted GitHub personal token detected: " + loc[:8] + "..." + + // 2. Check structured headers, tags, metadata, and maps for unmasked sensitive keys + for k, v := range c.Metadata.Tags { + lower := strings.ToLower(strings.TrimSpace(k)) + if (sensitiveKeyNames[lower] || sensitiveHeaderNames[lower]) && !isRedactedPlaceholder(v) { + return true, fmt.Sprintf("unredacted sensitive metadata tag %q detected ([***])", k) + } } - if loc := awsAccessKeyRegex.FindString(raw); loc != "" { - return true, "unredacted AWS key detected: " + loc[:8] + "..." + + for _, inter := range c.Interactions { + for k, v := range inter.Request.Headers { + if sensitiveHeaderNames[strings.ToLower(strings.TrimSpace(k))] && !isRedactedPlaceholder(v) { + return true, fmt.Sprintf("unredacted sensitive request header %q detected ([***])", k) + } + } + for k, v := range inter.Response.Headers { + if sensitiveHeaderNames[strings.ToLower(strings.TrimSpace(k))] && !isRedactedPlaceholder(v) { + return true, fmt.Sprintf("unredacted sensitive response header %q detected ([***])", k) + } + } + for k, v := range inter.Metadata { + lower := strings.ToLower(strings.TrimSpace(k)) + if (sensitiveKeyNames[lower] || sensitiveHeaderNames[lower]) && !isRedactedPlaceholder(v) { + return true, fmt.Sprintf("unredacted sensitive interaction metadata %q detected ([***])", k) + } + } + if keyName, found := findUnredactedMapKey(inter.Request.ToolInput); found { + return true, fmt.Sprintf("unredacted sensitive tool_input field %q detected ([***])", keyName) + } + if m, ok := inter.Request.Body.(map[string]interface{}); ok { + if keyName, found := findUnredactedMapKey(m); found { + return true, fmt.Sprintf("unredacted sensitive request body field %q detected ([***])", keyName) + } + } + if m, ok := inter.Response.Body.(map[string]interface{}); ok { + if keyName, found := findUnredactedMapKey(m); found { + return true, fmt.Sprintf("unredacted sensitive response body field %q detected ([***])", keyName) + } + } } + return false, "" } + +func findUnredactedMapKey(m map[string]interface{}) (string, bool) { + for k, v := range m { + lower := strings.ToLower(strings.TrimSpace(k)) + if sensitiveKeyNames[lower] || sensitiveHeaderNames[lower] { + switch s := v.(type) { + case string: + if !isRedactedPlaceholder(s) { + return k, true + } + default: + if v != nil { + return k, true + } + } + continue + } + if nested, ok := v.(map[string]interface{}); ok { + if nk, found := findUnredactedMapKey(nested); found { + return nk, true + } + } + if slice, ok := v.([]interface{}); ok { + for _, item := range slice { + if nested, ok := item.(map[string]interface{}); ok { + if nk, found := findUnredactedMapKey(nested); found { + return nk, true + } + } + } + } + } + return "", false +} diff --git a/pkg/cassette/redact_test.go b/pkg/cassette/redact_test.go index 1bc37a3..8397546 100644 --- a/pkg/cassette/redact_test.go +++ b/pkg/cassette/redact_test.go @@ -26,6 +26,11 @@ func TestRedaction_APIKeys(t *testing.T) { input: "Using key sk-proj-1234567890abcdef1234567890", expected: "Using key [REDACTED_OPENAI_KEY]", }, + { + name: "Google Gemini API key", + input: "Key AIzaSyDaGmWKa4JsXZ-HjGw7ISLn_3namBGewQe", + expected: "Key [REDACTED_GOOGLE_KEY]", + }, { name: "GitHub PAT", input: "Token github_pat_11ABCD123456789012345678901234567890123456789012345678901234567890", @@ -37,10 +42,40 @@ func TestRedaction_APIKeys(t *testing.T) { expected: "Token [REDACTED_GITHUB_TOKEN]", }, { - name: "AWS Access Key", + name: "GitHub gho OAuth token", + input: "Token gho_123456789012345678901234567890123456", + expected: "Token [REDACTED_GITHUB_TOKEN]", + }, + { + name: "AWS Access Key AKIA", input: "Credentials AKIAIOSFODNN7EXAMPLE", expected: "Credentials [REDACTED_AWS_KEY]", }, + { + name: "AWS STS Key ASIA", + input: "Credentials ASIAIOSFODNN7EXAMPLE", + expected: "Credentials [REDACTED_AWS_KEY]", + }, + { + name: "Slack Bot Token", + input: "Slack " + strings.Join([]string{"xoxb", "123456789012", "1234567890123", "abcdefghijklmnopqrstuvwx"}, "-"), + expected: "Slack [REDACTED_SLACK_TOKEN]", + }, + { + name: "HuggingFace Token", + input: "HF " + strings.Join([]string{"hf", "abcdefghijklmnopqrstuvwxyzABCDEFGH"}, "_"), + expected: "HF [REDACTED_HF_TOKEN]", + }, + { + name: "URL Basic Auth Credentials", + input: "postgres://admin:superSecretPass99@db.internal:5432/prod", + expected: "postgres://admin:[REDACTED]@db.internal:5432/prod", + }, + { + name: "PEM Private Key", + input: "Key:\n-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA12345\n-----END RSA PRIVATE KEY-----", + expected: "Key:\n[REDACTED_PRIVATE_KEY]", + }, } for _, tc := range cases { @@ -49,6 +84,11 @@ func TestRedaction_APIKeys(t *testing.T) { if got != tc.expected { t.Errorf("expected %q, got %q", tc.expected, got) } + // Verify idempotency + gotAgain := RedactString(got) + if gotAgain != got { + t.Errorf("RedactString not idempotent: %q -> %q", got, gotAgain) + } }) } } @@ -57,16 +97,18 @@ func TestRedaction_Headers(t *testing.T) { headers := map[string]string{ "Authorization": "Bearer sk-ant-secret1234567890", "X-Api-Key": "super_secret_token", + "x-goog-api-key": "AIzaSyDaGmWKa4JsXZ-HjGw7ISLn_3namBGewQe", + "Cookie": "session_id=xyz987secret", + "Set-Cookie": "auth=secret_cookie; HttpOnly", "Content-Type": "application/json", "Anthropic-Version": "2023-06-01", } redacted := RedactHeaders(headers) - if redacted["Authorization"] != "[REDACTED]" { - t.Errorf("expected Authorization to be [REDACTED], got %q", redacted["Authorization"]) - } - if redacted["X-Api-Key"] != "[REDACTED]" { - t.Errorf("expected X-Api-Key to be [REDACTED], got %q", redacted["X-Api-Key"]) + for _, h := range []string{"Authorization", "X-Api-Key", "x-goog-api-key", "Cookie", "Set-Cookie"} { + if redacted[h] != "[REDACTED]" { + t.Errorf("expected %s to be [REDACTED], got %q", h, redacted[h]) + } } if redacted["Content-Type"] != "application/json" { t.Errorf("expected Content-Type to remain unmasked, got %q", redacted["Content-Type"]) @@ -96,14 +138,23 @@ func TestRedaction_MapRecursion(t *testing.T) { } } -func TestCheckForUnredactedSecrets(t *testing.T) { +func TestCheckForUnredactedSecrets_MasksPreviewAndDetectsHeaders(t *testing.T) { + secretSuffix := "UNIQUE_SECRET_SUFFIX_99887766" c := NewCassette("test-cas") + c.Metadata.Tags["api_key"] = "raw_tag_secret" c.Interactions = append(c.Interactions, Interaction{ ID: "int_001", Sequence: 1, Type: InteractionTypeLLMChat, Request: Request{ - RawBody: "Using raw token sk-ant-1234567890123456789012345", + Endpoint: "https://user:pass123@api.example.com/v1/chat?api_key=raw_q_key", + Headers: map[string]string{ + "Cookie": "sid=unredacted_cookie", + }, + RawBody: "Using raw token sk-ant-" + secretSuffix, + }, + Metadata: map[string]string{ + "token": "raw_meta_token", }, }) @@ -114,10 +165,29 @@ func TestCheckForUnredactedSecrets(t *testing.T) { if !strings.Contains(reason, "Anthropic") { t.Errorf("expected reason to mention Anthropic, got %s", reason) } + if strings.Contains(reason, "UNIQUE") || strings.Contains(reason, secretSuffix[:6]) { + t.Fatalf("SECURITY LEAK: CheckForUnredactedSecrets leaked secret characters in error message: %s", reason) + } + + // Also verify standalone Cookie / Bearer detection + cHeader := NewCassette("test-cookie") + cHeader.Interactions = append(cHeader.Interactions, Interaction{ + ID: "int_cookie", + Sequence: 1, + Type: InteractionTypeHTTP, + Request: Request{ + Headers: map[string]string{"Cookie": "sess=raw_session_value"}, + }, + }) + if ok, r := CheckForUnredactedSecrets(cHeader); !ok { + t.Fatalf("expected unredacted Cookie header to be flagged") + } else if strings.Contains(r, "raw_session_value") { + t.Fatalf("expected Cookie value masked in error reason: %s", r) + } RedactCassette(c) - hasSecretAfter, _ := CheckForUnredactedSecrets(c) + hasSecretAfter, afterReason := CheckForUnredactedSecrets(c) if hasSecretAfter { - t.Fatalf("expected cassette to be clean after RedactCassette") + t.Fatalf("expected cassette to be clean after RedactCassette, got: %s", afterReason) } } diff --git a/pkg/cassette/serializer.go b/pkg/cassette/serializer.go index 7850347..0b32230 100644 --- a/pkg/cassette/serializer.go +++ b/pkg/cassette/serializer.go @@ -10,11 +10,15 @@ import ( "gopkg.in/yaml.v3" ) -// ToYAML serializes a Cassette into formatted YAML bytes. +// ToYAML serializes a Cassette into formatted YAML bytes without mutating the caller's Cassette in place. func ToYAML(c *Cassette) ([]byte, error) { - c.ComputeMetrics() - RedactCassette(c) - return yaml.Marshal(c) + if c == nil { + return nil, fmt.Errorf("cannot serialize nil cassette") + } + snapshot := c.Clone() + snapshot.ComputeMetrics() + RedactCassette(snapshot) + return yaml.Marshal(snapshot) } // FromYAML parses YAML bytes into a Cassette and validates it. @@ -29,11 +33,15 @@ func FromYAML(data []byte) (*Cassette, error) { return &c, nil } -// ToJSON serializes a Cassette into indented JSON bytes. +// ToJSON serializes a Cassette into indented JSON bytes without mutating the caller's Cassette in place. func ToJSON(c *Cassette) ([]byte, error) { - c.ComputeMetrics() - RedactCassette(c) - return json.MarshalIndent(c, "", " ") + if c == nil { + return nil, fmt.Errorf("cannot serialize nil cassette") + } + snapshot := c.Clone() + snapshot.ComputeMetrics() + RedactCassette(snapshot) + return json.MarshalIndent(snapshot, "", " ") } // FromJSON parses JSON bytes into a Cassette and validates it. diff --git a/pkg/cassette/sse.go b/pkg/cassette/sse.go new file mode 100644 index 0000000..6718e5a --- /dev/null +++ b/pkg/cassette/sse.go @@ -0,0 +1,851 @@ +package cassette + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "math" + "net/http" + "sort" + "strconv" + "strings" + "time" +) + +// Anthropic Messages API Server-Sent Event type names. +const ( + SSEEventMessageStart = "message_start" + SSEEventContentBlockStart = "content_block_start" + SSEEventContentBlockDelta = "content_block_delta" + SSEEventContentBlockStop = "content_block_stop" + SSEEventMessageDelta = "message_delta" + SSEEventMessageStop = "message_stop" + SSEEventPing = "ping" + SSEEventError = "error" +) + +// Anthropic Messages API content_block_delta inner delta types. +const ( + SSEDeltaText = "text_delta" + SSEDeltaThinking = "thinking_delta" + SSEDeltaSignature = "signature_delta" + SSEDeltaInputJSON = "input_json_delta" + SSEDeltaCitations = "citations_delta" +) + +// OpenAISSEDone is the sentinel data marker terminating OpenAI-compatible SSE streams. +const OpenAISSEDone = "[DONE]" + +// SSEEvent captures a single Server-Sent Event frame and its inter-chunk arrival delta. +type SSEEvent struct { + Index int `json:"index" yaml:"index"` + Event string `json:"event,omitempty" yaml:"event,omitempty"` + ID string `json:"id,omitempty" yaml:"id,omitempty"` + Retry int `json:"retry,omitempty" yaml:"retry,omitempty"` + Comment string `json:"comment,omitempty" yaml:"comment,omitempty"` + Data string `json:"data,omitempty" yaml:"data,omitempty"` + DeltaMs int64 `json:"delta_ms,omitempty" yaml:"delta_ms,omitempty"` +} + +// SSETimingMode controls inter-chunk timing behavior during SSE stream replay. +type SSETimingMode string + +const ( + SSETimingInstant SSETimingMode = "instant" + SSETimingScaled SSETimingMode = "scaled" + SSETimingExact SSETimingMode = "exact" +) + +// SSETimingConfig configures chunk pacing when replaying recorded SSE streams. +type SSETimingConfig struct { + Mode SSETimingMode + Scale float64 + MaxChunkDelay time.Duration +} + +// ParseSSETimingMode parses a CLI or configuration string into an SSETimingMode. +func ParseSSETimingMode(raw string) (SSETimingMode, error) { + switch strings.ToLower(strings.TrimSpace(raw)) { + case "", "instant": + return SSETimingInstant, nil + case "scaled": + return SSETimingScaled, nil + case "exact": + return SSETimingExact, nil + default: + return "", fmt.Errorf("unsupported SSE timing mode %q (expected instant, scaled, or exact)", raw) + } +} + +// ToolUseSummary represents a reconstructed tool invocation assembled from streamed JSON fragments. +type ToolUseSummary struct { + Index int `json:"index" yaml:"index"` + ID string `json:"id" yaml:"id"` + Name string `json:"name" yaml:"name"` + RawInput string `json:"raw_input,omitempty" yaml:"raw_input,omitempty"` + Input map[string]interface{} `json:"input,omitempty" yaml:"input,omitempty"` + ParseErr string `json:"parse_error,omitempty" yaml:"parse_error,omitempty"` +} + +// StreamSummary aggregates the semantic result of an Anthropic or OpenAI SSE stream. +type StreamSummary struct { + Provider string `json:"provider,omitempty" yaml:"provider,omitempty"` + MessageID string `json:"message_id,omitempty" yaml:"message_id,omitempty"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + Role string `json:"role,omitempty" yaml:"role,omitempty"` + Text string `json:"text,omitempty" yaml:"text,omitempty"` + ThinkingBlocks []string `json:"thinking_blocks,omitempty" yaml:"thinking_blocks,omitempty"` + Signatures []string `json:"signatures,omitempty" yaml:"signatures,omitempty"` + Citations []interface{} `json:"citations,omitempty" yaml:"citations,omitempty"` + ToolCalls []ToolUseSummary `json:"tool_calls,omitempty" yaml:"tool_calls,omitempty"` + StopReason string `json:"stop_reason,omitempty" yaml:"stop_reason,omitempty"` + StopSequence string `json:"stop_sequence,omitempty" yaml:"stop_sequence,omitempty"` + Usage Usage `json:"usage,omitempty" yaml:"usage,omitempty"` + ErrorMessage string `json:"error_message,omitempty" yaml:"error_message,omitempty"` + EventCount int `json:"event_count" yaml:"event_count"` + TotalDeltaMs int64 `json:"total_delta_ms" yaml:"total_delta_ms"` +} + +// DecodeSSE parses an SSE byte stream into discrete SSEEvents with zero line-size ceiling. +// DeltaMs is normalized to 0 for deterministic offline decoding. +func DecodeSSE(r io.Reader) ([]SSEEvent, error) { + events, err := RecordSSEStream(r, nil) + for i := range events { + events[i].DeltaMs = 0 + } + return events, err +} + +// RecordSSEStream reads an SSE stream using bufio.Reader (supporting arbitrarily large lines +// well beyond bufio.Scanner's 64KB cap), measures inter-event DeltaMs arrival latency, and +// invokes onEvent immediately upon each completed SSE frame for zero-lag live proxy flushing. +func RecordSSEStream(r io.Reader, onEvent func(SSEEvent) error) ([]SSEEvent, error) { + br := bufio.NewReaderSize(r, 64*1024) + var events []SSEEvent + + var ( + curEvent string + curID string + curRetry int + curComments []string + curData []string + hasFields bool + ) + + lastTick := time.Now() + + flushEvent := func() error { + if !hasFields { + return nil + } + now := time.Now() + delta := now.Sub(lastTick).Milliseconds() + if delta < 0 { + delta = 0 + } + lastTick = now + + ev := SSEEvent{ + Index: len(events), + Event: curEvent, + ID: curID, + Retry: curRetry, + Comment: strings.Join(curComments, "\n"), + Data: strings.Join(curData, "\n"), + DeltaMs: delta, + } + events = append(events, ev) + + curEvent = "" + curID = "" + curRetry = 0 + curComments = curComments[:0] + curData = curData[:0] + hasFields = false + + if onEvent != nil { + return onEvent(ev) + } + return nil + } + + processLine := func(rawLine string) error { + line := strings.TrimRight(rawLine, "\r\n") + if line == "" { + return flushEvent() + } + + // Comment line (": ...") + if strings.HasPrefix(line, ":") { + comment := line[1:] + if strings.HasPrefix(comment, " ") { + comment = comment[1:] + } + curComments = append(curComments, comment) + hasFields = true + return nil + } + + field := line + value := "" + if colonIdx := strings.IndexByte(line, ':'); colonIdx != -1 { + field = line[:colonIdx] + value = line[colonIdx+1:] + // Per WHATWG SSE spec, strip at most one single leading U+0020 SPACE after colon. + if strings.HasPrefix(value, " ") { + value = value[1:] + } + } + + switch field { + case "event": + curEvent = value + hasFields = true + case "data": + curData = append(curData, value) + hasFields = true + case "id": + if !strings.ContainsRune(value, '\x00') { + curID = value + hasFields = true + } + case "retry": + if ms, err := strconv.Atoi(strings.TrimSpace(value)); err == nil && ms >= 0 { + curRetry = ms + hasFields = true + } + default: + // Per SSE specification, unknown field names are ignored. + } + return nil + } + + for { + line, err := br.ReadString('\n') + if len(line) > 0 { + if pErr := processLine(line); pErr != nil { + return events, pErr + } + } + if err != nil { + if err == io.EOF { + if fErr := flushEvent(); fErr != nil { + return events, fErr + } + return events, nil + } + return events, err + } + } +} + +// EncodeSSEEvent formats a single SSEEvent into standard wire-protocol SSE bytes. +func EncodeSSEEvent(ev SSEEvent) []byte { + var buf bytes.Buffer + encodeSSEEventInto(&buf, ev) + return buf.Bytes() +} + +// EncodeSSEStream encodes a slice of SSEEvents into a contiguous SSE byte stream. +func EncodeSSEStream(events []SSEEvent) []byte { + var buf bytes.Buffer + for _, ev := range events { + encodeSSEEventInto(&buf, ev) + } + return buf.Bytes() +} + +func encodeSSEEventInto(buf *bytes.Buffer, ev SSEEvent) { + wroteAny := false + if ev.Comment != "" { + for _, line := range strings.Split(ev.Comment, "\n") { + buf.WriteString(": ") + buf.WriteString(line) + buf.WriteByte('\n') + } + wroteAny = true + } + if ev.Event != "" { + buf.WriteString("event: ") + buf.WriteString(ev.Event) + buf.WriteByte('\n') + wroteAny = true + } + if ev.ID != "" { + buf.WriteString("id: ") + buf.WriteString(ev.ID) + buf.WriteByte('\n') + wroteAny = true + } + if ev.Retry > 0 { + buf.WriteString("retry: ") + buf.WriteString(strconv.Itoa(ev.Retry)) + buf.WriteByte('\n') + wroteAny = true + } + if ev.Data != "" || !wroteAny { + for _, line := range strings.Split(ev.Data, "\n") { + buf.WriteString("data: ") + buf.WriteString(line) + buf.WriteByte('\n') + } + } + buf.WriteByte('\n') +} + +// ReplaySSEStream writes recorded SSE events to w with optional per-chunk flushing and timing simulation. +// Honors ctx.Done() promptly without leaking timers. +func ReplaySSEStream(ctx context.Context, w io.Writer, flusher http.Flusher, events []SSEEvent, cfg SSETimingConfig) error { + if err := ctx.Err(); err != nil { + return err + } + + var timer *time.Timer + defer func() { + if timer != nil { + if !timer.Stop() { + select { + case <-timer.C: + default: + } + } + } + }() + + var buf bytes.Buffer + for _, ev := range events { + if err := ctx.Err(); err != nil { + return err + } + + delay := computeChunkDelay(ev.DeltaMs, cfg) + if delay > 0 { + if timer == nil { + timer = time.NewTimer(delay) + } else { + if !timer.Stop() { + select { + case <-timer.C: + default: + } + } + timer.Reset(delay) + } + + select { + case <-ctx.Done(): + return ctx.Err() + case <-timer.C: + } + } + + buf.Reset() + encodeSSEEventInto(&buf, ev) + if _, err := w.Write(buf.Bytes()); err != nil { + return err + } + if flusher != nil { + flusher.Flush() + } + } + + return nil +} + +func computeChunkDelay(deltaMs int64, cfg SSETimingConfig) time.Duration { + if deltaMs <= 0 { + return 0 + } + var d time.Duration + switch cfg.Mode { + case SSETimingInstant, "": + return 0 + case SSETimingScaled: + if cfg.Scale <= 0 { + return 0 + } + d = time.Duration(float64(deltaMs) * cfg.Scale * float64(time.Millisecond)) + case SSETimingExact: + d = time.Duration(deltaMs) * time.Millisecond + default: + return 0 + } + if cfg.MaxChunkDelay > 0 && d > cfg.MaxChunkDelay { + d = cfg.MaxChunkDelay + } + return d +} + +type activeAnthropicBlock struct { + index int + blockType string + toolID string + toolName string + initInput map[string]interface{} + textBuf strings.Builder + thinkingBuf strings.Builder + sigBuf strings.Builder + jsonBuf strings.Builder + closed bool +} + +type openAIToolAcc struct { + index int + id string + name string + argsBuf strings.Builder +} + +// ReconstructStreamSummary reassembles an Anthropic Messages API or OpenAI-compatible SSE +// stream into a complete structured StreamSummary (assistant text, extended thinking blocks, +// signatures, citations, concatenated & parsed tool_use JSON inputs, stop_reason, and Usage). +func ReconstructStreamSummary(events []SSEEvent) StreamSummary { + summary := StreamSummary{ + EventCount: len(events), + } + if len(events) == 0 { + return summary + } + + var fullText strings.Builder + blocks := make(map[int]*activeAnthropicBlock) + var blockOrder []int + + openAITools := make(map[int]*openAIToolAcc) + var openAIToolOrder []int + + getOrCreateBlock := func(idx int) *activeAnthropicBlock { + b, exists := blocks[idx] + if !exists { + b = &activeAnthropicBlock{index: idx} + blocks[idx] = b + blockOrder = append(blockOrder, idx) + } + return b + } + + finalizeBlock := func(b *activeAnthropicBlock) { + if b == nil || b.closed { + return + } + b.closed = true + switch b.blockType { + case "thinking": + if b.thinkingBuf.Len() > 0 { + summary.ThinkingBlocks = append(summary.ThinkingBlocks, b.thinkingBuf.String()) + } + if b.sigBuf.Len() > 0 { + summary.Signatures = append(summary.Signatures, b.sigBuf.String()) + } + case "tool_use", "server_tool_use": + rawJSON := strings.TrimSpace(b.jsonBuf.String()) + tc := ToolUseSummary{ + Index: b.index, + ID: b.toolID, + Name: b.toolName, + RawInput: rawJSON, + } + if rawJSON != "" { + parsed, err := decodeJSONMapUseNumber([]byte(rawJSON)) + if err != nil { + tc.ParseErr = err.Error() + } else { + tc.Input = parsed + } + } else if b.initInput != nil { + tc.Input = b.initInput + } else { + tc.Input = map[string]interface{}{} + } + summary.ToolCalls = append(summary.ToolCalls, tc) + } + } + + for _, ev := range events { + summary.TotalDeltaMs += ev.DeltaMs + trimmedData := strings.TrimSpace(ev.Data) + if trimmedData == "" { + continue + } + if trimmedData == OpenAISSEDone { + if summary.Provider == "" { + summary.Provider = "openai" + } + continue + } + + rawMap, err := decodeJSONMapUseNumber([]byte(trimmedData)) + if err != nil { + continue + } + + evType := ev.Event + if evType == "" { + if t, ok := rawMap["type"].(string); ok { + evType = t + } + } + + switch evType { + case SSEEventPing: + if summary.Provider == "" { + summary.Provider = "anthropic" + } + continue + + case SSEEventMessageStart: + summary.Provider = "anthropic" + if msg, ok := rawMap["message"].(map[string]interface{}); ok { + if id, ok := msg["id"].(string); ok { + summary.MessageID = id + } + if model, ok := msg["model"].(string); ok { + summary.Model = model + } + if role, ok := msg["role"].(string); ok { + summary.Role = role + } + if sr, ok := msg["stop_reason"].(string); ok && sr != "" { + summary.StopReason = sr + } + if uMap, ok := msg["usage"].(map[string]interface{}); ok { + mergeAnthropicUsage(&summary.Usage, uMap) + } + } + + case SSEEventContentBlockStart: + summary.Provider = "anthropic" + idx := asInt(rawMap["index"]) + b := getOrCreateBlock(idx) + if cb, ok := rawMap["content_block"].(map[string]interface{}); ok { + if bt, ok := cb["type"].(string); ok { + b.blockType = bt + } + switch b.blockType { + case "text": + if txt, ok := cb["text"].(string); ok && txt != "" { + b.textBuf.WriteString(txt) + fullText.WriteString(txt) + } + case "thinking": + if th, ok := cb["thinking"].(string); ok && th != "" { + b.thinkingBuf.WriteString(th) + } + if sig, ok := cb["signature"].(string); ok && sig != "" { + b.sigBuf.WriteString(sig) + } + case "tool_use", "server_tool_use": + if id, ok := cb["id"].(string); ok { + b.toolID = id + } + if name, ok := cb["name"].(string); ok { + b.toolName = name + } + if inp, ok := cb["input"].(map[string]interface{}); ok && len(inp) > 0 { + b.initInput = inp + } + } + } + + case SSEEventContentBlockDelta: + summary.Provider = "anthropic" + idx := asInt(rawMap["index"]) + b := getOrCreateBlock(idx) + if delta, ok := rawMap["delta"].(map[string]interface{}); ok { + dtype, _ := delta["type"].(string) + switch dtype { + case SSEDeltaText: + if b.blockType == "" { + b.blockType = "text" + } + if txt, ok := delta["text"].(string); ok { + b.textBuf.WriteString(txt) + fullText.WriteString(txt) + } + case SSEDeltaThinking: + if b.blockType == "" { + b.blockType = "thinking" + } + if th, ok := delta["thinking"].(string); ok { + b.thinkingBuf.WriteString(th) + } + case SSEDeltaSignature: + if b.blockType == "" { + b.blockType = "thinking" + } + if sig, ok := delta["signature"].(string); ok { + b.sigBuf.WriteString(sig) + } + case SSEDeltaInputJSON: + if b.blockType == "" { + b.blockType = "tool_use" + } + if pj, ok := delta["partial_json"].(string); ok { + b.jsonBuf.WriteString(pj) + } + case SSEDeltaCitations: + if cit, exists := delta["citation"]; exists && cit != nil { + summary.Citations = append(summary.Citations, cit) + } + } + } + + case SSEEventContentBlockStop: + summary.Provider = "anthropic" + idx := asInt(rawMap["index"]) + if b, ok := blocks[idx]; ok { + finalizeBlock(b) + } + + case SSEEventMessageDelta: + summary.Provider = "anthropic" + if delta, ok := rawMap["delta"].(map[string]interface{}); ok { + if sr, ok := delta["stop_reason"].(string); ok && sr != "" { + summary.StopReason = sr + } + if ss, ok := delta["stop_sequence"].(string); ok && ss != "" { + summary.StopSequence = ss + } + } + if uMap, ok := rawMap["usage"].(map[string]interface{}); ok { + mergeAnthropicUsage(&summary.Usage, uMap) + } + + case SSEEventMessageStop: + summary.Provider = "anthropic" + + case SSEEventError: + if errObj, ok := rawMap["error"].(map[string]interface{}); ok { + msg, _ := errObj["message"].(string) + errType, _ := errObj["type"].(string) + if errType != "" && msg != "" { + summary.ErrorMessage = errType + ": " + msg + } else if msg != "" { + summary.ErrorMessage = msg + } else { + summary.ErrorMessage = errType + } + } else if msg, ok := rawMap["error"].(string); ok { + summary.ErrorMessage = msg + } + + default: + // Check OpenAI Chat Completions / Responses streaming chunks + if choices, ok := rawMap["choices"].([]interface{}); ok { + summary.Provider = "openai" + if id, ok := rawMap["id"].(string); ok && id != "" { + summary.MessageID = id + } + if model, ok := rawMap["model"].(string); ok && model != "" { + summary.Model = model + } + for _, cItem := range choices { + cMap, ok := cItem.(map[string]interface{}) + if !ok { + continue + } + if fr, ok := cMap["finish_reason"].(string); ok && fr != "" { + summary.StopReason = fr + } + if delta, ok := cMap["delta"].(map[string]interface{}); ok { + if role, ok := delta["role"].(string); ok && role != "" { + summary.Role = role + } + if content, ok := delta["content"].(string); ok { + fullText.WriteString(content) + } + if reasoning, ok := delta["reasoning_content"].(string); ok && reasoning != "" { + b := getOrCreateBlock(-1) + b.blockType = "thinking" + b.thinkingBuf.WriteString(reasoning) + } + if tcs, ok := delta["tool_calls"].([]interface{}); ok { + for _, tcItem := range tcs { + tcMap, ok := tcItem.(map[string]interface{}) + if !ok { + continue + } + tcIdx := asInt(tcMap["index"]) + acc, exists := openAITools[tcIdx] + if !exists { + acc = &openAIToolAcc{index: tcIdx} + openAITools[tcIdx] = acc + openAIToolOrder = append(openAIToolOrder, tcIdx) + } + if id, ok := tcMap["id"].(string); ok && id != "" { + acc.id = id + } + if fn, ok := tcMap["function"].(map[string]interface{}); ok { + if name, ok := fn["name"].(string); ok && name != "" { + acc.name = name + } + if argsPart, ok := fn["arguments"].(string); ok { + acc.argsBuf.WriteString(argsPart) + } + } + } + } + } + } + } + if uMap, ok := rawMap["usage"].(map[string]interface{}); ok { + mergeOpenAIOrAnthropicUsage(&summary.Usage, uMap) + } + } + } + + // Finalize any Anthropic blocks that remained open (e.g. truncated stream) + sort.Ints(blockOrder) + for _, idx := range blockOrder { + finalizeBlock(blocks[idx]) + } + + // Finalize any OpenAI streamed tool calls + sort.Ints(openAIToolOrder) + for _, idx := range openAIToolOrder { + acc := openAITools[idx] + rawArgs := strings.TrimSpace(acc.argsBuf.String()) + tc := ToolUseSummary{ + Index: acc.index, + ID: acc.id, + Name: acc.name, + RawInput: rawArgs, + } + if rawArgs != "" { + parsed, err := decodeJSONMapUseNumber([]byte(rawArgs)) + if err != nil { + tc.ParseErr = err.Error() + } else { + tc.Input = parsed + } + } else { + tc.Input = map[string]interface{}{} + } + summary.ToolCalls = append(summary.ToolCalls, tc) + } + + summary.Text = fullText.String() + return summary +} + +func mergeAnthropicUsage(dst *Usage, uMap map[string]interface{}) { + if v, ok := uMap["input_tokens"]; ok { + if n := asInt(v); n > 0 || dst.InputTokens == 0 { + dst.InputTokens = n + } + } + if v, ok := uMap["output_tokens"]; ok { + // In Anthropic SSE, message_delta usage.output_tokens is cumulative total output tokens. + if n := asInt(v); n > 0 || dst.OutputTokens == 0 { + dst.OutputTokens = n + } + } + if v, ok := uMap["cache_creation_input_tokens"]; ok { + if n := asInt(v); n > 0 || dst.CacheCreationInputTokens == 0 { + dst.CacheCreationInputTokens = n + } + } + if v, ok := uMap["cache_read_input_tokens"]; ok { + if n := asInt(v); n > 0 || dst.CacheReadInputTokens == 0 { + dst.CacheReadInputTokens = n + } + } + if v, ok := uMap["total_tokens"]; ok { + if n := asInt(v); n > 0 { + dst.TotalTokens = n + } + } +} + +func mergeOpenAIOrAnthropicUsage(dst *Usage, uMap map[string]interface{}) { + mergeAnthropicUsage(dst, uMap) + if v, ok := uMap["prompt_tokens"]; ok { + dst.InputTokens = asInt(v) + } + if v, ok := uMap["completion_tokens"]; ok { + dst.OutputTokens = asInt(v) + } +} + +// ExtractUsageFromBody extracts provider Usage statistics from a decoded JSON response body map. +func ExtractUsageFromBody(body interface{}) Usage { + var u Usage + m, ok := body.(map[string]interface{}) + if !ok { + return u + } + if uMap, ok := m["usage"].(map[string]interface{}); ok { + mergeOpenAIOrAnthropicUsage(&u, uMap) + } + return u +} + +func decodeJSONMapUseNumber(data []byte) (map[string]interface{}, error) { + dec := json.NewDecoder(bytes.NewReader(data)) + dec.UseNumber() + var raw map[string]interface{} + if err := dec.Decode(&raw); err != nil { + return nil, err + } + norm, _ := normalizeDecodedNumbers(raw).(map[string]interface{}) + return norm, nil +} + +func normalizeDecodedNumbers(v interface{}) interface{} { + switch val := v.(type) { + case map[string]interface{}: + out := make(map[string]interface{}, len(val)) + for k, el := range val { + out[k] = normalizeDecodedNumbers(el) + } + return out + case []interface{}: + out := make([]interface{}, len(val)) + for i, el := range val { + out[i] = normalizeDecodedNumbers(el) + } + return out + case json.Number: + s := val.String() + if i, err := strconv.ParseInt(s, 10, 64); err == nil { + return i + } + if u, err := strconv.ParseUint(s, 10, 64); err == nil { + return u + } + if f, err := strconv.ParseFloat(s, 64); err == nil { + if !math.IsInf(f, 0) && !math.IsNaN(f) && math.Trunc(f) == f && + f >= -9007199254740992 && f <= 9007199254740992 { + return int64(f) + } + return f + } + return s + default: + return val + } +} + +func asInt(v interface{}) int { + switch n := v.(type) { + case int: + return n + case int64: + return int(n) + case uint64: + return int(n) + case float64: + return int(n) + case json.Number: + if i, err := n.Int64(); err == nil { + return int(i) + } + if f, err := n.Float64(); err == nil { + return int(f) + } + } + return 0 +} diff --git a/pkg/cassette/sse_bench_test.go b/pkg/cassette/sse_bench_test.go new file mode 100644 index 0000000..11ecbb1 --- /dev/null +++ b/pkg/cassette/sse_bench_test.go @@ -0,0 +1,74 @@ +package cassette + +import ( + "bytes" + "context" + "io" + "strings" + "testing" +) + +func BenchmarkDecodeSSE_AnthropicStream(b *testing.B) { + raw := []byte(strings.Repeat(sampleAnthropicSSE, 4)) + b.SetBytes(int64(len(raw))) + b.ReportAllocs() + b.ResetTimer() + + for i := 0; i < b.N; i++ { + _, err := DecodeSSE(bytes.NewReader(raw)) + if err != nil { + b.Fatal(err) + } + } +} + +func BenchmarkReconstructStreamSummary(b *testing.B) { + events, err := DecodeSSE(strings.NewReader(sampleAnthropicSSE)) + if err != nil { + b.Fatal(err) + } + b.ReportAllocs() + b.ResetTimer() + + for i := 0; i < b.N; i++ { + s := ReconstructStreamSummary(events) + if s.Model == "" { + b.Fatal("empty model") + } + } +} + +func BenchmarkReplaySSEStream_Instant(b *testing.B) { + events, err := DecodeSSE(strings.NewReader(sampleAnthropicSSE)) + if err != nil { + b.Fatal(err) + } + encoded := EncodeSSEStream(events) + b.SetBytes(int64(len(encoded))) + b.ReportAllocs() + ctx := context.Background() + cfg := SSETimingConfig{Mode: SSETimingInstant} + b.ResetTimer() + + for i := 0; i < b.N; i++ { + if err := ReplaySSEStream(ctx, io.Discard, nil, events, cfg); err != nil { + b.Fatal(err) + } + } +} + +func BenchmarkEncodeSSEStream(b *testing.B) { + events, err := DecodeSSE(strings.NewReader(sampleAnthropicSSE)) + if err != nil { + b.Fatal(err) + } + b.ReportAllocs() + b.ResetTimer() + + for i := 0; i < b.N; i++ { + out := EncodeSSEStream(events) + if len(out) == 0 { + b.Fatal("empty encoded stream") + } + } +} diff --git a/pkg/cassette/sse_test.go b/pkg/cassette/sse_test.go new file mode 100644 index 0000000..596f325 --- /dev/null +++ b/pkg/cassette/sse_test.go @@ -0,0 +1,324 @@ +package cassette + +import ( + "bytes" + "context" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "time" +) + +type testFlusher struct { + buf bytes.Buffer + flushes atomic.Int64 +} + +func (tf *testFlusher) Write(p []byte) (int, error) { + return tf.buf.Write(p) +} + +func (tf *testFlusher) Flush() { + tf.flushes.Add(1) +} + +const sampleAnthropicSSE = `: keep-alive + +event: message_start +data: {"type":"message_start","message":{"id":"msg_019X7z","type":"message","role":"assistant","model":"claude-sonnet-4-20250514","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":412,"cache_creation_input_tokens":100,"cache_read_input_tokens":50,"output_tokens":1}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":"","signature":""}} + +event: ping +data: {"type":"ping"} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"Let me inspect "}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"pkg/auth/token.go first."}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"sig_part_1_"}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"sig_part_2"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: content_block_start +data: {"type":"content_block_start","index":1,"content_block":{"type":"text","text":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"I will read "}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"the token validator."}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"citations_delta","citation":{"type":"char_location","cited_text":"ValidateToken","document_index":0,"start_char_index":10,"end_char_index":23}}} + +event: content_block_stop +data: {"type":"content_block_stop","index":1} + +event: content_block_start +data: {"type":"content_block_start","index":2,"content_block":{"type":"tool_use","id":"toolu_01ABC","name":"read_file","input":{}}} + +event: content_block_delta +data: {"type":"content_block_delta","index":2,"delta":{"type":"input_json_delta","partial_json":"{\"path\": \"pkg/auth/"}} + +event: content_block_delta +data: {"type":"content_block_delta","index":2,"delta":{"type":"input_json_delta","partial_json":"token.go\", \"limit\": 9007199254740993}"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":2} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"output_tokens":94}} + +event: message_stop +data: {"type":"message_stop"} + +` + +func TestSSE_AnthropicDecodeEncodeAndSummary(t *testing.T) { + events, err := DecodeSSE(strings.NewReader(sampleAnthropicSSE)) + if err != nil { + t.Fatalf("DecodeSSE failed: %v", err) + } + if len(events) != 20 { + t.Fatalf("expected 20 SSE events, got %d", len(events)) + } + if events[0].Comment != "keep-alive" { + t.Errorf("expected first event comment 'keep-alive', got %q", events[0].Comment) + } + + summary := ReconstructStreamSummary(events) + if summary.Provider != "anthropic" { + t.Errorf("expected provider 'anthropic', got %q", summary.Provider) + } + if summary.MessageID != "msg_019X7z" { + t.Errorf("expected message_id 'msg_019X7z', got %q", summary.MessageID) + } + if summary.Model != "claude-sonnet-4-20250514" { + t.Errorf("expected model 'claude-sonnet-4-20250514', got %q", summary.Model) + } + if summary.Text != "I will read the token validator." { + t.Errorf("unexpected reconstructed text: %q", summary.Text) + } + if len(summary.ThinkingBlocks) != 1 || summary.ThinkingBlocks[0] != "Let me inspect pkg/auth/token.go first." { + t.Errorf("unexpected thinking blocks: %+v", summary.ThinkingBlocks) + } + if len(summary.Signatures) != 1 || summary.Signatures[0] != "sig_part_1_sig_part_2" { + t.Errorf("unexpected signatures: %+v", summary.Signatures) + } + if len(summary.Citations) != 1 { + t.Errorf("expected 1 citation, got %d", len(summary.Citations)) + } + if len(summary.ToolCalls) != 1 { + t.Fatalf("expected 1 tool call, got %d", len(summary.ToolCalls)) + } + tc := summary.ToolCalls[0] + if tc.ID != "toolu_01ABC" || tc.Name != "read_file" { + t.Errorf("unexpected tool call metadata: %+v", tc) + } + if tc.Input["path"] != "pkg/auth/token.go" { + t.Errorf("unexpected tool input path: %v", tc.Input["path"]) + } + // Verify 64-bit integer precision preserved via UseNumber + if tc.Input["limit"] != int64(9007199254740993) { + t.Errorf("expected exact int64 9007199254740993, got %T(%v)", tc.Input["limit"], tc.Input["limit"]) + } + if summary.StopReason != "tool_use" { + t.Errorf("expected stop_reason 'tool_use', got %q", summary.StopReason) + } + if summary.Usage.InputTokens != 412 || summary.Usage.OutputTokens != 94 || + summary.Usage.CacheCreationInputTokens != 100 || summary.Usage.CacheReadInputTokens != 50 { + t.Errorf("unexpected usage metrics: %+v", summary.Usage) + } + + // Verify wire round-trip + encoded := EncodeSSEStream(events) + redecoded, err := DecodeSSE(bytes.NewReader(encoded)) + if err != nil { + t.Fatalf("re-decode failed: %v", err) + } + if len(redecoded) != len(events) { + t.Fatalf("expected %d events after round-trip, got %d", len(events), len(redecoded)) + } + for i := range events { + if redecoded[i] != events[i] { + t.Errorf("event %d differed after round-trip: expected %+v, got %+v", i, events[i], redecoded[i]) + } + } +} + +func TestSSE_OpenAIStreamCompatibility(t *testing.T) { + openAISSE := strings.Join([]string{ + `data: {"id":"chatcmpl-99","model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello "}}]}`, + "", + `data: {"id":"chatcmpl-99","model":"gpt-4o","choices":[{"index":0,"delta":{"content":"world!"}}]}`, + "", + `data: {"id":"chatcmpl-99","model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1","function":{"name":"run_sql","arguments":"{\"q\":"}}]}}]}`, + "", + `data: {"id":"chatcmpl-99","model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"SELECT 1\"}"}}]},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":30,"completion_tokens":12,"total_tokens":42}}`, + "", + `data: [DONE]`, + "", + }, "\r\n") + + events, err := DecodeSSE(strings.NewReader(openAISSE)) + if err != nil { + t.Fatalf("DecodeSSE failed on OpenAI CRLF stream: %v", err) + } + if len(events) != 5 { + t.Fatalf("expected 5 OpenAI SSE events, got %d", len(events)) + } + + summary := ReconstructStreamSummary(events) + if summary.Provider != "openai" { + t.Errorf("expected provider openai, got %q", summary.Provider) + } + if summary.Text != "Hello world!" { + t.Errorf("expected 'Hello world!', got %q", summary.Text) + } + if len(summary.ToolCalls) != 1 || summary.ToolCalls[0].Name != "run_sql" || summary.ToolCalls[0].Input["q"] != "SELECT 1" { + t.Errorf("unexpected OpenAI tool call summary: %+v", summary.ToolCalls) + } + if summary.Usage.InputTokens != 30 || summary.Usage.OutputTokens != 12 || summary.Usage.TotalTokens != 42 { + t.Errorf("unexpected OpenAI usage: %+v", summary.Usage) + } +} + +func TestSSE_LargeLineExceeding64KBScannerLimit(t *testing.T) { + hugeText := strings.Repeat("A", 200*1024) // 200 KB single line (> 64KB bufio.Scanner limit) + stream := "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"" + hugeText + "\"}}\n\n" + + var flushed int + events, err := RecordSSEStream(strings.NewReader(stream), func(ev SSEEvent) error { + flushed++ + return nil + }) + if err != nil { + t.Fatalf("RecordSSEStream failed on >64KB line: %v", err) + } + if len(events) != 1 || flushed != 1 { + t.Fatalf("expected 1 event flushed live, got len=%d flushed=%d", len(events), flushed) + } + summary := ReconstructStreamSummary(events) + if len(summary.Text) != 200*1024 { + t.Errorf("expected 204800 bytes reconstructed, got %d", len(summary.Text)) + } +} + +func TestSSE_ReplayTimingModesAndContextCancel(t *testing.T) { + events := []SSEEvent{ + {Index: 0, Event: SSEEventMessageStart, Data: `{"type":"message_start"}`, DeltaMs: 40}, + {Index: 1, Event: SSEEventContentBlockDelta, Data: `{"type":"content_block_delta"}`, DeltaMs: 40}, + {Index: 2, Event: SSEEventMessageStop, Data: `{"type":"message_stop"}`, DeltaMs: 40}, + } + + // 1. Instant mode (~0ms) + tf := &testFlusher{} + start := time.Now() + if err := ReplaySSEStream(context.Background(), tf, tf, events, SSETimingConfig{Mode: SSETimingInstant}); err != nil { + t.Fatalf("instant replay failed: %v", err) + } + if elapsed := time.Since(start); elapsed > 50*time.Millisecond { + t.Errorf("instant replay took too long: %v", elapsed) + } + if tf.flushes.Load() != 3 { + t.Errorf("expected 3 flushes, got %d", tf.flushes.Load()) + } + + // 2. Scaled mode with MaxChunkDelay cap + tfScaled := &testFlusher{} + startScaled := time.Now() + err := ReplaySSEStream(context.Background(), tfScaled, tfScaled, events, SSETimingConfig{ + Mode: SSETimingScaled, + Scale: 0.25, // 40ms * 0.25 = 10ms per event (~30ms total) + MaxChunkDelay: 15 * time.Millisecond, + }) + if err != nil { + t.Fatalf("scaled replay failed: %v", err) + } + if elapsed := time.Since(startScaled); elapsed < 20*time.Millisecond { + t.Errorf("expected scaled replay delay >= 20ms, got %v", elapsed) + } + + // 3. Context cancellation mid-stream without blocking + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Millisecond) + defer cancel() + longEvents := []SSEEvent{ + {Index: 0, Event: SSEEventPing, Data: `{"type":"ping"}`, DeltaMs: 5000}, + } + startCancel := time.Now() + err = ReplaySSEStream(ctx, &bytes.Buffer{}, nil, longEvents, SSETimingConfig{Mode: SSETimingExact}) + if err == nil { + t.Fatalf("expected context deadline error during exact replay") + } + if elapsed := time.Since(startCancel); elapsed > 250*time.Millisecond { + t.Errorf("expected prompt cancellation on ctx.Done(), took %v", elapsed) + } +} + +func TestSSE_CassetteSerializationRedactionAndMetrics(t *testing.T) { + tmpDir := t.TempDir() + cas := NewCassette("cas_sse_metrics") + + events, err := DecodeSSE(strings.NewReader(sampleAnthropicSSE)) + if err != nil { + t.Fatalf("DecodeSSE failed: %v", err) + } + // Inject secret in an SSE event to verify redaction + events = append(events, SSEEvent{ + Index: len(events), + Event: SSEEventContentBlockDelta, + Data: `{"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"key sk-ant-api03-1234567890abcdef1234567890"}}`, + }) + + cas.Interactions = append(cas.Interactions, Interaction{ + ID: "sse_001", + Sequence: 1, + Type: InteractionTypeSSE, + Request: Request{ + Method: "POST", + Endpoint: "/v1/messages", + }, + Response: Response{ + StatusCode: 200, + LatencyMs: 350, + SSEEvents: events, + }, + }) + + casPath := filepath.Join(tmpDir, "sse_cassette.yaml") + if err := Save(casPath, cas); err != nil { + t.Fatalf("Save failed: %v", err) + } + + // Ensure original in-memory cassette was NOT mutated by Save/ToYAML + if strings.Contains(cas.Interactions[0].Response.SSEEvents[len(events)-1].Data, "[REDACTED_ANTHROPIC_KEY]") { + t.Errorf("Save mutated caller cassette in-place instead of cloning") + } + + loaded, err := Load(casPath) + if err != nil { + t.Fatalf("Load failed: %v", err) + } + hasSecrets, reason := CheckForUnredactedSecrets(loaded) + if hasSecrets { + t.Fatalf("expected loaded SSE cassette to have all secrets redacted, got: %s", reason) + } + + // Anthropic tokens: 412 in + 94 out + 100 cache_create + 50 cache_read = 656 tokens + if loaded.Metadata.Metrics.EstimatedTokensSaved != 656 { + t.Errorf("expected 656 estimated tokens saved from Anthropic SSE stream, got %d", loaded.Metadata.Metrics.EstimatedTokensSaved) + } + if loaded.Metadata.Metrics.EstimatedCostSavedUSD <= 0 { + t.Errorf("expected positive cost saved USD, got %f", loaded.Metadata.Metrics.EstimatedCostSavedUSD) + } +} diff --git a/pkg/cassette/types.go b/pkg/cassette/types.go index 7b905f3..d146bdb 100644 --- a/pkg/cassette/types.go +++ b/pkg/cassette/types.go @@ -1,6 +1,7 @@ package cassette import ( + "encoding/json" "errors" "fmt" "time" @@ -16,6 +17,7 @@ const ( InteractionTypeLLMChat InteractionType = "llm_chat" InteractionTypeTool InteractionType = "tool_execution" InteractionTypeHTTP InteractionType = "http" + InteractionTypeSSE InteractionType = "sse" InteractionTypeStdio InteractionType = "stdio" ) @@ -83,13 +85,14 @@ type FuzzySignature struct { Model string `json:"model,omitempty" yaml:"model,omitempty"` } -// Response captures the recorded result, HTTP status, or tool output. +// Response captures the recorded result, HTTP status, SSE stream, or tool output. type Response struct { StatusCode int `json:"status_code" yaml:"status_code"` LatencyMs int64 `json:"latency_ms,omitempty" yaml:"latency_ms,omitempty"` Headers map[string]string `json:"headers,omitempty" yaml:"headers,omitempty"` Body interface{} `json:"body,omitempty" yaml:"body,omitempty"` RawBody string `json:"raw_body,omitempty" yaml:"raw_body,omitempty"` + SSEEvents []SSEEvent `json:"sse_events,omitempty" yaml:"sse_events,omitempty"` ToolResult string `json:"tool_result,omitempty" yaml:"tool_result,omitempty"` Error string `json:"error,omitempty" yaml:"error,omitempty"` Usage Usage `json:"usage,omitempty" yaml:"usage,omitempty"` @@ -97,9 +100,11 @@ type Response struct { // Usage tracks token counts reported by the upstream model provider. type Usage struct { - InputTokens int `json:"input_tokens,omitempty" yaml:"input_tokens,omitempty"` - OutputTokens int `json:"output_tokens,omitempty" yaml:"output_tokens,omitempty"` - TotalTokens int `json:"total_tokens,omitempty" yaml:"total_tokens,omitempty"` + InputTokens int `json:"input_tokens,omitempty" yaml:"input_tokens,omitempty"` + OutputTokens int `json:"output_tokens,omitempty" yaml:"output_tokens,omitempty"` + TotalTokens int `json:"total_tokens,omitempty" yaml:"total_tokens,omitempty"` + CacheCreationInputTokens int `json:"cache_creation_input_tokens,omitempty" yaml:"cache_creation_input_tokens,omitempty"` + CacheReadInputTokens int `json:"cache_read_input_tokens,omitempty" yaml:"cache_read_input_tokens,omitempty"` } // NewCassette creates an initialized, schema-compliant Cassette instance. @@ -109,13 +114,120 @@ func NewCassette(cassetteID string) *Cassette { Metadata: Metadata{ CassetteID: cassetteID, RecordedAt: time.Now().UTC(), - Generator: "AgentFixture v1.0.0", + Generator: "AgentFixture v1.1.0", Tags: make(map[string]string), }, Interactions: make([]Interaction, 0), } } +// Clone returns a deep copy of the Cassette safe for concurrent serialization or mutation. +func (c *Cassette) Clone() *Cassette { + if c == nil { + return nil + } + out := &Cassette{ + Version: c.Version, + Metadata: c.Metadata, + } + if c.Metadata.Tags != nil { + out.Metadata.Tags = cloneStringMap(c.Metadata.Tags) + } + if c.Interactions != nil { + out.Interactions = make([]Interaction, len(c.Interactions)) + for i := range c.Interactions { + out.Interactions[i] = c.Interactions[i].Clone() + } + } + return out +} + +// Clone returns a deep copy of an Interaction. +func (inter Interaction) Clone() Interaction { + cp := inter + cp.Metadata = cloneStringMap(inter.Metadata) + cp.Request = inter.Request.Clone() + cp.Response = inter.Response.Clone() + return cp +} + +// Clone returns a deep copy of a Request. +func (r Request) Clone() Request { + cp := r + cp.Headers = cloneStringMap(r.Headers) + cp.Body = cloneAny(r.Body) + cp.ToolInput = cloneAnyMap(r.ToolInput) + cp.Matcher.FuzzySignature.NormalizedArgs = cloneAnyMap(r.Matcher.FuzzySignature.NormalizedArgs) + return cp +} + +// Clone returns a deep copy of a Response. +func (r Response) Clone() Response { + cp := r + cp.Headers = cloneStringMap(r.Headers) + cp.Body = cloneAny(r.Body) + if r.SSEEvents != nil { + cp.SSEEvents = make([]SSEEvent, len(r.SSEEvents)) + copy(cp.SSEEvents, r.SSEEvents) + } + return cp +} + +func cloneStringMap(m map[string]string) map[string]string { + if m == nil { + return nil + } + cp := make(map[string]string, len(m)) + for k, v := range m { + cp[k] = v + } + return cp +} + +func cloneAnyMap(m map[string]interface{}) map[string]interface{} { + if m == nil { + return nil + } + cp := make(map[string]interface{}, len(m)) + for k, v := range m { + cp[k] = cloneAny(v) + } + return cp +} + +func cloneAny(v interface{}) interface{} { + if v == nil { + return nil + } + switch val := v.(type) { + case map[string]interface{}: + return cloneAnyMap(val) + case []interface{}: + cp := make([]interface{}, len(val)) + for i, el := range val { + cp[i] = cloneAny(el) + } + return cp + case map[string]string: + return cloneStringMap(val) + default: + // For custom structs/slices, fallback via JSON if needed; primitives copy by value. + switch val.(type) { + case string, bool, int, int64, int32, float64, float32, uint, uint64, json.Number: + return val + } + b, err := json.Marshal(val) + if err != nil { + return val + } + var cloned interface{} + if err := json.Unmarshal(b, &cloned); err != nil { + return val + } + return cloned + } +} + // Validate ensures cassette structure and required fields comply with agentfixture/v1. func (c *Cassette) Validate() error { if c.Version == "" { @@ -138,24 +250,68 @@ func (c *Cassette) Validate() error { if inter.Type == "" { return fmt.Errorf("interaction '%s' missing type", inter.ID) } + for j, ev := range inter.Response.SSEEvents { + if ev.Index < 0 { + return fmt.Errorf("interaction '%s' sse_event at index %d has negative index %d", inter.ID, j, ev.Index) + } + } } return nil } // ComputeMetrics recalculates interaction totals and estimated savings. -// Estimates cost based on Anthropic Claude Sonnet standard rates ($3/M in, $15/M out). +// Estimates cost based on Anthropic Claude Sonnet standard rates ($3/M in, $15/M out, +// $3.75/M cache write, $0.30/M cache read) when input/output usage is present, or a +// blended $6/M rate when only aggregate or heuristic token counts are available. func (c *Cassette) ComputeMetrics() { var totalTokens int64 var totalLatency int64 + var totalCost float64 + for _, inter := range c.Interactions { - totalLatency += inter.Response.LatencyMs - if inter.Response.Usage.TotalTokens > 0 { - totalTokens += int64(inter.Response.Usage.TotalTokens) + if inter.Response.LatencyMs > 0 { + totalLatency += inter.Response.LatencyMs + } else if len(inter.Response.SSEEvents) > 0 { + for _, ev := range inter.Response.SSEEvents { + totalLatency += ev.DeltaMs + } + } + + u := inter.Response.Usage + if u.TotalTokens == 0 && u.InputTokens == 0 && u.OutputTokens == 0 && + u.CacheCreationInputTokens == 0 && u.CacheReadInputTokens == 0 && + len(inter.Response.SSEEvents) > 0 { + u = ReconstructStreamSummary(inter.Response.SSEEvents).Usage + } + + hasGranular := u.InputTokens > 0 || u.OutputTokens > 0 || + u.CacheCreationInputTokens > 0 || u.CacheReadInputTokens > 0 + + if hasGranular { + tok := u.TotalTokens + if tok == 0 { + tok = u.InputTokens + u.OutputTokens + u.CacheCreationInputTokens + u.CacheReadInputTokens + } + totalTokens += int64(tok) + totalCost += float64(u.InputTokens)*0.000003 + + float64(u.OutputTokens)*0.000015 + + float64(u.CacheCreationInputTokens)*0.00000375 + + float64(u.CacheReadInputTokens)*0.0000003 + } else if u.TotalTokens > 0 { + totalTokens += int64(u.TotalTokens) + totalCost += float64(u.TotalTokens) * 0.000006 } else { // Heuristic estimation: ~4 chars per token for text bodyLen := len(inter.Request.RawBody) + len(inter.Response.RawBody) + len(inter.Response.ToolResult) + if len(inter.Response.SSEEvents) > 0 && inter.Response.RawBody == "" { + for _, ev := range inter.Response.SSEEvents { + bodyLen += len(ev.Data) + } + } if bodyLen > 0 { - totalTokens += int64(bodyLen / 4) + est := int64(bodyLen / 4) + totalTokens += est + totalCost += float64(est) * 0.000006 } } } @@ -163,6 +319,5 @@ func (c *Cassette) ComputeMetrics() { c.Metadata.Metrics.TotalInteractions = len(c.Interactions) c.Metadata.Metrics.RecordedDurationMs = totalLatency c.Metadata.Metrics.EstimatedTokensSaved = totalTokens - // $3 per million input tokens, $15 per million output tokens -> avg approx $6 per million tokens - c.Metadata.Metrics.EstimatedCostSavedUSD = float64(totalTokens) * 0.000006 + c.Metadata.Metrics.EstimatedCostSavedUSD = totalCost } diff --git a/pkg/importer/traceport.go b/pkg/importer/traceport.go index c134f72..8110107 100644 --- a/pkg/importer/traceport.go +++ b/pkg/importer/traceport.go @@ -1,8 +1,11 @@ package importer import ( + "bufio" + "bytes" "encoding/json" "fmt" + "io" "os" "strings" "time" @@ -52,6 +55,7 @@ type TracePortSession struct { } // ImportTracePortSession converts raw TracePort flight recording JSON bytes into a clean Cassette. +// Redacts all secrets BEFORE computing deterministic StrictHash values so hashes never leak secret oracles. func ImportTracePortSession(sessionJSON []byte) (*cassette.Cassette, error) { var sess TracePortSession if err := json.Unmarshal(sessionJSON, &sess); err != nil { @@ -63,59 +67,73 @@ func ImportTracePortSession(sessionJSON []byte) (*cassette.Cassette, error) { casID = fmt.Sprintf("trace_%d", time.Now().UnixNano()) } + redactedCommand := cassette.RedactString(sess.Command) + cas := cassette.NewCassette(casID) cas.Metadata.RecordedAt = sess.StartTime cas.Metadata.AgentName = "traceport-agent" cas.Metadata.Source = "traceport_flight_recording" - cas.Metadata.Description = fmt.Sprintf("Imported from TracePort flight recording (Command: %s, Status: %s)", sess.Command, sess.Status) + cas.Metadata.Description = fmt.Sprintf("Imported from TracePort flight recording (Command: %s, Status: %s)", redactedCommand, sess.Status) if cas.Metadata.Tags == nil { cas.Metadata.Tags = make(map[string]string) } cas.Metadata.Tags["traceport_session_id"] = sess.ID - cas.Metadata.Tags["command"] = sess.Command + cas.Metadata.Tags["command"] = redactedCommand cas.Metadata.Tags["status"] = sess.Status - // Map frames into interactions seq := 1 n := len(sess.Frames) for i := 0; i < n; i++ { frame := sess.Frames[i] - // Record initial workspace checkpoint if available + // Record workspace checkpoint if available if frame.Type == EventCheckpoint && frame.GitSnapshot != "" { cas.Metadata.Tags["git_snapshot"] = frame.GitSnapshot continue } - // Handle tool calls + // 1. Handle tool calls (paired with optional tool_result or error frame) if frame.Type == EventToolCall && frame.Tool != nil { toolName := frame.Tool.Name - toolArgs := frame.Tool.Arguments - if toolArgs == nil { - toolArgs = make(map[string]interface{}) + rawArgs := frame.Tool.Arguments + if rawArgs == nil { + rawArgs = make(map[string]interface{}) } + redactedArgs := cassette.RedactMap(rawArgs) - // Check if result is directly present or in a subsequent tool_result frame - toolResult := frame.Tool.Result + toolResult := cassette.RedactString(frame.Tool.Result) + var errText string + statusCode := 0 latency := frame.Tool.DurationMs if toolResult == "" && i+1 < n { next := sess.Frames[i+1] - if next.Type == EventToolResult { + switch next.Type { + case EventToolResult: if next.Tool != nil && next.Tool.Result != "" { - toolResult = next.Tool.Result + toolResult = cassette.RedactString(next.Tool.Result) if next.Tool.DurationMs > 0 { latency = next.Tool.DurationMs } } else if next.Content != "" { - toolResult = next.Content + toolResult = cassette.RedactString(next.Content) + } + i++ + case EventError: + errText = cassette.RedactString(next.Content) + statusCode = 500 + if latency == 0 && !next.Timestamp.IsZero() && !frame.Timestamp.IsZero() { + if d := next.Timestamp.Sub(frame.Timestamp).Milliseconds(); d > 0 { + latency = d + } } + i++ } } - strictHash := matcher.ComputeToolHash(toolName, toolArgs) + strictHash := matcher.ComputeToolHash(toolName, redactedArgs) inter := cassette.Interaction{ ID: fmt.Sprintf("tp_step_%d", frame.Index), @@ -123,19 +141,20 @@ func ImportTracePortSession(sessionJSON []byte) (*cassette.Cassette, error) { Type: cassette.InteractionTypeTool, Request: cassette.Request{ ToolName: toolName, - ToolInput: toolArgs, + ToolInput: redactedArgs, Matcher: cassette.MatcherMeta{ StrictHash: strictHash, FuzzySignature: cassette.FuzzySignature{ NormalizedToolName: strings.ToLower(toolName), - NormalizedArgs: toolArgs, + NormalizedArgs: redactedArgs, }, }, }, Response: cassette.Response{ - StatusCode: 0, + StatusCode: statusCode, LatencyMs: latency, ToolResult: toolResult, + Error: errText, }, Metadata: map[string]string{ "traceport_frame_index": fmt.Sprintf("%d", frame.Index), @@ -149,51 +168,121 @@ func ImportTracePortSession(sessionJSON []byte) (*cassette.Cassette, error) { cas.Interactions = append(cas.Interactions, inter) seq++ - } else if frame.Type == EventInput && i+1 < n && sess.Frames[i+1].Type == EventOutput { - // Paired user input / assistant output - outputFrame := sess.Frames[i+1] + continue + } + + // 2. Paired user input / assistant output (or error response) + if frame.Type == EventInput && i+1 < n && (sess.Frames[i+1].Type == EventOutput || sess.Frames[i+1].Type == EventError) { + nextFrame := sess.Frames[i+1] + redactedInput := cassette.RedactString(frame.Content) + redactedNext := cassette.RedactString(nextFrame.Content) + + req := cassette.Request{ + RawBody: redactedInput, + } + req.Matcher = cassette.MatcherMeta{ + StrictHash: matcher.ComputeRequestHash(&req), + FuzzySignature: cassette.FuzzySignature{ + NormalizedPrompt: matcher.NormalizeText(redactedInput), + }, + } + + resp := cassette.Response{ + StatusCode: 200, + RawBody: redactedNext, + LatencyMs: nextFrame.Timestamp.Sub(frame.Timestamp).Milliseconds(), + } + if nextFrame.Type == EventError { + resp.StatusCode = 500 + resp.RawBody = "" + resp.Error = redactedNext + } + inter := cassette.Interaction{ ID: fmt.Sprintf("tp_turn_%d", frame.Index), Sequence: seq, Type: cassette.InteractionTypeLLMChat, - Request: cassette.Request{ - RawBody: frame.Content, + Request: req, + Response: resp, + Metadata: map[string]string{ + "input_index": fmt.Sprintf("%d", frame.Index), + "output_index": fmt.Sprintf("%d", nextFrame.Index), + }, + } + cas.Interactions = append(cas.Interactions, inter) + seq++ + i++ + continue + } + + // 3. Standalone EventError frame (never dropped) + if frame.Type == EventError { + redactedErr := cassette.RedactString(frame.Content) + interType := cassette.InteractionTypeLLMChat + var req cassette.Request + + if frame.Tool != nil && frame.Tool.Name != "" { + interType = cassette.InteractionTypeTool + redArgs := cassette.RedactMap(frame.Tool.Arguments) + if redArgs == nil { + redArgs = make(map[string]interface{}) + } + req = cassette.Request{ + ToolName: frame.Tool.Name, + ToolInput: redArgs, Matcher: cassette.MatcherMeta{ - StrictHash: matcher.CanonicalHash([]byte(frame.Content)), + StrictHash: matcher.ComputeToolHash(frame.Tool.Name, redArgs), FuzzySignature: cassette.FuzzySignature{ - NormalizedPrompt: matcher.NormalizeText(frame.Content), + NormalizedToolName: strings.ToLower(frame.Tool.Name), + NormalizedArgs: redArgs, }, }, - }, + } + } else { + req = cassette.Request{ + RawBody: redactedErr, + } + req.Matcher = cassette.MatcherMeta{ + StrictHash: matcher.ComputeRequestHash(&req), + FuzzySignature: cassette.FuzzySignature{ + NormalizedPrompt: matcher.NormalizeText(redactedErr), + }, + } + } + + inter := cassette.Interaction{ + ID: fmt.Sprintf("tp_err_%d", frame.Index), + Sequence: seq, + Type: interType, + Request: req, Response: cassette.Response{ - StatusCode: 200, - RawBody: outputFrame.Content, - LatencyMs: outputFrame.Timestamp.Sub(frame.Timestamp).Milliseconds(), + StatusCode: 500, + Error: redactedErr, }, Metadata: map[string]string{ - "input_index": fmt.Sprintf("%d", frame.Index), - "output_index": fmt.Sprintf("%d", outputFrame.Index), + "traceport_frame_index": fmt.Sprintf("%d", frame.Index), + "event_type": string(EventError), + "timestamp": frame.Timestamp.Format(time.RFC3339Nano), }, } cas.Interactions = append(cas.Interactions, inter) seq++ - i++ // skip the paired output frame } } - // Redact sensitive secrets from flight log interactions cassette.RedactCassette(cas) cas.ComputeMetrics() return cas, nil } -// ImportTracePortFile loads a TracePort session JSON file from disk, converts it, -// and saves the resulting cassette to the requested output path (YAML or JSON). +// ImportTracePortFile loads a TracePort session from disk (supporting legacy monolithic .json, +// `traceport inspect --json` output, and TracePort v1.1+ split .meta.json + .frames.jsonl WAL logs), +// converts it, and saves the resulting cassette to outputPath (YAML or JSON). func ImportTracePortFile(sessionPath, outputPath string) (*cassette.Cassette, error) { - data, err := os.ReadFile(sessionPath) + data, err := resolveTracePortSessionJSON(sessionPath) if err != nil { - return nil, fmt.Errorf("failed to read traceport session file: %w", err) + return nil, err } cas, err := ImportTracePortSession(data) @@ -209,3 +298,68 @@ func ImportTracePortFile(sessionPath, outputPath string) (*cassette.Cassette, er return cas, nil } + +func resolveTracePortSessionJSON(sessionPath string) ([]byte, error) { + switch { + case strings.HasSuffix(sessionPath, ".meta.json"): + prefix := strings.TrimSuffix(sessionPath, ".meta.json") + return assembleSplitTracePortSession(sessionPath, prefix+".frames.jsonl") + case strings.HasSuffix(sessionPath, ".frames.jsonl"): + prefix := strings.TrimSuffix(sessionPath, ".frames.jsonl") + return assembleSplitTracePortSession(prefix+".meta.json", sessionPath) + } + + data, err := os.ReadFile(sessionPath) + if err == nil { + return data, nil + } + + // Fallback: check if TracePort v1.1+ split files (.meta.json + .frames.jsonl) exist. + prefix := strings.TrimSuffix(sessionPath, ".json") + if _, mErr := os.Stat(prefix + ".meta.json"); mErr == nil { + return assembleSplitTracePortSession(prefix+".meta.json", prefix+".frames.jsonl") + } + + return nil, fmt.Errorf("failed to read traceport session file: %w", err) +} + +func assembleSplitTracePortSession(metaPath, framesPath string) ([]byte, error) { + var sess TracePortSession + if metaBytes, err := os.ReadFile(metaPath); err == nil { + if err := json.Unmarshal(metaBytes, &sess); err != nil { + return nil, fmt.Errorf("failed to parse traceport meta file %s: %w", metaPath, err) + } + } else if !os.IsNotExist(err) { + return nil, fmt.Errorf("failed to read traceport meta file %s: %w", metaPath, err) + } + + f, err := os.Open(framesPath) + if err != nil { + if os.IsNotExist(err) && sess.ID != "" { + return json.Marshal(sess) + } + return nil, fmt.Errorf("failed to read traceport frames file %s: %w", framesPath, err) + } + defer f.Close() + + reader := bufio.NewReaderSize(f, 64*1024) + for { + line, readErr := reader.ReadBytes('\n') + trimmed := bytes.TrimSpace(line) + if len(trimmed) > 0 { + var frame TracePortFrame + if err := json.Unmarshal(trimmed, &frame); err != nil { + return nil, fmt.Errorf("invalid frame line in %s: %w", framesPath, err) + } + sess.Frames = append(sess.Frames, frame) + } + if readErr == io.EOF { + break + } + if readErr != nil { + return nil, fmt.Errorf("error reading %s: %w", framesPath, readErr) + } + } + + return json.Marshal(sess) +} diff --git a/pkg/importer/traceport_test.go b/pkg/importer/traceport_test.go index 13ad100..26f884f 100644 --- a/pkg/importer/traceport_test.go +++ b/pkg/importer/traceport_test.go @@ -7,6 +7,7 @@ import ( "testing" "github.com/aloneangel45/AgentFixture/pkg/cassette" + "github.com/aloneangel45/AgentFixture/pkg/matcher" ) const sampleTracePortJSON = `{ @@ -104,10 +105,23 @@ func TestImportTracePortSession(t *testing.T) { t.Errorf("expected tool result to contain auth package, got %s", toolTurn.Response.ToolResult) } - // Secret masking verification + // Secret masking & zero secret hash oracle verification apiKeyValue := toolTurn.Request.ToolInput["api_key"] - if apiKeyValue != "[REDACTED]" && apiKeyValue != "[REDACTED_ANTHROPIC_KEY]" { - t.Errorf("expected api_key argument to be redacted, got %v", apiKeyValue) + if apiKeyValue != "[REDACTED]" { + t.Errorf("expected api_key argument to be [REDACTED], got %v", apiKeyValue) + } + + // Replaying with a dummy CI key must match StrictHash 100%! + strict := matcher.NewStrictMatcher() + replayCandidate := &cassette.Request{ + ToolName: "view_file", + ToolInput: map[string]interface{}{ + "path": "pkg/auth/auth.go", + "api_key": "ci-dummy-placeholder-key", + }, + } + if res := strict.Match(&toolTurn.Request, replayCandidate); !res.Matched { + t.Fatalf("StrictHash oracle leak / mismatch when replaying with dummy CI key: %s", res.Diff) } // Metrics @@ -116,6 +130,52 @@ func TestImportTracePortSession(t *testing.T) { } } +func TestImportTracePortSession_EventErrorFrames(t *testing.T) { + errSession := `{ + "id": "trace_errors_01", + "command": "agent --api-key=sk-ant-api03-1234567890abcdef1234567890", + "start_time": "2026-11-10T10:00:00Z", + "status": "failed", + "frames": [ + { + "index": 1, + "timestamp": "2026-11-10T10:00:01Z", + "type": "tool_call", + "tool": {"name": "exec_cmd", "arguments": {"cmd": "false"}} + }, + { + "index": 2, + "timestamp": "2026-11-10T10:00:02Z", + "type": "error", + "content": "exit status 1: permission denied" + }, + { + "index": 3, + "timestamp": "2026-11-10T10:00:03Z", + "type": "error", + "content": "upstream rate limit exceeded (token sk-ant-api03-1234567890abcdef1234567890)" + } + ] + }` + + cas, err := ImportTracePortSession([]byte(errSession)) + if err != nil { + t.Fatalf("ImportTracePortSession failed: %v", err) + } + if len(cas.Interactions) != 2 { + t.Fatalf("expected 2 interactions (paired tool error + standalone error), got %d", len(cas.Interactions)) + } + if cas.Interactions[0].Response.StatusCode != 500 || !strings.Contains(cas.Interactions[0].Response.Error, "permission denied") { + t.Errorf("unexpected paired tool error interaction: %+v", cas.Interactions[0].Response) + } + if cas.Interactions[1].ID != "tp_err_3" || cas.Interactions[1].Response.StatusCode != 500 { + t.Errorf("unexpected standalone error interaction: %+v", cas.Interactions[1]) + } + if hasSec, reason := cassette.CheckForUnredactedSecrets(cas); hasSec { + t.Fatalf("unredacted secret found in error session import: %s", reason) + } +} + func TestImportTracePortFile(t *testing.T) { tmpDir := t.TempDir() sessionFile := filepath.Join(tmpDir, "session.json") @@ -143,3 +203,41 @@ func TestImportTracePortFile(t *testing.T) { t.Errorf("expected 2 interactions in reloaded cassette, got %d", len(loaded.Interactions)) } } + +func TestImportTracePortFile_SplitV11MetaAndJSONL(t *testing.T) { + tmpDir := t.TempDir() + metaPath := filepath.Join(tmpDir, "trace_v11_01.meta.json") + framesPath := filepath.Join(tmpDir, "trace_v11_01.frames.jsonl") + outPath := filepath.Join(tmpDir, "v11.yaml") + + metaJSON := `{"id":"trace_v11_01","command":"claude --api-key=sk-ant-api03-secret999999999999","start_time":"2026-11-10T12:00:00Z","status":"completed","frame_count":3}` + framesJSONL := strings.Join([]string{ + `{"index":1,"timestamp":"2026-11-10T12:00:01Z","type":"checkpoint","content":"Initial flight snapshot","git_snapshot":"zstd:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}`, + `{"index":2,"timestamp":"2026-11-10T12:00:02Z","type":"tool_call","tool":{"name":"read_file","arguments":{"path":"main.go"}}}`, + `{"index":3,"timestamp":"2026-11-10T12:00:03Z","type":"tool_result","content":"package main"}`, + }, "\n") + "\n" + + if err := os.WriteFile(metaPath, []byte(metaJSON), 0600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(framesPath, []byte(framesJSONL), 0600); err != nil { + t.Fatal(err) + } + + // Test importing via .meta.json, .frames.jsonl, and bare session prefix + for _, inputPath := range []string{metaPath, framesPath, filepath.Join(tmpDir, "trace_v11_01")} { + cas, err := ImportTracePortFile(inputPath, outPath) + if err != nil { + t.Fatalf("ImportTracePortFile(%s) failed: %v", inputPath, err) + } + if len(cas.Interactions) != 1 || cas.Interactions[0].Request.ToolName != "read_file" { + t.Fatalf("unexpected imported interactions from %s: %+v", inputPath, cas.Interactions) + } + if !strings.HasPrefix(cas.Metadata.Tags["git_snapshot"], "zstd:") { + t.Errorf("expected zstd checkpoint tag preserved, got %q", cas.Metadata.Tags["git_snapshot"]) + } + if hasSec, reason := cassette.CheckForUnredactedSecrets(cas); hasSec { + t.Fatalf("unredacted secret in split v1.1 import: %s", reason) + } + } +} diff --git a/pkg/matcher/fuzzy.go b/pkg/matcher/fuzzy.go index c6f0ddc..06ee1e1 100644 --- a/pkg/matcher/fuzzy.go +++ b/pkg/matcher/fuzzy.go @@ -1,15 +1,25 @@ package matcher import ( + "bytes" "fmt" "math" + "net/url" "reflect" "regexp" "strings" + "unicode" + "unicode/utf8" "github.com/aloneangel45/AgentFixture/pkg/cassette" ) +// maxExactLevenshteinRunes caps O(N*M) dynamic programming edit distance so large prompts +// (e.g. 100KB+ context windows) fall back to O(N) hashed 4-gram Dice similarity in <1ms. +const maxExactLevenshteinRunes = 4096 + +// dynamicKeys lists volatile metadata keys stripped during fuzzy matching. +// Bare "id" is intentionally excluded so distinct entity IDs (e.g. user id=1 vs id=2) do not collide. var dynamicKeys = map[string]bool{ "timestamp": true, "time": true, @@ -17,7 +27,10 @@ var dynamicKeys = map[string]bool{ "nonce": true, "request_id": true, "req_id": true, - "id": true, + "trace_id": true, + "span_id": true, + "correlation_id": true, + "x_request_id": true, "uuid": true, "session_id": true, "client_timestamp": true, @@ -27,12 +40,11 @@ var dynamicKeys = map[string]bool{ "updated_at": true, } -var multiSpaceRegex = regexp.MustCompile(`\s+`) var isoDateRegex = regexp.MustCompile(`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})?`) var uuidRegex = regexp.MustCompile(`[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}`) // FuzzyMatcher performs semantic matching with whitespace normalization, -// case insensitivity, dynamic key exclusion, and Levenshtein similarity. +// case insensitivity, dynamic key exclusion, JSON canonicalization, and similarity scoring. type FuzzyMatcher struct { Threshold float64 } @@ -75,15 +87,13 @@ func (f *FuzzyMatcher) matchTool(expected, actual *cassette.Request) MatchResult } } - // Clean dynamic keys from tool arguments - expArgs := filterDynamicKeys(expected.ToolInput) - actArgs := filterDynamicKeys(actual.ToolInput) + expArgs := filterDynamicKeys(cassette.RedactMap(expected.ToolInput)) + actArgs := filterDynamicKeys(cassette.RedactMap(actual.ToolInput)) if reflect.DeepEqual(expArgs, actArgs) { return MatchResult{Matched: true, Score: 1.0} } - // Compute similarity score across arguments score, diff := compareFilteredArgs(expArgs, actArgs) matched := score >= f.Threshold return MatchResult{ @@ -112,8 +122,34 @@ func (f *FuzzyMatcher) matchHTTP(expected, actual *cassette.Request) MatchResult } } - expBody := NormalizeText(expected.RawBody) - actBody := NormalizeText(actual.RawBody) + // Attempt structured JSON normalization first (strips dynamic keys, normalizes strings, ignores key order) + expJSON, expOK := extractStructuredJSON(expected) + actJSON, actOK := extractStructuredJSON(actual) + if expOK && actOK { + fExp := filterDynamicValue(cassette.RedactValue(expJSON)) + fAct := filterDynamicValue(cassette.RedactValue(actJSON)) + cbExp, err1 := CanonicalJSON(fExp) + cbAct, err2 := CanonicalJSON(fAct) + if err1 == nil && err2 == nil { + if bytes.Equal(cbExp, cbAct) { + return MatchResult{Matched: true, Score: 1.0} + } + score := StringSimilarity(string(cbExp), string(cbAct)) + matched := score >= f.Threshold + var diff string + if !matched { + diff = fmt.Sprintf("semantic JSON body similarity %.2f below threshold %.2f", score, f.Threshold) + } + return MatchResult{ + Matched: matched, + Score: score, + Diff: diff, + } + } + } + + expBody := NormalizeText(cassette.RedactString(expected.RawBody)) + actBody := NormalizeText(cassette.RedactString(actual.RawBody)) if expBody == actBody { return MatchResult{Matched: true, Score: 1.0} @@ -134,21 +170,203 @@ func (f *FuzzyMatcher) matchHTTP(expected, actual *cassette.Request) MatchResult } } -// NormalizeText strips dynamic dates, UUIDs, collapses whitespace, and trims. +func extractStructuredJSON(req *cassette.Request) (interface{}, bool) { + rawTrimmed := strings.TrimSpace(req.RawBody) + if len(rawTrimmed) > 0 && (rawTrimmed[0] == '{' || rawTrimmed[0] == '[') { + if parsed, err := DecodeJSONUseNumber([]byte(rawTrimmed)); err == nil { + switch parsed.(type) { + case map[string]interface{}, []interface{}: + return parsed, true + } + } + } + if req.Body != nil { + switch req.Body.(type) { + case map[string]interface{}, []interface{}: + return req.Body, true + } + } + return nil, false +} + +// NormalizeText strips dynamic ISO-8601 dates and UUIDs, collapses runs of whitespace, and trims. func NormalizeText(s string) string { if s == "" { return "" } - res := isoDateRegex.ReplaceAllString(s, "[TIMESTAMP]") - res = uuidRegex.ReplaceAllString(res, "[UUID]") - res = multiSpaceRegex.ReplaceAllString(res, " ") - return strings.TrimSpace(res) + hasISO, hasUUID := scanDynamicPatterns(s) + res := s + if hasISO { + res = isoDateRegex.ReplaceAllString(res, "[TIMESTAMP]") + } + if hasUUID { + res = uuidRegex.ReplaceAllString(res, "[UUID]") + } + return collapseWhitespace(res) +} + +// scanDynamicPatterns uses SIMD IndexByte jumps over 'T' and '-' to detect structural +// ISO-8601 timestamps (YYYY-MM-DDTHH:MM:SS) or UUIDs (8-4-4-4-12) before invoking regexp. +func scanDynamicPatterns(s string) (hasISO, hasUUID bool) { + n := len(s) + if n < 19 { + return false, false + } + for off := 10; off+8 < n; { + idx := strings.IndexByte(s[off:n-8], 'T') + if idx < 0 { + break + } + p := off + idx + if s[p-3] == '-' && s[p-6] == '-' && s[p+3] == ':' && s[p+6] == ':' { + hasISO = true + break + } + off = p + 1 + } + if n >= 36 { + for off := 8; off+27 < n; { + idx := strings.IndexByte(s[off:n-27], '-') + if idx < 0 { + break + } + p := off + idx + if s[p+5] == '-' && s[p+10] == '-' && s[p+15] == '-' { + hasUUID = true + break + } + off = p + 1 + } + } + return hasISO, hasUUID +} + +func trimASCIISpace(s string) string { + start := 0 + for start < len(s) { + c := s[start] + if c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\v' || c == '\f' { + start++ + } else { + break + } + } + end := len(s) + for end > start { + c := s[end-1] + if c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\v' || c == '\f' { + end-- + } else { + break + } + } + return s[start:end] +} + +func collapseWhitespace(s string) string { + s = trimASCIISpace(s) + if s == "" { + return "" + } + // Fast scan: if already normalized single-space ASCII, return zero-copy slice. + needsCopy := false + prevSpace := false + for i := 0; i < len(s); i++ { + c := s[i] + if c >= 0x80 || c == '\t' || c == '\n' || c == '\r' || c == '\v' || c == '\f' { + needsCopy = true + break + } + if c == ' ' { + if prevSpace { + needsCopy = true + break + } + prevSpace = true + } else { + prevSpace = false + } + } + if !needsCopy { + return s + } + + var b strings.Builder + b.Grow(len(s)) + inSpace := true // trim leading Unicode whitespace if present + segStart := -1 + for i := 0; i < len(s); { + c := s[i] + if c < 0x80 { + if c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\v' || c == '\f' { + if segStart >= 0 { + b.WriteString(s[segStart:i]) + segStart = -1 + } + if !inSpace { + b.WriteByte(' ') + inSpace = true + } + } else { + if segStart < 0 { + segStart = i + } + inSpace = false + } + i++ + continue + } + if segStart >= 0 { + b.WriteString(s[segStart:i]) + segStart = -1 + } + r, width := utf8.DecodeRuneInString(s[i:]) + if unicode.IsSpace(r) { + if !inSpace { + b.WriteByte(' ') + inSpace = true + } + } else { + b.WriteRune(r) + inSpace = false + } + i += width + } + if segStart >= 0 { + b.WriteString(s[segStart:]) + } + out := b.String() + if len(out) > 0 && out[len(out)-1] == ' ' { + return out[:len(out)-1] + } + return out } func normalizeEndpoint(endpoint string) string { - parts := strings.SplitN(endpoint, "?", 2) - path := strings.TrimRight(parts[0], "/") - return strings.ToLower(path) + norm := NormalizeEndpoint(endpoint) + if norm == "" { + return "" + } + qIdx := strings.IndexByte(norm, '?') + if qIdx == -1 { + return strings.ToLower(norm) + } + pathPart := strings.ToLower(norm[:qIdx]) + rawQuery := norm[qIdx+1:] + vals, err := url.ParseQuery(rawQuery) + if err != nil { + return strings.ToLower(norm) + } + for k := range vals { + if dynamicKeys[strings.ToLower(k)] { + vals.Del(k) + } + } + encoded := vals.Encode() + if encoded == "" { + return pathPart + } + return pathPart + "?" + strings.ToLower(encoded) } func filterDynamicKeys(m map[string]interface{}) map[string]interface{} { @@ -157,22 +375,32 @@ func filterDynamicKeys(m map[string]interface{}) map[string]interface{} { } res := make(map[string]interface{}, len(m)) for k, v := range m { - lower := strings.ToLower(k) + lower := strings.ToLower(strings.TrimSpace(k)) if dynamicKeys[lower] { continue } - switch val := v.(type) { - case map[string]interface{}: - res[k] = filterDynamicKeys(val) - case string: - res[k] = NormalizeText(val) - default: - res[k] = val - } + res[k] = filterDynamicValue(v) } return res } +func filterDynamicValue(v interface{}) interface{} { + switch val := v.(type) { + case map[string]interface{}: + return filterDynamicKeys(val) + case []interface{}: + items := make([]interface{}, len(val)) + for i, el := range val { + items[i] = filterDynamicValue(el) + } + return items + case string: + return NormalizeText(val) + default: + return canonicalizeValue(val) + } +} + func compareFilteredArgs(exp, act map[string]interface{}) (float64, string) { allKeys := make(map[string]bool) for k := range exp { @@ -221,30 +449,128 @@ func compareFilteredArgs(exp, act map[string]interface{}) (float64, string) { return score, strings.Join(diffs, "; ") } -// StringSimilarity calculates a normalized similarity ratio (0.0 to 1.0) using Levenshtein distance. +// StringSimilarity calculates a normalized UTF-8 rune similarity ratio (0.0 to 1.0). +// Strips common prefixes and suffixes zero-copy, uses exact Levenshtein distance up to 4,096 runes, +// and switches to zero-allocation O(N) hashed 4-gram Dice similarity on large payloads (>4,096 differing runes). func StringSimilarity(s1, s2 string) float64 { if s1 == s2 { return 1.0 } - len1 := len(s1) - len2 := len(s2) - if len1 == 0 || len2 == 0 { + if len(s1) == 0 || len(s2) == 0 { + return 0.0 + } + + prefixStr, suffixStr, mid1, mid2 := trimCommonStringAffixes(s1, s2) + affixRunes := fastRuneCount(prefixStr) + fastRuneCount(suffixStr) + midRunes1 := fastRuneCount(mid1) + midRunes2 := fastRuneCount(mid2) + + n1 := affixRunes + midRunes1 + n2 := affixRunes + midRunes2 + if n1 == 0 || n2 == 0 { return 0.0 } - dist := LevenshteinDistance(s1, s2) - maxLen := math.Max(float64(len1), float64(len2)) - return 1.0 - (float64(dist) / maxLen) + maxRuneLen := float64(n1) + if n2 > n1 { + maxRuneLen = float64(n2) + } + + if midRunes1 == 0 && midRunes2 == 0 { + return 1.0 + } + if midRunes1 == 0 || midRunes2 == 0 { + dist := midRunes1 + midRunes2 + return clampUnit(1.0 - float64(dist)/maxRuneLen) + } + + if midRunes1 <= maxExactLevenshteinRunes && midRunes2 <= maxExactLevenshteinRunes { + dist := levenshteinRunes([]rune(mid1), []rune(mid2)) + return clampUnit(1.0 - float64(dist)/maxRuneLen) + } + + // Fast zero-allocation O(N) 4-gram Sørensen-Dice similarity for large (>4096 rune) interiors + midSim := ngramDiceStringSimilarity(mid1, mid2, midRunes1, midRunes2) + matchedEquiv := float64(affixRunes) + midSim*(float64(midRunes1+midRunes2)/2.0) + return clampUnit(matchedEquiv / maxRuneLen) } -// LevenshteinDistance calculates the edit distance between two strings. +// LevenshteinDistance calculates edit distance in UTF-8 runes, stripping shared prefix/suffix +// and capping DP work at 4,096 differing runes with fast 4-gram Dice estimation above that threshold. func LevenshteinDistance(s1, s2 string) int { - r1, r2 := []rune(s1), []rune(s2) - n, m := len(r1), len(r2) + if s1 == s2 { + return 0 + } + if len(s1) == 0 { + return utf8.RuneCountInString(s2) + } + if len(s2) == 0 { + return utf8.RuneCountInString(s1) + } - if n == 0 { - return m + _, _, mid1, mid2 := trimCommonStringAffixes(s1, s2) + midRunes1 := utf8.RuneCountInString(mid1) + midRunes2 := utf8.RuneCountInString(mid2) + if midRunes1 == 0 { + return midRunes2 } + if midRunes2 == 0 { + return midRunes1 + } + + if midRunes1 <= maxExactLevenshteinRunes && midRunes2 <= maxExactLevenshteinRunes { + return levenshteinRunes([]rune(mid1), []rune(mid2)) + } + + maxMid := midRunes1 + if midRunes2 > maxMid { + maxMid = midRunes2 + } + sim := ngramDiceStringSimilarity(mid1, mid2, midRunes1, midRunes2) + return int(math.Round((1.0 - sim) * float64(maxMid))) +} + +func trimCommonStringAffixes(s1, s2 string) (prefix, suffix, mid1, mid2 string) { + minBytes := len(s1) + if len(s2) < minBytes { + minBytes = len(s2) + } + p := 0 + for p < minBytes && s1[p] == s2[p] { + p++ + } + // Align back to UTF-8 rune start if split mid-character + if p < minBytes { + for p > 0 && !utf8.RuneStart(s1[p]) { + p-- + } + } + + rem1 := s1[p:] + rem2 := s2[p:] + remMin := len(rem1) + if len(rem2) < remMin { + remMin = len(rem2) + } + suf := 0 + for suf < remMin && rem1[len(rem1)-1-suf] == rem2[len(rem2)-1-suf] { + suf++ + } + if suf > 0 && suf < remMin { + for suf > 0 && !utf8.RuneStart(rem1[len(rem1)-suf]) { + suf-- + } + } + + return s1[:p], rem1[len(rem1)-suf:], rem1[:len(rem1)-suf], rem2[:len(rem2)-suf] +} + +func levenshteinRunes(r1, r2 []rune) int { + // Ensure r2 is the shorter slice to minimize row memory + if len(r1) < len(r2) { + r1, r2 = r2, r1 + } + n, m := len(r1), len(r2) if m == 0 { return n } @@ -256,16 +582,17 @@ func LevenshteinDistance(s1, s2 string) int { for i := 1; i <= n; i++ { prev := i + ri := r1[i-1] for j := 1; j <= m; j++ { var cost int - if r1[i-1] != r2[j-1] { + if ri != r2[j-1] { cost = 1 } - cur := row[j] + 1 // deletion - if prev+1 < cur { // insertion + cur := row[j] + 1 + if prev+1 < cur { cur = prev + 1 } - if row[j-1]+cost < cur { // substitution + if row[j-1]+cost < cur { cur = row[j-1] + cost } row[j-1] = prev @@ -273,6 +600,131 @@ func LevenshteinDistance(s1, s2 string) int { } row[m] = prev } - return row[m] } + +// ngramDiceStringSimilarity computes Sørensen-Dice similarity over 4-rune shingles using a fixed +// 16,384-bucket stack counting table with zero heap allocations (<0.6ms on 100KB prompts). +func ngramDiceStringSimilarity(s1, s2 string, runes1, runes2 int) float64 { + const k = 4 + if runes1 < k || runes2 < k { + dist := levenshteinRunes([]rune(s1), []rune(s2)) + mx := runes1 + if runes2 > mx { + mx = runes2 + } + if mx == 0 { + return 1.0 + } + return clampUnit(1.0 - float64(dist)/float64(mx)) + } + + const bucketBits = 12 + const bucketSize = 1 << bucketBits // 4096 (8KB stack array, fits cleanly in L1d cache) + const bucketShift = 32 - bucketBits + var counts [bucketSize]uint16 + + c1 := runes1 - k + 1 + c2 := runes2 - k + 1 + + if len(s1) == runes1 && len(s2) == runes2 { + // Content-defined mod-hash sampling (shift-invariant across arbitrary odd/even edits) + var sampleMask uint32 + if c1 > 8192 && c2 > 8192 { + sampleMask = 3 // sample ~1/4 of 4-grams deterministically by content + } + total1 := 0 + for i := 0; i < c1; i++ { + b := s1[i : i+4] + w := uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24 + hFull := w * 2654435761 + if (hFull & sampleMask) != 0 { + continue + } + h := hFull >> bucketShift + if counts[h] < 65535 { + counts[h]++ + } + total1++ + } + intersection := 0 + total2 := 0 + for i := 0; i < c2; i++ { + b := s2[i : i+4] + w := uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24 + hFull := w * 2654435761 + if (hFull & sampleMask) != 0 { + continue + } + h := hFull >> bucketShift + if counts[h] > 0 { + counts[h]-- + intersection++ + } + total2++ + } + if total1+total2 == 0 { + return 0.0 + } + return float64(2*intersection) / float64(total1+total2) + } + + const bucketMask = bucketSize - 1 + + var w0, w1, w2, w3 rune + idx := 0 + for _, r := range s1 { + w0, w1, w2, w3 = w1, w2, w3, r + idx++ + if idx >= k { + h := hash4Runes(w0, w1, w2, w3) & bucketMask + if counts[h] < 65535 { + counts[h]++ + } + } + } + + intersection := 0 + idx = 0 + for _, r := range s2 { + w0, w1, w2, w3 = w1, w2, w3, r + idx++ + if idx >= k { + h := hash4Runes(w0, w1, w2, w3) & bucketMask + if counts[h] > 0 { + counts[h]-- + intersection++ + } + } + } + + return float64(2*intersection) / float64(c1+c2) +} + +func fastRuneCount(s string) int { + for i := 0; i < len(s); i++ { + if s[i] >= 0x80 { + return utf8.RuneCountInString(s) + } + } + return len(s) +} + +func hash4Runes(a, b, c, d rune) uint32 { + h := uint32(2166136261) + h = (h ^ uint32(a)) * 16777619 + h = (h ^ uint32(b)) * 16777619 + h = (h ^ uint32(c)) * 16777619 + h = (h ^ uint32(d)) * 16777619 + return h ^ (h >> 15) +} + +func clampUnit(v float64) float64 { + if v < 0.0 { + return 0.0 + } + if v > 1.0 { + return 1.0 + } + return v +} diff --git a/pkg/matcher/matcher.go b/pkg/matcher/matcher.go index e334003..861568d 100644 --- a/pkg/matcher/matcher.go +++ b/pkg/matcher/matcher.go @@ -1,10 +1,16 @@ package matcher import ( + "bytes" "crypto/sha256" "encoding/hex" "encoding/json" + "fmt" + "io" + "math" + "net/url" "sort" + "strconv" "strings" "github.com/aloneangel45/AgentFixture/pkg/cassette" @@ -23,7 +29,28 @@ type Matcher interface { Match(expected, actual *cassette.Request) MatchResult } -// CanonicalJSON recursively sorts map keys and returns a deterministic JSON byte slice. +// DecodeJSONUseNumber parses JSON bytes using json.Decoder.UseNumber() so 64-bit integers +// never lose precision via IEEE-754 float64 conversion. +func DecodeJSONUseNumber(data []byte) (interface{}, error) { + trimmed := bytes.TrimSpace(data) + if len(trimmed) == 0 { + return nil, fmt.Errorf("empty json input") + } + dec := json.NewDecoder(bytes.NewReader(trimmed)) + dec.UseNumber() + var v interface{} + if err := dec.Decode(&v); err != nil { + return nil, err + } + var extra interface{} + if err := dec.Decode(&extra); err != io.EOF { + return nil, fmt.Errorf("trailing tokens in json input") + } + return v, nil +} + +// CanonicalJSON recursively sorts map keys, normalizes numbers without 64-bit precision loss, +// and returns a deterministic JSON byte slice. func CanonicalJSON(v interface{}) ([]byte, error) { norm := canonicalizeValue(v) return json.Marshal(norm) @@ -48,33 +75,119 @@ func canonicalizeValue(v interface{}) interface{} { items[i] = canonicalizeValue(el) } return items + case json.Number: + return normalizeJSONNumber(val) + case float64: + if !math.IsInf(val, 0) && !math.IsNaN(val) && math.Trunc(val) == val && + val >= -9007199254740992 && val <= 9007199254740992 { + return int64(val) + } + return val + case int: + return int64(val) + case int32: + return int64(val) default: return val } } +func normalizeJSONNumber(num json.Number) interface{} { + s := num.String() + if i, err := strconv.ParseInt(s, 10, 64); err == nil { + return i + } + if u, err := strconv.ParseUint(s, 10, 64); err == nil { + return u + } + if f, err := strconv.ParseFloat(s, 64); err == nil { + if !math.IsInf(f, 0) && !math.IsNaN(f) && math.Trunc(f) == f && + f >= -9007199254740992 && f <= 9007199254740992 { + return int64(f) + } + return f + } + return s +} + // CanonicalHash returns a sha256: hex string for arbitrary data. func CanonicalHash(data []byte) string { h := sha256.Sum256(data) return "sha256:" + hex.EncodeToString(h[:]) } -// ComputeToolHash calculates a deterministic SHA-256 hash for tool execution requests. +// NormalizeEndpoint converts full URLs or relative paths into a canonical relative path +// with deterministically sorted query parameters and redacted credentials. +func NormalizeEndpoint(raw string) string { + s := cassette.RedactString(strings.TrimSpace(raw)) + if s == "" { + return "" + } + + var pathPart, rawQuery string + lower := strings.ToLower(s) + if strings.HasPrefix(lower, "http://") || strings.HasPrefix(lower, "https://") { + if u, err := url.Parse(s); err == nil { + pathPart = u.Path + rawQuery = u.RawQuery + } else { + pathPart = s + } + } else { + if qIdx := strings.IndexByte(s, '?'); qIdx != -1 { + pathPart = s[:qIdx] + rawQuery = s[qIdx+1:] + } else { + pathPart = s + } + } + + if pathPart == "" { + pathPart = "/" + } else if !strings.HasPrefix(pathPart, "/") { + pathPart = "/" + pathPart + } + if len(pathPart) > 1 { + pathPart = strings.TrimRight(pathPart, "/") + if pathPart == "" { + pathPart = "/" + } + } + + if rawQuery != "" { + if vals, err := url.ParseQuery(rawQuery); err == nil { + encoded := vals.Encode() + if encoded != "" { + return pathPart + "?" + encoded + } + } else { + return pathPart + "?" + rawQuery + } + } + return pathPart +} + +// ComputeToolHash calculates a deterministic, secret-redacted SHA-256 hash for tool execution requests. func ComputeToolHash(toolName string, args map[string]interface{}) string { type canonicalTool struct { Name string `json:"name"` Args interface{} `json:"args"` } + redactedArgs := cassette.RedactMap(args) t := canonicalTool{ Name: strings.TrimSpace(toolName), - Args: canonicalizeValue(args), + Args: canonicalizeValue(redactedArgs), } b, _ := json.Marshal(t) return CanonicalHash(b) } -// ComputeRequestHash computes a canonical SHA-256 hash for any Request. +// ComputeRequestHash computes a canonical, secret-redacted SHA-256 hash for any Request. +// Redacting secrets prior to hashing eliminates any secret hash oracle and allows CI dummy keys to match. func ComputeRequestHash(req *cassette.Request) string { + if req == nil { + return "" + } if req.ToolName != "" { return ComputeToolHash(req.ToolName, req.ToolInput) } @@ -84,16 +197,34 @@ func ComputeRequestHash(req *cassette.Request) string { parts = append(parts, strings.ToUpper(strings.TrimSpace(req.Method))) } if req.Endpoint != "" { - parts = append(parts, strings.TrimSpace(req.Endpoint)) + parts = append(parts, NormalizeEndpoint(req.Endpoint)) } - if req.Body != nil { - cb, err := CanonicalJSON(req.Body) - if err == nil { + rawTrimmed := strings.TrimSpace(req.RawBody) + if rawTrimmed != "" { + redactedRaw := cassette.RedactString(rawTrimmed) + if parsed, err := DecodeJSONUseNumber([]byte(redactedRaw)); err == nil { + redactedVal := cassette.RedactValue(parsed) + if cb, err := CanonicalJSON(redactedVal); err == nil { + parts = append(parts, string(cb)) + } else { + parts = append(parts, redactedRaw) + } + } else if req.Body != nil { + redactedBody := cassette.RedactValue(req.Body) + if cb, err := CanonicalJSON(redactedBody); err == nil { + parts = append(parts, string(cb)) + } else { + parts = append(parts, redactedRaw) + } + } else { + parts = append(parts, redactedRaw) + } + } else if req.Body != nil { + redactedBody := cassette.RedactValue(req.Body) + if cb, err := CanonicalJSON(redactedBody); err == nil { parts = append(parts, string(cb)) } - } else if req.RawBody != "" { - parts = append(parts, strings.TrimSpace(req.RawBody)) } payload := strings.Join(parts, "\n") diff --git a/pkg/matcher/matcher_bench_test.go b/pkg/matcher/matcher_bench_test.go new file mode 100644 index 0000000..a91900c --- /dev/null +++ b/pkg/matcher/matcher_bench_test.go @@ -0,0 +1,105 @@ +package matcher + +import ( + "fmt" + "strings" + "testing" + + "github.com/aloneangel45/AgentFixture/pkg/cassette" +) + +func BenchmarkStrictMatcher_JSON(b *testing.B) { + strict := NewStrictMatcher() + req1 := &cassette.Request{ + Method: "POST", + Endpoint: "https://api.anthropic.com/v1/messages?beta=true", + RawBody: `{"model":"claude-sonnet-4-20250514","max_tokens":1024,"stream":true,"messages":[{"role":"user","content":"Analyze pkg/proxy/http_proxy.go for lock contention"}]}`, + } + req1.Matcher.StrictHash = ComputeRequestHash(req1) + + req2 := &cassette.Request{ + Method: "POST", + Endpoint: "/v1/messages?beta=true", + RawBody: `{"stream":true,"messages":[{"content":"Analyze pkg/proxy/http_proxy.go for lock contention","role":"user"}],"max_tokens":1024,"model":"claude-sonnet-4-20250514"}`, + } + + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + res := strict.Match(req1, req2) + if !res.Matched { + b.Fatal("expected match") + } + } +} + +func BenchmarkFuzzyMatcher_ShortPrompt(b *testing.B) { + fuzzy := NewFuzzyMatcher(0.80) + req1 := &cassette.Request{ + Method: "POST", + Endpoint: "/v1/messages", + RawBody: `{"model":"claude-sonnet-4-20250514","request_id":"req_111","prompt":"Inspect auth middleware at 2026-11-10T10:00:00Z"}`, + } + req2 := &cassette.Request{ + Method: "POST", + Endpoint: "/v1/messages", + RawBody: `{"model":"claude-sonnet-4-20250514","request_id":"req_222","prompt":"Inspect auth middleware at 2026-11-11T15:30:00Z"}`, + } + + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + res := fuzzy.Match(req1, req2) + if !res.Matched { + b.Fatal("expected match") + } + } +} + +func BenchmarkFuzzyMatcher_100KBPrompt_MiddleEdit(b *testing.B) { + fuzzy := NewFuzzyMatcher(0.80) + var sb1, sb2 strings.Builder + for i := 0; i < 1600; i++ { + line := fmt.Sprintf("Line %04d: package systems_architecture_context_block_%d_payload_data\n", i, i%97) + sb1.WriteString(line) + if i%50 == 0 { + sb2.WriteString(fmt.Sprintf("Line %04d: package systems_architecture_MODIFIED_block_%d_payload_data\n", i, i%97)) + } else { + sb2.WriteString(line) + } + } + p1 := sb1.String() // ~110 KB + p2 := sb2.String() // ~110 KB with edits across the entire 100KB interior + + req1 := &cassette.Request{Method: "POST", Endpoint: "/v1/messages", RawBody: p1} + req2 := &cassette.Request{Method: "POST", Endpoint: "/v1/messages", RawBody: p2} + + b.SetBytes(int64(len(p1) + len(p2))) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + res := fuzzy.Match(req1, req2) + if !res.Matched { + b.Fatalf("expected match on 100KB similar prompt, got score=%.3f", res.Score) + } + } +} + +func BenchmarkFuzzyMatcher_100KBDisjointPrompt(b *testing.B) { + fuzzy := NewFuzzyMatcher(0.80) + p1 := strings.Repeat("alpha_beta_gamma_delta_12345 ", 3600) // ~104 KB + p2 := strings.Repeat("omega_sigma_theta_kappa_98765 ", 3600) // ~108 KB + + req1 := &cassette.Request{Method: "POST", Endpoint: "/v1/messages", RawBody: p1} + req2 := &cassette.Request{Method: "POST", Endpoint: "/v1/messages", RawBody: p2} + + b.SetBytes(int64(len(p1) + len(p2))) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + res := fuzzy.Match(req1, req2) + if res.Matched { + b.Fatalf("expected mismatch on disjoint 100KB prompts, got score=%.3f", res.Score) + } + } +} diff --git a/pkg/matcher/matcher_test.go b/pkg/matcher/matcher_test.go index aa5c743..5ba67e3 100644 --- a/pkg/matcher/matcher_test.go +++ b/pkg/matcher/matcher_test.go @@ -1,8 +1,10 @@ package matcher import ( + "fmt" "strings" "testing" + "time" "github.com/aloneangel45/AgentFixture/pkg/cassette" ) @@ -45,6 +47,58 @@ func TestCanonicalJSON_KeyOrderIndependence(t *testing.T) { } } +func TestStrictMatcher_64BitIntegerPrecisionAndRawBodyCanonicalization(t *testing.T) { + strict := NewStrictMatcher() + + // 9007199254740993 and 9007199254740992 collide if decoded as float64! + req1 := &cassette.Request{ + Method: "POST", + Endpoint: "/v1/messages", + RawBody: `{"seed": 9007199254740993, "model": "claude-sonnet-4"}`, + } + req2Collision := &cassette.Request{ + Method: "POST", + Endpoint: "/v1/messages", + RawBody: `{"model": "claude-sonnet-4", "seed": 9007199254740992}`, + } + req3Reordered := &cassette.Request{ + Method: "POST", + Endpoint: "https://api.anthropic.com/v1/messages", + RawBody: `{"model":"claude-sonnet-4","seed":9007199254740993}`, + } + + if strict.Match(req1, req2Collision).Matched { + t.Fatalf("PRECISION BUG: 9007199254740993 falsely matched 9007199254740992 due to float64 truncation") + } + if !strict.Match(req1, req3Reordered).Matched { + t.Fatalf("expected reordered JSON RawBody + full URL vs path to match strictly") + } +} + +func TestStrictMatcher_QueryCanonicalizationAndSecretSafeHash(t *testing.T) { + strict := NewStrictMatcher() + + recorded := &cassette.Request{ + Method: "POST", + Endpoint: "https://api.anthropic.com/v1/messages?beta=true&version=2", + RawBody: `{"prompt":"hello","auth":"Bearer sk-ant-api03-realSecretKey1234567890abcdef"}`, + } + recordedHash := ComputeRequestHash(recorded) + recorded.Matcher.StrictHash = recordedHash + + // Replay with reordered query params, relative path, and dummy Bearer token + replayed := &cassette.Request{ + Method: "POST", + Endpoint: "/v1/messages?version=2&beta=true", + RawBody: `{"auth":"Bearer af-replay-dummy-key","prompt":"hello"}`, + } + + res := strict.Match(recorded, replayed) + if !res.Matched { + t.Fatalf("expected secret-redacted hash + canonical query to match, got diff: %s", res.Diff) + } +} + func TestStrictMatcher_Tools(t *testing.T) { strict := NewStrictMatcher() @@ -133,52 +187,118 @@ func TestStrictMatcher_HTTP(t *testing.T) { } } -func TestFuzzyMatcher_ToolDynamicKeysAndCase(t *testing.T) { +func TestFuzzyMatcher_ToolDynamicKeysSliceRecursionAndBareIDPreserved(t *testing.T) { fuzzy := NewFuzzyMatcher(0.80) expected := &cassette.Request{ ToolName: "VIEW_FILE", ToolInput: map[string]interface{}{ - "path": "pkg/auth/auth.go", - "timestamp": "2026-11-05T20:00:00Z", - "nonce": 12345, + "path": "pkg/auth/auth.go", + "timestamp": "2026-11-05T20:00:00Z", + "request_id": "req_111", + "nonce": 12345, + "items": []interface{}{ + map[string]interface{}{ + "name": "item_a", + "trace_id": "tr_aaa", + "created_at": "2026-01-01T00:00:00Z", + }, + }, }, } actual := &cassette.Request{ ToolName: "view_file", ToolInput: map[string]interface{}{ - "path": "pkg/auth/auth.go", - "timestamp": "2026-11-06T12:00:00Z", // differing dynamic timestamp - "nonce": 99999, // differing dynamic nonce + "path": "pkg/auth/auth.go", + "timestamp": "2026-11-06T12:00:00Z", + "request_id": "req_999", + "nonce": 99999, + "items": []interface{}{ + map[string]interface{}{ + "name": "item_a", + "trace_id": "tr_bbb", + "created_at": "2026-02-02T00:00:00Z", + }, + }, }, } res := fuzzy.Match(expected, actual) if !res.Matched || res.Score < 0.99 { - t.Fatalf("expected fuzzy match despite dynamic keys and case differences, got %+v", res) + t.Fatalf("expected fuzzy match despite nested slice dynamic keys, got %+v", res) + } + + // Bare "id" MUST NOT be ignored! get_user(id=1) != get_user(id=2) + user1 := &cassette.Request{ + ToolName: "get_user", + ToolInput: map[string]interface{}{"id": 1}, + } + user2 := &cassette.Request{ + ToolName: "get_user", + ToolInput: map[string]interface{}{"id": 99999}, + } + if fuzzy.Match(user1, user2).Matched { + t.Fatalf("expected bare 'id' key difference to NOT be stripped as a dynamic key") } } -func TestFuzzyMatcher_HTTPPromptNormalization(t *testing.T) { +func TestFuzzyMatcher_HTTPPromptNormalizationAndUTF8Runes(t *testing.T) { fuzzy := NewFuzzyMatcher(0.80) expected := &cassette.Request{ Method: "POST", - Endpoint: "https://api.anthropic.com/v1/messages", + Endpoint: "https://api.anthropic.com/v1/messages?beta=true×tamp=111", RawBody: "Current time is 2026-11-05T20:00:00Z. Please inspect session 12345678-1234-1234-1234-123456789abc.", } actual := &cassette.Request{ Method: "POST", - Endpoint: "https://api.anthropic.com/v1/messages", + Endpoint: "/v1/messages?beta=true×tamp=999", RawBody: "Current time is 2026-11-06T08:15:20Z. Please inspect session 98765432-4321-4321-4321-cba987654321.", } res := fuzzy.Match(expected, actual) if !res.Matched { - t.Fatalf("expected fuzzy match after timestamp, uuid, and whitespace normalization, got %+v", res) + t.Fatalf("expected fuzzy match after URL, timestamp, uuid, and whitespace normalization, got %+v", res) + } + + // Verify UTF-8 multi-byte rune similarity (4 runes differing in 1 rune -> 0.75, NOT 1 - 1/12 = 0.916!) + simUTF8 := StringSimilarity("日本語テスト", "日本語テスX") // 6 runes, 1 rune diff -> 5/6 = 0.8333 + if simUTF8 < 0.82 || simUTF8 > 0.85 { + t.Errorf("expected UTF-8 rune similarity ~0.8333 (5/6), got %f", simUTF8) + } +} + +func TestFuzzyMatcher_100KBPromptPerformanceGuard(t *testing.T) { + fuzzy := NewFuzzyMatcher(0.80) + var sb1, sb2 strings.Builder + for i := 0; i < 1500; i++ { + sb1.WriteString(fmt.Sprintf("chunk_%04d: deterministic agent regression context payload line\n", i)) + if i%40 == 0 { + sb2.WriteString(fmt.Sprintf("chunk_%04d: deterministic agent MODIFIED context payload line\n", i)) + } else { + sb2.WriteString(fmt.Sprintf("chunk_%04d: deterministic agent regression context payload line\n", i)) + } + } + req1 := &cassette.Request{Method: "POST", Endpoint: "/v1/messages", RawBody: sb1.String()} + req2 := &cassette.Request{Method: "POST", Endpoint: "/v1/messages", RawBody: sb2.String()} + + // Warmup + measure + _ = fuzzy.Match(req1, req2) + start := time.Now() + const runs = 10 + for i := 0; i < runs; i++ { + res := fuzzy.Match(req1, req2) + if !res.Matched { + t.Fatalf("expected match on 100KB prompt, got %+v", res) + } + } + avg := time.Since(start) / runs + if avg > 5*time.Millisecond { + t.Errorf("100KB prompt FuzzyMatcher too slow: avg %v per match", avg) } + t.Logf("100KB prompt FuzzyMatcher average latency: %v", avg) } func TestLevenshteinDistance(t *testing.T) { diff --git a/pkg/matcher/strict.go b/pkg/matcher/strict.go index fe4205b..1735d7b 100644 --- a/pkg/matcher/strict.go +++ b/pkg/matcher/strict.go @@ -29,24 +29,28 @@ func (s *StrictMatcher) Match(expected, actual *cassette.Request) MatchResult { } } - expHash := expected.Matcher.StrictHash - if expHash == "" { - expHash = ComputeRequestHash(expected) + actHash := ComputeRequestHash(actual) + if expected.Matcher.StrictHash != "" && expected.Matcher.StrictHash == actHash { + return MatchResult{ + Matched: true, + Score: 1.0, + } } - actHash := actual.Matcher.StrictHash - if actHash == "" { - actHash = ComputeRequestHash(actual) + var expHash string + if hasRequestContent(expected) { + expHash = ComputeRequestHash(expected) + } else { + expHash = expected.Matcher.StrictHash } - if expHash == actHash { + if expHash != "" && expHash == actHash { return MatchResult{ Matched: true, Score: 1.0, } } - // Produce actionable diff explanation diff := s.explainDiff(expected, actual) return MatchResult{ Matched: false, @@ -55,6 +59,10 @@ func (s *StrictMatcher) Match(expected, actual *cassette.Request) MatchResult { } } +func hasRequestContent(r *cassette.Request) bool { + return r.ToolName != "" || r.Method != "" || r.Endpoint != "" || r.RawBody != "" || r.Body != nil +} + func (s *StrictMatcher) explainDiff(expected, actual *cassette.Request) string { var diffs []string @@ -62,15 +70,19 @@ func (s *StrictMatcher) explainDiff(expected, actual *cassette.Request) string { if expected.ToolName != actual.ToolName { diffs = append(diffs, fmt.Sprintf("tool_name mismatch: expected %q, got %q", expected.ToolName, actual.ToolName)) } - if !reflect.DeepEqual(expected.ToolInput, actual.ToolInput) { - diffs = append(diffs, fmt.Sprintf("tool_input args differ: expected %v, got %v", expected.ToolInput, actual.ToolInput)) + redExp := canonicalizeValue(cassette.RedactMap(expected.ToolInput)) + redAct := canonicalizeValue(cassette.RedactMap(actual.ToolInput)) + if !reflect.DeepEqual(redExp, redAct) { + diffs = append(diffs, fmt.Sprintf("tool_input args differ: expected %v, got %v", redExp, redAct)) } } else { if !strings.EqualFold(expected.Method, actual.Method) { diffs = append(diffs, fmt.Sprintf("method mismatch: expected %s, got %s", expected.Method, actual.Method)) } - if expected.Endpoint != actual.Endpoint { - diffs = append(diffs, fmt.Sprintf("endpoint mismatch: expected %s, got %s", expected.Endpoint, actual.Endpoint)) + normExpEnd := NormalizeEndpoint(expected.Endpoint) + normActEnd := NormalizeEndpoint(actual.Endpoint) + if normExpEnd != normActEnd { + diffs = append(diffs, fmt.Sprintf("endpoint mismatch: expected %s, got %s", normExpEnd, normActEnd)) } if expected.RawBody != actual.RawBody { diffs = append(diffs, "body content mismatch") @@ -78,7 +90,7 @@ func (s *StrictMatcher) explainDiff(expected, actual *cassette.Request) string { } if len(diffs) == 0 { - return "payload structural differences in whitespace, headers, or encoding" + return "payload structural differences in JSON values, precision, or encoding" } return strings.Join(diffs, "; ") } diff --git a/pkg/proxy/http_proxy.go b/pkg/proxy/http_proxy.go index ab29d95..92d3259 100644 --- a/pkg/proxy/http_proxy.go +++ b/pkg/proxy/http_proxy.go @@ -2,6 +2,8 @@ package proxy import ( "bytes" + "compress/gzip" + "context" "encoding/json" "fmt" "io" @@ -25,6 +27,14 @@ const ( ModePassthrough Mode = "passthrough" ) +// DriftRecord captures a non-exact (<1.0) semantic match observed during replay. +type DriftRecord struct { + InteractionID string + ToolOrPath string + Similarity float64 + Reason string +} + // Config configures the HTTP proxy/replay server. type Config struct { Mode Mode @@ -34,18 +44,24 @@ type Config struct { SimulateDelay bool ListenAddr string MaxReplayDelay time.Duration + SSETiming cassette.SSETimingConfig } -// Server provides deterministic HTTP playback and recording interception. +// Server provides deterministic HTTP/SSE playback and recording interception. type Server struct { - cfg Config - listener net.Listener - httpServer *http.Server - mu sync.Mutex - cassette *cassette.Cassette - matchedIDs map[string]bool - unmatched []cassette.Request - upstream *url.URL + cfg Config + listener net.Listener + httpServer *http.Server + upstreamClient *http.Client + inFlightMu sync.Mutex + inFlightCond *sync.Cond + inFlight int + mu sync.Mutex + cassette *cassette.Cassette + matchedIDs map[string]bool + unmatched []cassette.Request + driftRecords []DriftRecord + upstream *url.URL } // NewServer initializes a proxy Server with the given configuration. @@ -72,22 +88,35 @@ func NewServer(cfg Config) (*Server, error) { } } - cas := cfg.Cassette - if cas == nil { + var cas *cassette.Cassette + if cfg.Cassette != nil { + cas = cfg.Cassette.Clone() + } else { cas = cassette.NewCassette(fmt.Sprintf("cas_%d", time.Now().UnixNano())) } s := &Server{ - cfg: cfg, - cassette: cas, - matchedIDs: make(map[string]bool), - unmatched: make([]cassette.Request, 0), - upstream: upstreamParsed, + cfg: cfg, + cassette: cas, + matchedIDs: make(map[string]bool), + unmatched: make([]cassette.Request, 0), + driftRecords: make([]DriftRecord, 0), + upstream: upstreamParsed, + upstreamClient: &http.Client{ + // No fixed Client.Timeout so long-running LLM SSE / thinking streams never abort mid-flight. + Transport: &http.Transport{ + Proxy: nil, + DialContext: (&net.Dialer{Timeout: 15 * time.Second, KeepAlive: 30 * time.Second}).DialContext, + TLSHandshakeTimeout: 15 * time.Second, + ResponseHeaderTimeout: 30 * time.Second, + IdleConnTimeout: 90 * time.Second, + }, + }, } - mux := http.NewServeMux() - mux.HandleFunc("/", s.handleHTTP) - s.httpServer = &http.Server{Handler: mux} + // Register handler directly (bypassing ServeMux path redirects so HTTP CONNECT and proxy URIs work cleanly). + s.inFlightCond = sync.NewCond(&s.inFlightMu) + s.httpServer = &http.Server{Handler: http.HandlerFunc(s.handleHTTP)} return s, nil } @@ -107,12 +136,33 @@ func (s *Server) Start() error { return nil } -// Close shuts down the server and releases all network ports. +// WaitIdle blocks until all active HTTP handler goroutines finish recording or replaying. +func (s *Server) WaitIdle() { + s.inFlightMu.Lock() + for s.inFlight > 0 { + s.inFlightCond.Wait() + } + s.inFlightMu.Unlock() +} + +// Close gracefully drains active handlers, shuts down the server, and releases network ports. func (s *Server) Close() error { + var err error if s.httpServer != nil { - return s.httpServer.Close() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + err = s.httpServer.Shutdown(ctx) + if err != nil { + _ = s.httpServer.Close() + } } - return nil + s.WaitIdle() + return err +} + +// Mode returns the configured proxy Mode. +func (s *Server) Mode() Mode { + return s.cfg.Mode } // Addr returns the network address string (e.g. 127.0.0.1:54321). @@ -125,33 +175,86 @@ func (s *Server) Addr() string { // URL returns the base HTTP URL pointing to this proxy. func (s *Server) URL() string { + if s.Addr() == "" { + return "" + } return "http://" + s.Addr() } -// GetCassette returns the in-memory Cassette containing recorded or loaded interactions. +// GetCassette returns a thread-safe clone of the Cassette containing recorded or loaded interactions. func (s *Server) GetCassette() *cassette.Cassette { + s.WaitIdle() s.mu.Lock() defer s.mu.Unlock() - return s.cassette + return s.cassette.Clone() } -// MatchedCount returns how many cassette interactions have been replayed. +// MatchedCount returns how many unique cassette interactions have been replayed. func (s *Server) MatchedCount() int { + s.WaitIdle() s.mu.Lock() defer s.mu.Unlock() return len(s.matchedIDs) } +// MatchedIDs returns a snapshot map of all cassette interaction IDs matched during replay. +func (s *Server) MatchedIDs() map[string]bool { + s.WaitIdle() + s.mu.Lock() + defer s.mu.Unlock() + cp := make(map[string]bool, len(s.matchedIDs)) + for k, v := range s.matchedIDs { + cp[k] = v + } + return cp +} + +// DriftRecords returns any non-exact semantic matches recorded during fuzzy replay. +func (s *Server) DriftRecords() []DriftRecord { + s.WaitIdle() + s.mu.Lock() + defer s.mu.Unlock() + cp := make([]DriftRecord, len(s.driftRecords)) + copy(cp, s.driftRecords) + return cp +} + // UnmatchedRequests returns any requests received during replay that failed to match. func (s *Server) UnmatchedRequests() []cassette.Request { + s.WaitIdle() s.mu.Lock() defer s.mu.Unlock() copied := make([]cassette.Request, len(s.unmatched)) - copy(copied, s.unmatched) + for i := range s.unmatched { + copied[i] = s.unmatched[i].Clone() + } return copied } func (s *Server) handleHTTP(w http.ResponseWriter, r *http.Request) { + s.inFlightMu.Lock() + s.inFlight++ + s.inFlightMu.Unlock() + defer func() { + s.inFlightMu.Lock() + s.inFlight-- + if s.inFlight == 0 { + s.inFlightCond.Broadcast() + } + s.inFlightMu.Unlock() + }() + // Zero Outbound Network Guard: strictly forbid HTTPS CONNECT tunneling in replay mode (and unsupported tunnel proxy use). + if r.Method == http.MethodConnect { + w.Header().Set("Content-Type", "application/json") + w.Header().Set("X-AgentFixture-Blocked", "connect-tunnel-forbidden") + w.WriteHeader(http.StatusForbidden) + _ = json.NewEncoder(w).Encode(map[string]string{ + "error": "AgentFixture blocked HTTP CONNECT tunnel: zero outbound network connections permitted", + "target": r.Host, + }) + return + } + bodyBytes, err := io.ReadAll(r.Body) if err != nil { http.Error(w, "Failed to read request body", http.StatusBadRequest) @@ -159,7 +262,7 @@ func (s *Server) handleHTTP(w http.ResponseWriter, r *http.Request) { } _ = r.Body.Close() - headers := make(map[string]string) + headers := make(map[string]string, len(r.Header)) for k, v := range r.Header { if len(v) > 0 { headers[k] = strings.Join(v, ", ") @@ -167,13 +270,23 @@ func (s *Server) handleHTTP(w http.ResponseWriter, r *http.Request) { } var jsonBody interface{} - if len(bodyBytes) > 0 && strings.Contains(r.Header.Get("Content-Type"), "json") { - _ = json.Unmarshal(bodyBytes, &jsonBody) + if len(bodyBytes) > 0 && strings.Contains(strings.ToLower(r.Header.Get("Content-Type")), "json") { + if parsed, decErr := matcher.DecodeJSONUseNumber(bodyBytes); decErr == nil { + jsonBody = parsed + } + } + + endpoint := r.URL.Path + if endpoint == "" { + endpoint = "/" + } + if r.URL.RawQuery != "" { + endpoint = endpoint + "?" + r.URL.RawQuery } candidate := cassette.Request{ Method: r.Method, - Endpoint: r.URL.Path, + Endpoint: endpoint, Headers: headers, Body: jsonBody, RawBody: string(bodyBytes), @@ -192,77 +305,153 @@ func (s *Server) handleHTTP(w http.ResponseWriter, r *http.Request) { } } -func (s *Server) handleReplay(w http.ResponseWriter, r *http.Request, candidate *cassette.Request) { - s.mu.Lock() - defer s.mu.Unlock() - +func (s *Server) selectMatchLocked(candidate *cassette.Request) (*cassette.Interaction, float64, string) { var bestInter *cassette.Interaction var bestScore float64 var lastDiff string + // Pass 1: Prefer unconsumed interactions in recorded sequence order for stateful multi-turn agents. for i := range s.cassette.Interactions { inter := &s.cassette.Interactions[i] + if s.matchedIDs[inter.ID] { + continue + } res := s.cfg.Matcher.Match(&inter.Request, candidate) if res.Matched && res.Score > bestScore { bestScore = res.Score bestInter = inter if res.Score == 1.0 { - break + return bestInter, bestScore, "" } } if res.Diff != "" { lastDiff = res.Diff } } - if bestInter != nil { - s.matchedIDs[bestInter.ID] = true + return bestInter, bestScore, lastDiff + } - if s.cfg.SimulateDelay && bestInter.Response.LatencyMs > 0 { - sleepDur := time.Duration(bestInter.Response.LatencyMs) * time.Millisecond - if sleepDur > s.cfg.MaxReplayDelay { - sleepDur = s.cfg.MaxReplayDelay + // Pass 2: Fallback to already-matched interactions for idempotent retries. + for i := range s.cassette.Interactions { + inter := &s.cassette.Interactions[i] + if !s.matchedIDs[inter.ID] { + continue + } + res := s.cfg.Matcher.Match(&inter.Request, candidate) + if res.Matched && res.Score > bestScore { + bestScore = res.Score + bestInter = inter + if res.Score == 1.0 { + return bestInter, bestScore, "" } - time.Sleep(sleepDur) } + if res.Diff != "" { + lastDiff = res.Diff + } + } + return bestInter, bestScore, lastDiff +} - for k, v := range bestInter.Response.Headers { - if !isHopByHopHeader(k) { - w.Header().Set(k, v) - } +func (s *Server) handleReplay(w http.ResponseWriter, r *http.Request, candidate *cassette.Request) { + s.mu.Lock() + bestInter, bestScore, lastDiff := s.selectMatchLocked(candidate) + + if bestInter == nil { + missed := candidate.Clone() + cassette.RedactRequest(&missed) + s.unmatched = append(s.unmatched, missed) + s.mu.Unlock() + + w.Header().Set("Content-Type", "application/json") + w.Header().Set("X-AgentFixture-Mismatch", "true") + w.WriteHeader(http.StatusNotFound) + + resp := map[string]interface{}{ + "error": "AgentFixture replay mismatch: no recorded interaction matched request", + "candidate_method": missed.Method, + "candidate_endpoint": missed.Endpoint, + "diff_explanation": lastDiff, + } + _ = json.NewEncoder(w).Encode(resp) + return + } + + s.matchedIDs[bestInter.ID] = true + if bestScore > 0 && bestScore < 1.0 { + toolOrPath := bestInter.Request.ToolName + if toolOrPath == "" { + toolOrPath = bestInter.Request.Endpoint + } + reason := lastDiff + if reason == "" { + reason = fmt.Sprintf("matched via %s matcher below exact 1.0 score", s.cfg.Matcher.Name()) + } + s.driftRecords = append(s.driftRecords, DriftRecord{ + InteractionID: bestInter.ID, + ToolOrPath: toolOrPath, + Similarity: bestScore, + Reason: reason, + }) + } + matchedSnapshot := bestInter.Clone() + // Release lock BEFORE sleeping, writing headers, or streaming SSE chunks! + s.mu.Unlock() + + if s.cfg.SimulateDelay && matchedSnapshot.Response.LatencyMs > 0 && + matchedSnapshot.Type != cassette.InteractionTypeSSE && len(matchedSnapshot.Response.SSEEvents) == 0 { + sleepDur := time.Duration(matchedSnapshot.Response.LatencyMs) * time.Millisecond + if sleepDur > s.cfg.MaxReplayDelay { + sleepDur = s.cfg.MaxReplayDelay + } + select { + case <-r.Context().Done(): + return + case <-time.After(sleepDur): + } + } + + for k, v := range matchedSnapshot.Response.Headers { + if !isHopByHopOrEncodingHeader(k) { + w.Header().Set(k, v) } - w.Header().Set("X-AgentFixture-Replay", "true") - w.Header().Set("X-AgentFixture-Interaction-ID", bestInter.ID) + } + w.Header().Set("X-AgentFixture-Replay", "true") + w.Header().Set("X-AgentFixture-Interaction-ID", matchedSnapshot.ID) + + status := matchedSnapshot.Response.StatusCode + if status == 0 { + status = http.StatusOK + } - status := bestInter.Response.StatusCode - if status == 0 { - status = http.StatusOK + isSSE := matchedSnapshot.Type == cassette.InteractionTypeSSE || + len(matchedSnapshot.Response.SSEEvents) > 0 || + strings.Contains(strings.ToLower(w.Header().Get("Content-Type")), "text/event-stream") + + if isSSE { + if w.Header().Get("Content-Type") == "" { + w.Header().Set("Content-Type", "text/event-stream; charset=utf-8") } + w.Header().Set("Cache-Control", "no-cache") w.WriteHeader(status) - if bestInter.Response.RawBody != "" { - _, _ = w.Write([]byte(bestInter.Response.RawBody)) - } else if bestInter.Response.ToolResult != "" { - _, _ = w.Write([]byte(bestInter.Response.ToolResult)) - } else if bestInter.Response.Body != nil { - _ = json.NewEncoder(w).Encode(bestInter.Response.Body) + events := matchedSnapshot.Response.SSEEvents + if len(events) == 0 && matchedSnapshot.Response.RawBody != "" { + events, _ = cassette.DecodeSSE(strings.NewReader(matchedSnapshot.Response.RawBody)) } + flusher, _ := w.(http.Flusher) + _ = cassette.ReplaySSEStream(r.Context(), w, flusher, events, s.cfg.SSETiming) return } - // Mismatch encountered - s.unmatched = append(s.unmatched, *candidate) - w.Header().Set("Content-Type", "application/json") - w.Header().Set("X-AgentFixture-Mismatch", "true") - w.WriteHeader(http.StatusNotFound) - - resp := map[string]interface{}{ - "error": "AgentFixture replay mismatch: no recorded interaction matched request", - "candidate_method": candidate.Method, - "candidate_endpoint": candidate.Endpoint, - "diff_explanation": lastDiff, + w.WriteHeader(status) + if matchedSnapshot.Response.RawBody != "" { + _, _ = w.Write([]byte(matchedSnapshot.Response.RawBody)) + } else if matchedSnapshot.Response.ToolResult != "" { + _, _ = w.Write([]byte(matchedSnapshot.Response.ToolResult)) + } else if matchedSnapshot.Response.Body != nil { + _ = json.NewEncoder(w).Encode(matchedSnapshot.Response.Body) } - _ = json.NewEncoder(w).Encode(resp) } func (s *Server) handleRecord(w http.ResponseWriter, r *http.Request, candidate *cassette.Request, bodyBytes []byte) { @@ -271,18 +460,17 @@ func (s *Server) handleRecord(w http.ResponseWriter, r *http.Request, candidate return } - outURL := *s.upstream - outURL.Path = r.URL.Path - outURL.RawQuery = r.URL.RawQuery - - outReq, err := http.NewRequest(r.Method, outURL.String(), bytes.NewReader(bodyBytes)) + targetURL := joinUpstreamURL(s.upstream, r.URL) + outReq, err := http.NewRequestWithContext(r.Context(), r.Method, targetURL, bytes.NewReader(bodyBytes)) if err != nil { http.Error(w, fmt.Sprintf("Failed to construct upstream request: %v", err), http.StatusInternalServerError) return } + outReq.Host = s.upstream.Host for k, vv := range r.Header { - if !isHopByHopHeader(k) { + // Strip Accept-Encoding & Content-Length so Go's http.Transport transparently negotiates and decompresses gzip! + if !isHopByHopOrEncodingHeader(k) && !strings.EqualFold(k, "Accept-Encoding") && !strings.EqualFold(k, "Host") { for _, v := range vv { outReq.Header.Add(k, v) } @@ -290,55 +478,146 @@ func (s *Server) handleRecord(w http.ResponseWriter, r *http.Request, candidate } start := time.Now() - client := &http.Client{Timeout: 30 * time.Second} - resp, err := client.Do(outReq) - duration := time.Since(start).Milliseconds() - + resp, err := s.upstreamClient.Do(outReq) if err != nil { http.Error(w, fmt.Sprintf("Upstream request failed: %v", err), http.StatusBadGateway) return } defer resp.Body.Close() - respBody, err := io.ReadAll(resp.Body) - if err != nil { - http.Error(w, fmt.Sprintf("Failed to read upstream response body: %v", err), http.StatusBadGateway) - return + var bodyReader io.Reader = resp.Body + if strings.EqualFold(resp.Header.Get("Content-Encoding"), "gzip") { + gzReader, gzErr := gzip.NewReader(resp.Body) + if gzErr != nil { + http.Error(w, fmt.Sprintf("Failed to initialize upstream gzip reader: %v", gzErr), http.StatusBadGateway) + return + } + defer gzReader.Close() + bodyReader = gzReader } respHeaders := make(map[string]string) for k, v := range resp.Header { - if len(v) > 0 { + if len(v) > 0 && !isHopByHopOrEncodingHeader(k) { respHeaders[k] = strings.Join(v, ", ") } } + contentType := strings.ToLower(resp.Header.Get("Content-Type")) + if strings.Contains(contentType, "text/event-stream") { + for k, vv := range resp.Header { + if !isHopByHopOrEncodingHeader(k) { + for _, val := range vv { + w.Header().Add(k, val) + } + } + } + w.Header().Set("Cache-Control", "no-cache") + w.Header().Set("X-AgentFixture-Recorded", "true") + w.WriteHeader(resp.StatusCode) + + flusher, _ := w.(http.Flusher) + events, streamErr := cassette.RecordSSEStream(bodyReader, func(ev cassette.SSEEvent) error { + encoded := cassette.EncodeSSEEvent(ev) + if _, wErr := w.Write(encoded); wErr != nil { + return wErr + } + if flusher != nil { + flusher.Flush() + } + return nil + }) + duration := time.Since(start).Milliseconds() + + summary := cassette.ReconstructStreamSummary(events) + meta := make(map[string]string) + if summary.Model != "" { + meta["model"] = summary.Model + } + if summary.StopReason != "" { + meta["stop_reason"] = summary.StopReason + } + if summary.MessageID != "" { + meta["message_id"] = summary.MessageID + } + if streamErr != nil && streamErr != io.EOF { + meta["stream_error"] = cassette.RedactString(streamErr.Error()) + } + + storedReq := candidate.Clone() + cassette.RedactRequest(&storedReq) + storedReq.Matcher.StrictHash = matcher.ComputeRequestHash(&storedReq) + + inter := cassette.Interaction{ + Type: cassette.InteractionTypeSSE, + Request: storedReq, + Response: cassette.Response{ + StatusCode: resp.StatusCode, + LatencyMs: duration, + Headers: respHeaders, + SSEEvents: events, + Usage: summary.Usage, + }, + Metadata: meta, + } + cassette.RedactInteraction(&inter) + + s.mu.Lock() + seq := len(s.cassette.Interactions) + 1 + inter.ID = fmt.Sprintf("int_%03d", seq) + inter.Sequence = seq + if s.cassette.Metadata.Model == "" && summary.Model != "" { + s.cassette.Metadata.Model = summary.Model + } + s.cassette.Interactions = append(s.cassette.Interactions, inter) + s.mu.Unlock() + return + } + + respBody, err := io.ReadAll(bodyReader) + duration := time.Since(start).Milliseconds() + if err != nil { + http.Error(w, fmt.Sprintf("Failed to read upstream response body: %v", err), http.StatusBadGateway) + return + } + var respJSON interface{} - if strings.Contains(resp.Header.Get("Content-Type"), "json") { - _ = json.Unmarshal(respBody, &respJSON) + var usage cassette.Usage + if strings.Contains(contentType, "json") && len(respBody) > 0 { + if parsed, decErr := matcher.DecodeJSONUseNumber(respBody); decErr == nil { + respJSON = parsed + usage = cassette.ExtractUsageFromBody(parsed) + } } - s.mu.Lock() - seq := len(s.cassette.Interactions) + 1 + storedReq := candidate.Clone() + cassette.RedactRequest(&storedReq) + storedReq.Matcher.StrictHash = matcher.ComputeRequestHash(&storedReq) + inter := cassette.Interaction{ - ID: fmt.Sprintf("int_%03d", seq), - Sequence: seq, - Type: cassette.InteractionTypeHTTP, - Request: *candidate, + Type: cassette.InteractionTypeHTTP, + Request: storedReq, Response: cassette.Response{ StatusCode: resp.StatusCode, LatencyMs: duration, Headers: respHeaders, Body: respJSON, RawBody: string(respBody), + Usage: usage, }, } + cassette.RedactInteraction(&inter) + + s.mu.Lock() + seq := len(s.cassette.Interactions) + 1 + inter.ID = fmt.Sprintf("int_%03d", seq) + inter.Sequence = seq s.cassette.Interactions = append(s.cassette.Interactions, inter) s.mu.Unlock() - for k, v := range resp.Header { - if !isHopByHopHeader(k) { - for _, val := range v { + for k, vv := range resp.Header { + if !isHopByHopOrEncodingHeader(k) { + for _, val := range vv { w.Header().Add(k, val) } } @@ -353,25 +632,23 @@ func (s *Server) handlePassthrough(w http.ResponseWriter, r *http.Request, bodyB http.Error(w, "Passthrough requires upstream URL", http.StatusBadGateway) return } - outURL := *s.upstream - outURL.Path = r.URL.Path - outURL.RawQuery = r.URL.RawQuery - - outReq, err := http.NewRequest(r.Method, outURL.String(), bytes.NewReader(bodyBytes)) + targetURL := joinUpstreamURL(s.upstream, r.URL) + outReq, err := http.NewRequestWithContext(r.Context(), r.Method, targetURL, bytes.NewReader(bodyBytes)) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } + outReq.Host = s.upstream.Host + for k, vv := range r.Header { - if !isHopByHopHeader(k) { + if !isHopByHopHeader(k) && !strings.EqualFold(k, "Host") { for _, v := range vv { outReq.Header.Add(k, v) } } } - client := &http.Client{Timeout: 30 * time.Second} - resp, err := client.Do(outReq) + resp, err := s.upstreamClient.Do(outReq) if err != nil { http.Error(w, err.Error(), http.StatusBadGateway) return @@ -379,14 +656,63 @@ func (s *Server) handlePassthrough(w http.ResponseWriter, r *http.Request, bodyB defer resp.Body.Close() for k, vv := range resp.Header { - for _, v := range vv { - w.Header().Add(k, v) + if !isHopByHopHeader(k) { + for _, v := range vv { + w.Header().Add(k, v) + } } } w.WriteHeader(resp.StatusCode) + flusher, _ := w.(http.Flusher) + if flusher != nil && strings.Contains(strings.ToLower(resp.Header.Get("Content-Type")), "text/event-stream") { + buf := make([]byte, 4096) + for { + n, rErr := resp.Body.Read(buf) + if n > 0 { + _, _ = w.Write(buf[:n]) + flusher.Flush() + } + if rErr != nil { + break + } + } + return + } _, _ = io.Copy(w, resp.Body) } +func joinUpstreamURL(upstream *url.URL, reqURL *url.URL) string { + out := *upstream + basePath := strings.TrimRight(upstream.Path, "/") + reqPath := reqURL.Path + if reqPath == "" { + reqPath = "/" + } else if !strings.HasPrefix(reqPath, "/") { + reqPath = "/" + reqPath + } + + if basePath == "" { + out.Path = reqPath + } else if reqPath == "/" { + out.Path = basePath + "/" + } else if reqPath == basePath || strings.HasPrefix(reqPath, basePath+"/") { + out.Path = reqPath + } else { + out.Path = basePath + reqPath + } + out.RawPath = "" + + switch { + case upstream.RawQuery != "" && reqURL.RawQuery != "": + out.RawQuery = upstream.RawQuery + "&" + reqURL.RawQuery + case reqURL.RawQuery != "": + out.RawQuery = reqURL.RawQuery + default: + out.RawQuery = upstream.RawQuery + } + return out.String() +} + func isHopByHopHeader(name string) bool { switch strings.ToLower(name) { case "connection", "keep-alive", "proxy-authenticate", @@ -396,3 +722,15 @@ func isHopByHopHeader(name string) bool { return false } } + +func isHopByHopOrEncodingHeader(name string) bool { + if isHopByHopHeader(name) { + return true + } + switch strings.ToLower(name) { + case "content-encoding", "content-length": + return true + default: + return false + } +} diff --git a/pkg/proxy/offline.go b/pkg/proxy/offline.go new file mode 100644 index 0000000..9bec394 --- /dev/null +++ b/pkg/proxy/offline.go @@ -0,0 +1,82 @@ +package proxy + +import ( + "context" + "errors" + "fmt" + "net" + "net/http" + "net/url" + "strings" + "time" +) + +// ErrOutboundNetworkBlocked is returned whenever an offline HTTP client attempts to dial a non-loopback address. +var ErrOutboundNetworkBlocked = errors.New("agentfixture: outbound network access blocked in offline replay mode") + +// IsLoopbackHost reports whether host refers strictly to a local loopback interface (127.0.0.0/8, ::1, or localhost) +// without performing external DNS resolution. +func IsLoopbackHost(host string) bool { + h := strings.TrimSpace(host) + if h == "" { + return false + } + h = strings.TrimPrefix(h, "[") + h = strings.TrimSuffix(h, "]") + if zoneIdx := strings.IndexByte(h, '%'); zoneIdx != -1 { + h = h[:zoneIdx] + } + if strings.EqualFold(h, "localhost") { + return true + } + ip := net.ParseIP(h) + return ip != nil && ip.IsLoopback() +} + +// NewOfflineHTTPClient constructs an *http.Client guaranteed to make zero outbound network connections. +// Its custom DialContext blocks all TCP/UDP dials to non-loopback addresses (anything outside 127.0.0.1, ::1, localhost). +// When proxyURL is non-empty, plain HTTP requests to external hostnames are routed through the local replay proxy, +// while HTTPS CONNECT tunnels are rejected with 403 Forbidden. +func NewOfflineHTTPClient(proxyURL string) *http.Client { + var parsedProxy *url.URL + if strings.TrimSpace(proxyURL) != "" { + if u, err := url.Parse(strings.TrimSpace(proxyURL)); err == nil && u.Host != "" { + parsedProxy = u + } + } + + dialer := &net.Dialer{ + Timeout: 10 * time.Second, + KeepAlive: 30 * time.Second, + } + + transport := &http.Transport{ + Proxy: func(req *http.Request) (*url.URL, error) { + if parsedProxy == nil { + return nil, nil + } + // Requests already targeting loopback connect directly without proxy indirection. + if strings.EqualFold(req.URL.Host, parsedProxy.Host) || IsLoopbackHost(req.URL.Hostname()) { + return nil, nil + } + return parsedProxy, nil + }, + DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { + host, port, err := net.SplitHostPort(addr) + if err != nil { + return nil, fmt.Errorf("%w: invalid address %q: %v", ErrOutboundNetworkBlocked, addr, err) + } + if !IsLoopbackHost(host) { + return nil, fmt.Errorf("%w: attempted %s connection to %s", ErrOutboundNetworkBlocked, network, addr) + } + return dialer.DialContext(ctx, network, net.JoinHostPort(host, port)) + }, + ForceAttemptHTTP2: false, + IdleConnTimeout: 30 * time.Second, + ResponseHeaderTimeout: 30 * time.Second, + } + + return &http.Client{ + Transport: transport, + } +} diff --git a/pkg/proxy/proxy_test.go b/pkg/proxy/proxy_test.go index a1e8c34..a6e2622 100644 --- a/pkg/proxy/proxy_test.go +++ b/pkg/proxy/proxy_test.go @@ -2,7 +2,9 @@ package proxy import ( "bytes" - "encoding/json" + "compress/gzip" + "errors" + "fmt" "io" "net/http" "net/http/httptest" @@ -96,18 +98,68 @@ func TestProxy_ReplayMode_MatchingAndMismatch(t *testing.T) { } } -func TestProxy_RecordMode(t *testing.T) { - // Mock upstream service +func TestProxy_MultiTurnStatefulSequentialReplay(t *testing.T) { + cas := cassette.NewCassette("cas_stateful") + for i, content := range []string{"turn_1_state", "turn_2_state"} { + cas.Interactions = append(cas.Interactions, cassette.Interaction{ + ID: fmt.Sprintf("int_%02d", i+1), + Sequence: i + 1, + Type: cassette.InteractionTypeHTTP, + Request: cassette.Request{ + Method: "GET", + Endpoint: "/v1/state", + }, + Response: cassette.Response{ + StatusCode: 200, + RawBody: content, + }, + }) + } + + srv, err := NewServer(Config{Mode: ModeReplay, Cassette: cas}) + if err != nil { + t.Fatal(err) + } + if err := srv.Start(); err != nil { + t.Fatal(err) + } + defer srv.Close() + + for _, want := range []string{"turn_1_state", "turn_2_state", "turn_1_state"} { + resp, err := http.Get(srv.URL() + "/v1/state") + if err != nil { + t.Fatal(err) + } + got, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if string(got) != want { + t.Fatalf("expected %q, got %q", want, string(got)) + } + } + if srv.MatchedCount() != 2 { + t.Errorf("expected both interactions consumed across 2 turns, got %d", srv.MatchedCount()) + } +} + +func TestProxy_RecordMode_GzipAndBasePathJoin(t *testing.T) { upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/base/v1/messages" { + t.Errorf("expected joined path /base/v1/messages, got %s", r.URL.Path) + } w.Header().Set("Content-Type", "application/json") + w.Header().Set("Content-Encoding", "gzip") + var gzBuf bytes.Buffer + gz := gzip.NewWriter(&gzBuf) + _, _ = gz.Write([]byte(`{"result":"decompressed clean json","usage":{"input_tokens":15,"output_tokens":5}}`)) + _ = gz.Close() w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte(`{"result":"live upstream response"}`)) + _, _ = w.Write(gzBuf.Bytes()) })) defer upstream.Close() srv, err := NewServer(Config{ Mode: ModeRecord, - UpstreamURL: upstream.URL, + UpstreamURL: upstream.URL + "/base", }) if err != nil { t.Fatalf("failed to create recording proxy: %v", err) @@ -117,66 +169,137 @@ func TestProxy_RecordMode(t *testing.T) { } defer srv.Close() - reqBody := `{"query":"test"}` - resp, err := http.Post(srv.URL()+"/api/v1/test", "application/json", strings.NewReader(reqBody)) + req, _ := http.NewRequest("POST", srv.URL()+"/v1/messages", strings.NewReader(`{"query":"test"}`)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept-Encoding", "gzip, deflate") + req.Header.Set("Authorization", "Bearer sk-ant-api03-1234567890abcdef1234567890") + + resp, err := http.DefaultClient.Do(req) if err != nil { t.Fatalf("record request failed: %v", err) } defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - t.Errorf("expected 200 OK from recorded upstream, got %d", resp.StatusCode) - } - cas := srv.GetCassette() if len(cas.Interactions) != 1 { t.Fatalf("expected 1 interaction recorded, got %d", len(cas.Interactions)) } - if !strings.Contains(cas.Interactions[0].Response.RawBody, "live upstream response") { - t.Errorf("unexpected recorded response: %s", cas.Interactions[0].Response.RawBody) + inter := cas.Interactions[0] + if !strings.Contains(inter.Response.RawBody, "decompressed clean json") { + t.Fatalf("expected decompressed plaintext JSON in cassette, got: %q", inter.Response.RawBody) + } + if _, hasCE := inter.Response.Headers["Content-Encoding"]; hasCE { + t.Errorf("expected Content-Encoding header stripped from stored cassette headers") + } + if inter.Response.Usage.InputTokens != 15 || inter.Response.Usage.OutputTokens != 5 { + t.Errorf("expected extracted usage 15/5, got %+v", inter.Response.Usage) + } + if hasSec, reason := cassette.CheckForUnredactedSecrets(cas); hasSec { + t.Errorf("expected recorded cassette to be auto-redacted, got: %s", reason) } } -func TestStdioReplayer_JSONRPC(t *testing.T) { - cas := cassette.NewCassette("cas_stdio_test") - cas.Interactions = append(cas.Interactions, cassette.Interaction{ - ID: "tool_001", - Sequence: 1, - Type: cassette.InteractionTypeTool, - Request: cassette.Request{ - ToolName: "view_file", - ToolInput: map[string]interface{}{ - "path": "README.md", - }, - }, - Response: cassette.Response{ - StatusCode: 0, - ToolResult: "# AgentFixture Documentation", - }, - }) +func TestProxy_OfflineGuard_BlocksConnectAndExternalDial(t *testing.T) { + srv, err := NewServer(Config{Mode: ModeReplay, Cassette: cassette.NewCassette("offline_test")}) + if err != nil { + t.Fatal(err) + } + if err := srv.Start(); err != nil { + t.Fatal(err) + } + defer srv.Close() - replayer := NewStdioReplayer(cas, matcher.NewStrictMatcher()) + // 1. Direct HTTP CONNECT request must be rejected with 403 Forbidden + connectReq, _ := http.NewRequest(http.MethodConnect, srv.URL(), nil) + connectReq.Host = "api.anthropic.com:443" + resp, err := http.DefaultClient.Do(connectReq) + if err != nil { + t.Fatalf("CONNECT request transport error: %v", err) + } + _ = resp.Body.Close() + if resp.StatusCode != http.StatusForbidden { + t.Fatalf("expected 403 Forbidden on HTTP CONNECT, got %d", resp.StatusCode) + } - input := `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"view_file","arguments":{"path":"README.md"}}}` + "\n" - inBuf := strings.NewReader(input) - var outBuf bytes.Buffer + // 2. Offline HTTP client without proxy blocks external TCP dials immediately + offlineDirect := NewOfflineHTTPClient("") + _, dialErr := offlineDirect.Get("http://example.com/test") + if dialErr == nil || !errors.Is(dialErr, ErrOutboundNetworkBlocked) { + t.Fatalf("expected ErrOutboundNetworkBlocked on external dial, got: %v", dialErr) + } - if err := replayer.ProcessStream(inBuf, &outBuf); err != nil { - t.Fatalf("ProcessStream failed: %v", err) + // 3. Offline HTTP client with replay proxy rejects HTTPS outbound tunnel + offlineProxied := NewOfflineHTTPClient(srv.URL()) + _, httpsErr := offlineProxied.Get("https://api.anthropic.com/v1/messages") + if httpsErr == nil { + t.Fatalf("expected HTTPS outbound tunnel via replay proxy to be blocked") } +} - var resp JSONRPCResponse - if err := json.Unmarshal(outBuf.Bytes(), &resp); err != nil { - t.Fatalf("failed to unmarshal JSON-RPC output: %v", err) +func TestStdioReplayer_ContentLengthNotificationsSynthesisAndRecorder(t *testing.T) { + // 1. Record live MCP stdio session via StdioRecorder (in-memory streams) + rec := NewStdioRecorder(cassette.NewCassette("mcp_rec_01")) + serverInR, serverInW := io.Pipe() + serverOutR, serverOutW := io.Pipe() + + hugeArg := strings.Repeat("X", 80*1024) // 80KB (>64KB default scanner buffer!) + clientReq := fmt.Sprintf(`{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"analyze_blob","arguments":{"blob":%q,"api_key":"sk-ant-api03-1234567890abcdef1234567890"}}}`, hugeArg) + framedReq := fmt.Sprintf("Content-Length: %d\r\n\r\n%s", len(clientReq), clientReq) + + go func() { + defer serverOutW.Close() + br := NewStdioReplayer(nil, nil) // just to verify unused; read raw frame via helper + _ = br + // Read request arriving at mock MCP server and reply with Content-Length frame + buf := make([]byte, len(framedReq)) + _, _ = io.ReadFull(serverInR, buf) + reply := `{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"blob_ok"}]}}` + _, _ = fmt.Fprintf(serverOutW, "Content-Length: %d\r\n\r\n%s", len(reply), reply) + }() + + var clientLiveOut bytes.Buffer + recCas, err := rec.RecordStreams(strings.NewReader(framedReq), serverInW, serverOutR, &clientLiveOut) + if err != nil { + t.Fatalf("StdioRecorder.RecordStreams failed: %v", err) } - if resp.Error != nil { - t.Fatalf("unexpected error response: %v", resp.Error) + if len(recCas.Interactions) != 1 { + t.Fatalf("expected 1 recorded MCP interaction, got %d", len(recCas.Interactions)) } + if recCas.Interactions[0].Response.ToolResult != "blob_ok" { + t.Errorf("expected recorded tool_result 'blob_ok', got %q", recCas.Interactions[0].Response.ToolResult) + } + if hasSec, reason := cassette.CheckForUnredactedSecrets(recCas); hasSec { + t.Fatalf("unredacted secret in recorded MCP cassette: %s", reason) + } + + // 2. Replay session testing: initialize + notification (ignored!) + tools/list + Content-Length tools/call + escaped error + replayer := NewStdioReplayer(recCas, matcher.NewStrictMatcher()) - resMap := resp.Result.(map[string]interface{}) - content := resMap["content"].([]interface{}) - first := content[0].(map[string]interface{}) - if first["text"] != "# AgentFixture Documentation" { - t.Errorf("expected tool result text, got %v", first["text"]) + initLine := `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}` + "\n" + notifLine := `{"jsonrpc":"2.0","method":"notifications/initialized"}` + "\n" + listLine := `{"jsonrpc":"2.0","id":2,"method":"tools/list"}` + "\n" + replayCallJSON := fmt.Sprintf(`{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"analyze_blob","arguments":{"blob":%q,"api_key":"dummy"}}}`, hugeArg) + replayCallFramed := fmt.Sprintf("Content-Length: %d\r\n\r\n%s", len(replayCallJSON), replayCallJSON) + badRawLine := `raw "quoted" unmatched line` + "\n" + + combinedInput := initLine + notifLine + listLine + replayCallFramed + badRawLine + var replayOut bytes.Buffer + if err := replayer.ProcessStream(strings.NewReader(combinedInput), &replayOut); err != nil { + t.Fatalf("StdioReplayer.ProcessStream failed: %v", err) + } + + outStr := replayOut.String() + // Verify initialize synthesized, tools/list includes analyze_blob, Content-Length response emitted, and raw error is valid JSON + if !strings.Contains(outStr, `"protocolVersion":"2024-11-05"`) { + t.Errorf("expected synthesized initialize response, got: %s", outStr) + } + if !strings.Contains(outStr, `"analyze_blob"`) { + t.Errorf("expected synthesized tools/list to expose analyze_blob, got: %s", outStr) + } + if !strings.Contains(outStr, "Content-Length:") || !strings.Contains(outStr, `"blob_ok"`) { + t.Errorf("expected Content-Length framed tool response with blob_ok, got: %s", outStr) + } + if !strings.Contains(outStr, `\"quoted\"`) { + t.Errorf("expected properly JSON-escaped quotes in fallback error output, got: %s", outStr) } } diff --git a/pkg/proxy/stdio_proxy.go b/pkg/proxy/stdio_proxy.go index 31e4165..0c0d7e7 100644 --- a/pkg/proxy/stdio_proxy.go +++ b/pkg/proxy/stdio_proxy.go @@ -2,19 +2,36 @@ package proxy import ( "bufio" + "bytes" + "context" "encoding/json" "fmt" "io" + "os" + "os/exec" + "strconv" "strings" + "sync" + "time" "github.com/aloneangel45/AgentFixture/pkg/cassette" "github.com/aloneangel45/AgentFixture/pkg/matcher" ) -// JSONRPCRequest models a standard JSON-RPC 2.0 / MCP tool invocation message. +// MaxStdioMessageBytes is the maximum supported MCP stdio frame or line size (16 MB). +const MaxStdioMessageBytes = 16 * 1024 * 1024 + +type stdioFraming int + +const ( + framingNDJSON stdioFraming = iota + framingContentLength +) + +// JSONRPCRequest models a standard JSON-RPC 2.0 / MCP invocation or notification message. type JSONRPCRequest struct { JSONRPC string `json:"jsonrpc"` - ID interface{} `json:"id"` + ID interface{} `json:"id,omitempty"` Method string `json:"method"` Params json.RawMessage `json:"params,omitempty"` } @@ -33,11 +50,13 @@ type JSONRPCResponse struct { Error interface{} `json:"error,omitempty"` } -// StdioReplayer evaluates line-delimited JSON-RPC or tool invocations from an input reader +// StdioReplayer evaluates NDJSON or Content-Length framed JSON-RPC / MCP tool invocations // and streams deterministic cassette responses to an output writer. type StdioReplayer struct { - cas *cassette.Cassette - matcher matcher.Matcher + mu sync.Mutex + cas *cassette.Cassette + matcher matcher.Matcher + matchedIDs map[string]bool } // NewStdioReplayer creates an initialized StdioReplayer. @@ -45,32 +64,47 @@ func NewStdioReplayer(cas *cassette.Cassette, m matcher.Matcher) *StdioReplayer if m == nil { m = matcher.NewStrictMatcher() } + if cas == nil { + cas = cassette.NewCassette("cas_stdio_empty") + } return &StdioReplayer{ - cas: cas, - matcher: m, + cas: cas, + matcher: m, + matchedIDs: make(map[string]bool), } } -// ProcessStream reads line-by-line tool or JSON-RPC messages and writes out cassette answers. +// ProcessStream reads NDJSON or Content-Length framed messages up to 16MB and writes cassette responses. func (sr *StdioReplayer) ProcessStream(in io.Reader, out io.Writer) error { - scanner := bufio.NewScanner(in) - for scanner.Scan() { - line := scanner.Text() - trimmed := strings.TrimSpace(line) + br := bufio.NewReaderSize(in, 64*1024) + for { + payload, framing, err := readStdioFrame(br) + if err != nil { + if err == io.EOF { + return nil + } + return err + } + trimmed := strings.TrimSpace(string(payload)) if trimmed == "" { continue } + var rawObj map[string]json.RawMessage var rpcReq JSONRPCRequest - if err := json.Unmarshal([]byte(trimmed), &rpcReq); err == nil && rpcReq.Method != "" { - // Structured JSON-RPC / MCP call - if err := sr.handleJSONRPC(rpcReq, out); err != nil { + if json.Unmarshal([]byte(trimmed), &rawObj) == nil && json.Unmarshal([]byte(trimmed), &rpcReq) == nil && rpcReq.Method != "" { + _, hasID := rawObj["id"] + // JSON-RPC 2.0 & MCP spec: Notifications omit "id" (or pass null id on notification methods) and MUST NOT receive a response. + if !hasID || rpcReq.ID == nil { + continue + } + if err := sr.handleJSONRPC(rpcReq, out, framing); err != nil { return err } continue } - // Fallback: raw line or tool invocation + // Fallback: raw line or non-JSON-RPC tool invocation req := &cassette.Request{ RawBody: trimmed, } @@ -80,28 +114,39 @@ func (sr *StdioReplayer) ProcessStream(in io.Reader, out io.Writer) error { if result == "" { result = inter.Response.RawBody } - _, _ = fmt.Fprintln(out, result) + if err := writeStdioFrame(out, []byte(result), framing); err != nil { + return err + } } else { - _, _ = fmt.Fprintf(out, `{"error":"No cassette interaction matched input: %s"}`+"\n", trimmed) + errPayload, _ := json.Marshal(map[string]string{ + "error": "No cassette interaction matched input: " + trimmed, + }) + if err := writeStdioFrame(out, errPayload, framing); err != nil { + return err + } } } - - return scanner.Err() } -func (sr *StdioReplayer) handleJSONRPC(rpcReq JSONRPCRequest, out io.Writer) error { +func (sr *StdioReplayer) handleJSONRPC(rpcReq JSONRPCRequest, out io.Writer, framing stdioFraming) error { var toolName string var toolArgs map[string]interface{} if strings.Contains(rpcReq.Method, "tool") && len(rpcReq.Params) > 0 { - var p MCPToolParams - if err := json.Unmarshal(rpcReq.Params, &p); err == nil { - toolName = p.Name - toolArgs = p.Arguments + if parsed, err := matcher.DecodeJSONUseNumber(rpcReq.Params); err == nil { + if pMap, ok := parsed.(map[string]interface{}); ok { + if n, ok := pMap["name"].(string); ok { + toolName = n + } + if a, ok := pMap["arguments"].(map[string]interface{}); ok { + toolArgs = a + } + } } } candidate := &cassette.Request{ + Method: rpcReq.Method, ToolName: toolName, ToolInput: toolArgs, RawBody: string(rpcReq.Params), @@ -114,7 +159,12 @@ func (sr *StdioReplayer) handleJSONRPC(rpcReq JSONRPCRequest, out io.Writer) err } if inter != nil { - if inter.Response.ToolResult != "" { + if inter.Response.Error != "" && inter.Response.ToolResult == "" && inter.Response.Body == nil && inter.Response.RawBody == "" { + resp.Error = map[string]interface{}{ + "code": -32000, + "message": inter.Response.Error, + } + } else if inter.Response.ToolResult != "" { resp.Result = map[string]interface{}{ "content": []map[string]string{ { @@ -128,25 +178,452 @@ func (sr *StdioReplayer) handleJSONRPC(rpcReq JSONRPCRequest, out io.Writer) err } else { resp.Result = inter.Response.RawBody } + } else if synth, ok := sr.synthesizeProtocolResponse(rpcReq.Method); ok { + resp.Result = synth } else { + label := toolName + if label == "" { + label = rpcReq.Method + } resp.Error = map[string]interface{}{ "code": -32601, - "message": fmt.Sprintf("AgentFixture: no match found for tool %s", toolName), + "message": fmt.Sprintf("AgentFixture: no match found for %s", label), } } - b, _ := json.Marshal(resp) - _, err := fmt.Fprintln(out, string(b)) - return err + b, err := json.Marshal(resp) + if err != nil { + return err + } + return writeStdioFrame(out, b, framing) +} + +func (sr *StdioReplayer) synthesizeProtocolResponse(method string) (interface{}, bool) { + switch method { + case "ping": + return map[string]interface{}{}, true + case "initialize": + return map[string]interface{}{ + "protocolVersion": "2024-11-05", + "capabilities": map[string]interface{}{ + "tools": map[string]interface{}{}, + }, + "serverInfo": map[string]interface{}{ + "name": "AgentFixture-MCP-Replay", + "version": "1.1.0", + }, + }, true + case "tools/list": + sr.mu.Lock() + defer sr.mu.Unlock() + seen := make(map[string]bool) + tools := make([]map[string]interface{}, 0) + for _, inter := range sr.cas.Interactions { + name := strings.TrimSpace(inter.Request.ToolName) + if name == "" || seen[name] { + continue + } + seen[name] = true + tools = append(tools, map[string]interface{}{ + "name": name, + "description": "Recorded AgentFixture MCP tool", + "inputSchema": map[string]interface{}{ + "type": "object", + }, + }) + } + return map[string]interface{}{"tools": tools}, true + default: + return nil, false + } } func (sr *StdioReplayer) findMatch(candidate *cassette.Request) *cassette.Interaction { + sr.mu.Lock() + defer sr.mu.Unlock() + + var bestInter *cassette.Interaction + var bestScore float64 + + // Pass 1: Prefer unconsumed interactions first for stateful multi-turn tool calls. for i := range sr.cas.Interactions { inter := &sr.cas.Interactions[i] + if sr.matchedIDs[inter.ID] { + continue + } res := sr.matcher.Match(&inter.Request, candidate) - if res.Matched { - return inter + if res.Matched && res.Score > bestScore { + bestScore = res.Score + bestInter = inter + if res.Score == 1.0 { + break + } + } + } + + // Pass 2: Fallback to already-consumed interactions for idempotent retries. + if bestInter == nil { + for i := range sr.cas.Interactions { + inter := &sr.cas.Interactions[i] + if !sr.matchedIDs[inter.ID] { + continue + } + res := sr.matcher.Match(&inter.Request, candidate) + if res.Matched && res.Score > bestScore { + bestScore = res.Score + bestInter = inter + if res.Score == 1.0 { + break + } + } } } - return nil + + if bestInter != nil { + sr.matchedIDs[bestInter.ID] = true + } + return bestInter +} + +// StdioRecorder taps bidirectional JSON-RPC / MCP stdio traffic between a host client and a child MCP server process. +type StdioRecorder struct { + mu sync.Mutex + cas *cassette.Cassette + pending map[string]pendingRPC +} + +type pendingRPC struct { + req JSONRPCRequest + toolName string + toolArgs map[string]interface{} + sentAt time.Time +} + +// NewStdioRecorder creates an initialized StdioRecorder backed by cas. +func NewStdioRecorder(cas *cassette.Cassette) *StdioRecorder { + if cas == nil { + cas = cassette.NewCassette(fmt.Sprintf("cas_mcp_%d", time.Now().UnixNano())) + } + cas.Metadata.Source = "agentfixture_mcp_stdio_recorder" + return &StdioRecorder{ + cas: cas, + pending: make(map[string]pendingRPC), + } +} + +// GetCassette returns a thread-safe clone of the recorded MCP cassette. +func (r *StdioRecorder) GetCassette() *cassette.Cassette { + r.mu.Lock() + defer r.mu.Unlock() + return r.cas.Clone() +} + +// RecordCommand launches cmdArgs as an MCP stdio child process, bridges clientIn -> child.Stdin +// and child.Stdout -> clientOut while recording all JSON-RPC interactions into the cassette. +func (r *StdioRecorder) RecordCommand(ctx context.Context, cmdArgs []string, env []string, clientIn io.Reader, clientOut io.Writer) (*cassette.Cassette, error) { + if len(cmdArgs) == 0 { + return nil, fmt.Errorf("mcp command cannot be empty") + } + cmd := exec.CommandContext(ctx, cmdArgs[0], cmdArgs[1:]...) + if len(env) > 0 { + cmd.Env = env + } + cmd.Stderr = os.Stderr + + childStdin, err := cmd.StdinPipe() + if err != nil { + return nil, fmt.Errorf("failed to attach mcp stdin pipe: %w", err) + } + childStdout, err := cmd.StdoutPipe() + if err != nil { + return nil, fmt.Errorf("failed to attach mcp stdout pipe: %w", err) + } + + if err := cmd.Start(); err != nil { + return nil, fmt.Errorf("failed to start mcp server %q: %w", cmdArgs[0], err) + } + + errCh := make(chan error, 1) + go func() { + errCh <- r.pumpResponses(childStdout, clientOut) + }() + + go func() { + _ = r.pumpRequests(clientIn, childStdin) + _ = childStdin.Close() + }() + + respErr := <-errCh + waitErr := cmd.Wait() + if respErr != nil && respErr != io.EOF { + return r.GetCassette(), respErr + } + if waitErr != nil && ctx.Err() == nil { + return r.GetCassette(), waitErr + } + return r.GetCassette(), nil +} + +// RecordStreams bridges two io.Reader/Writer pairs directly (useful for in-process testing without spawning an OS process). +func (r *StdioRecorder) RecordStreams(clientIn io.Reader, serverIn io.WriteCloser, serverOut io.Reader, clientOut io.Writer) (*cassette.Cassette, error) { + var wg sync.WaitGroup + var respErr error + + wg.Add(1) + go func() { + defer wg.Done() + respErr = r.pumpResponses(serverOut, clientOut) + }() + + _ = r.pumpRequests(clientIn, serverIn) + _ = serverIn.Close() + wg.Wait() + + if respErr != nil && respErr != io.EOF { + return r.GetCassette(), respErr + } + return r.GetCassette(), nil +} + +func (r *StdioRecorder) pumpRequests(src io.Reader, dst io.Writer) error { + br := bufio.NewReaderSize(src, 64*1024) + for { + payload, framing, err := readStdioFrame(br) + if err != nil { + return err + } + trimmed := bytes.TrimSpace(payload) + if len(trimmed) > 0 { + var rawObj map[string]json.RawMessage + var rpcReq JSONRPCRequest + if json.Unmarshal(trimmed, &rawObj) == nil && json.Unmarshal(trimmed, &rpcReq) == nil && rpcReq.Method != "" { + if _, hasID := rawObj["id"]; hasID && rpcReq.ID != nil { + var toolName string + var toolArgs map[string]interface{} + if strings.Contains(rpcReq.Method, "tool") && len(rpcReq.Params) > 0 { + if parsed, pErr := matcher.DecodeJSONUseNumber(rpcReq.Params); pErr == nil { + if pMap, ok := parsed.(map[string]interface{}); ok { + toolName, _ = pMap["name"].(string) + toolArgs, _ = pMap["arguments"].(map[string]interface{}) + } + } + } + key := rpcIDKey(rpcReq.ID) + r.mu.Lock() + r.pending[key] = pendingRPC{ + req: rpcReq, + toolName: toolName, + toolArgs: toolArgs, + sentAt: time.Now(), + } + r.mu.Unlock() + } + } + } + if err := writeStdioFrame(dst, payload, framing); err != nil { + return err + } + } +} + +func (r *StdioRecorder) pumpResponses(src io.Reader, dst io.Writer) error { + br := bufio.NewReaderSize(src, 64*1024) + for { + payload, framing, err := readStdioFrame(br) + if err != nil { + return err + } + trimmed := bytes.TrimSpace(payload) + if len(trimmed) > 0 { + var rpcResp JSONRPCResponse + if json.Unmarshal(trimmed, &rpcResp) == nil && rpcResp.ID != nil { + key := rpcIDKey(rpcResp.ID) + r.mu.Lock() + pend, ok := r.pending[key] + if ok { + delete(r.pending, key) + } + r.mu.Unlock() + + if ok { + r.recordPairedInteraction(pend, rpcResp) + } + } + } + if err := writeStdioFrame(dst, payload, framing); err != nil { + return err + } + } +} + +func (r *StdioRecorder) recordPairedInteraction(pend pendingRPC, rpcResp JSONRPCResponse) { + latency := time.Since(pend.sentAt).Milliseconds() + interType := cassette.InteractionTypeStdio + if pend.toolName != "" { + interType = cassette.InteractionTypeTool + } + + redArgs := cassette.RedactMap(pend.toolArgs) + rawParams := cassette.RedactString(string(pend.req.Params)) + + req := cassette.Request{ + Method: pend.req.Method, + ToolName: pend.toolName, + ToolInput: redArgs, + RawBody: rawParams, + } + req.Matcher.StrictHash = matcher.ComputeRequestHash(&req) + if pend.toolName != "" { + req.Matcher.FuzzySignature = cassette.FuzzySignature{ + NormalizedToolName: strings.ToLower(pend.toolName), + NormalizedArgs: redArgs, + } + } + + var resp cassette.Response + resp.LatencyMs = latency + + if rpcResp.Error != nil { + resp.StatusCode = 500 + if errMap, ok := rpcResp.Error.(map[string]interface{}); ok { + if msg, ok := errMap["message"].(string); ok { + resp.Error = cassette.RedactString(msg) + } + } + if resp.Error == "" { + b, _ := json.Marshal(rpcResp.Error) + resp.Error = cassette.RedactString(string(b)) + } + } else { + resp.StatusCode = 0 + if textResult, ok := extractMCPTextContent(rpcResp.Result); ok && pend.toolName != "" { + resp.ToolResult = cassette.RedactString(textResult) + } else { + resp.Body = cassette.RedactValue(rpcResp.Result) + } + } + + r.mu.Lock() + seq := len(r.cas.Interactions) + 1 + inter := cassette.Interaction{ + ID: fmt.Sprintf("mcp_%03d", seq), + Sequence: seq, + Type: interType, + Request: req, + Response: resp, + } + cassette.RedactInteraction(&inter) + r.cas.Interactions = append(r.cas.Interactions, inter) + r.mu.Unlock() +} + +func extractMCPTextContent(result interface{}) (string, bool) { + m, ok := result.(map[string]interface{}) + if !ok { + return "", false + } + contentSlice, ok := m["content"].([]interface{}) + if !ok || len(contentSlice) != 1 { + return "", false + } + first, ok := contentSlice[0].(map[string]interface{}) + if !ok { + return "", false + } + if t, _ := first["type"].(string); t != "text" { + return "", false + } + txt, ok := first["text"].(string) + return txt, ok +} + +func rpcIDKey(id interface{}) string { + b, _ := json.Marshal(id) + return string(b) +} + +func readStdioFrame(br *bufio.Reader) ([]byte, stdioFraming, error) { + for { + line, err := readBoundedLine(br, MaxStdioMessageBytes) + if err != nil { + if err == io.EOF && len(line) == 0 { + return nil, framingNDJSON, io.EOF + } + if err != io.EOF { + return nil, framingNDJSON, err + } + } + + trimmed := strings.TrimRight(string(line), "\r\n") + if trimmed == "" { + if err == io.EOF { + return nil, framingNDJSON, io.EOF + } + continue + } + + if strings.HasPrefix(strings.ToLower(trimmed), "content-length:") { + valStr := strings.TrimSpace(trimmed[len("content-length:"):]) + n, convErr := strconv.Atoi(valStr) + if convErr != nil || n < 0 { + return nil, framingContentLength, fmt.Errorf("invalid Content-Length header %q", trimmed) + } + if n > MaxStdioMessageBytes { + return nil, framingContentLength, fmt.Errorf("MCP Content-Length %d exceeds 16MB limit", n) + } + + // Consume remaining MIME headers until blank line ("\r\n" or "\n") + for { + hdrLine, hdrErr := readBoundedLine(br, 64*1024) + if hdrErr != nil { + return nil, framingContentLength, hdrErr + } + if strings.TrimRight(string(hdrLine), "\r\n") == "" { + break + } + } + + payload := make([]byte, n) + if _, readErr := io.ReadFull(br, payload); readErr != nil { + return nil, framingContentLength, readErr + } + return payload, framingContentLength, nil + } + + return []byte(trimmed), framingNDJSON, nil + } +} + +func readBoundedLine(br *bufio.Reader, maxBytes int) ([]byte, error) { + var buf []byte + for { + frag, isPrefix, err := br.ReadLine() + buf = append(buf, frag...) + if len(buf) > maxBytes { + return nil, fmt.Errorf("stdio message line exceeds maximum %d bytes", maxBytes) + } + if err != nil { + return buf, err + } + if !isPrefix { + return buf, nil + } + } +} + +func writeStdioFrame(out io.Writer, payload []byte, framing stdioFraming) error { + if framing == framingContentLength { + header := fmt.Sprintf("Content-Length: %d\r\n\r\n", len(payload)) + if _, err := io.WriteString(out, header); err != nil { + return err + } + _, err := out.Write(payload) + return err + } + var buf bytes.Buffer + buf.Grow(len(payload) + 1) + buf.Write(payload) + buf.WriteByte('\n') + _, err := out.Write(buf.Bytes()) + return err } diff --git a/pkg/proxy/testing.go b/pkg/proxy/testing.go new file mode 100644 index 0000000..c21de6b --- /dev/null +++ b/pkg/proxy/testing.go @@ -0,0 +1,172 @@ +package proxy + +import ( + "net/http" + "testing" + + "github.com/aloneangel45/AgentFixture/pkg/cassette" + "github.com/aloneangel45/AgentFixture/pkg/matcher" +) + +// TestOption configures a VCR TestServer created via NewTestServer. +type TestOption func(*testServerConfig) + +type testServerConfig struct { + matcher matcher.Matcher + sseTiming cassette.SSETimingConfig + strictAllMatched bool + allowUnmatched bool + mode Mode + upstreamURL string +} + +// WithTestMatcher overrides the request Matcher (default: StrictMatcher). +func WithTestMatcher(m matcher.Matcher) TestOption { + return func(c *testServerConfig) { + c.matcher = m + } +} + +// WithTestSSETiming sets the SSE chunk replay timing configuration. +func WithTestSSETiming(cfg cassette.SSETimingConfig) TestOption { + return func(c *testServerConfig) { + c.sseTiming = cfg + } +} + +// WithTestStrictExhaustive fails tb.Cleanup if any cassette interaction was not exercised. +func WithTestStrictExhaustive(strict bool) TestOption { + return func(c *testServerConfig) { + c.strictAllMatched = strict + } +} + +// WithTestAllowUnmatched disables automatic tb.Cleanup failure when unexpected requests arrive. +func WithTestAllowUnmatched(allow bool) TestOption { + return func(c *testServerConfig) { + c.allowUnmatched = allow + } +} + +// WithTestRecordMode switches the TestServer into live recording mode against upstreamURL +// and automatically saves the cassette on tb.Cleanup. +func WithTestRecordMode(upstreamURL string) TestOption { + return func(c *testServerConfig) { + c.mode = ModeRecord + c.upstreamURL = upstreamURL + } +} + +// TestServer wraps a deterministic AgentFixture VCR Server bound to a Go testing.TB lifecycle. +type TestServer struct { + tb testing.TB + path string + srv *Server + client *http.Client + cfg testServerConfig +} + +// NewTestServer starts a zero-network deterministic VCR server for Go unit and integration tests. +func NewTestServer(tb testing.TB, cassettePath string, opts ...TestOption) *TestServer { + tb.Helper() + + cfg := testServerConfig{ + matcher: matcher.NewStrictMatcher(), + mode: ModeReplay, + sseTiming: cassette.SSETimingConfig{ + Mode: cassette.SSETimingInstant, + }, + } + for _, opt := range opts { + if opt != nil { + opt(&cfg) + } + } + + var cas *cassette.Cassette + var err error + if cfg.mode == ModeRecord { + cas = cassette.NewCassette(tb.Name()) + cas.Metadata.Source = "agentfixture_go_sdk" + } else { + cas, err = cassette.Load(cassettePath) + if err != nil { + tb.Fatalf("agentfixture: failed to load cassette %q: %v", cassettePath, err) + } + } + + srv, err := NewServer(Config{ + Mode: cfg.mode, + UpstreamURL: cfg.upstreamURL, + Cassette: cas, + Matcher: cfg.matcher, + SSETiming: cfg.sseTiming, + }) + if err != nil { + tb.Fatalf("agentfixture: failed to initialize test server: %v", err) + } + if err := srv.Start(); err != nil { + tb.Fatalf("agentfixture: failed to start test server: %v", err) + } + + ts := &TestServer{ + tb: tb, + path: cassettePath, + srv: srv, + client: NewOfflineHTTPClient(srv.URL()), + cfg: cfg, + } + + tb.Cleanup(func() { + _ = srv.Close() + if cfg.mode == ModeRecord && cassettePath != "" { + if saveErr := cassette.Save(cassettePath, srv.GetCassette()); saveErr != nil { + tb.Errorf("agentfixture: failed to save recorded cassette to %q: %v", cassettePath, saveErr) + } + return + } + if tb.Failed() { + return + } + if !cfg.allowUnmatched { + unmatched := srv.UnmatchedRequests() + if len(unmatched) > 0 { + tb.Errorf("agentfixture: %d unexpected request(s) failed to match cassette %q (first: %s %s)", + len(unmatched), cassettePath, unmatched[0].Method, unmatched[0].Endpoint) + } + } + if cfg.strictAllMatched { + ts.AssertAllMatched() + } + }) + + return ts +} + +// URL returns the base loopback URL (e.g. "http://127.0.0.1:54321") for ANTHROPIC_BASE_URL / OPENAI_BASE_URL. +func (ts *TestServer) URL() string { + return ts.srv.URL() +} + +// HTTPClient returns a zero-outbound-network *http.Client locked strictly to loopback dials. +func (ts *TestServer) HTTPClient() *http.Client { + return ts.client +} + +// Server exposes the underlying *Server instance. +func (ts *TestServer) Server() *Server { + return ts.srv +} + +// AssertAllMatched fails the test if any recorded interaction in the cassette went uncalled. +func (ts *TestServer) AssertAllMatched() { + ts.tb.Helper() + cas := ts.srv.GetCassette() + matched := ts.srv.MatchedIDs() + for _, inter := range cas.Interactions { + if !matched[inter.ID] { + ts.tb.Errorf("agentfixture: expected cassette interaction %q (%s %s) was never called", + inter.ID, inter.Request.Method, inter.Request.Endpoint) + } + } +} diff --git a/pkg/runner/runner.go b/pkg/runner/runner.go index d24b2c2..434c371 100644 --- a/pkg/runner/runner.go +++ b/pkg/runner/runner.go @@ -2,9 +2,12 @@ package runner import ( "bytes" + "context" + "errors" "fmt" "os" "os/exec" + "runtime" "strings" "time" @@ -21,7 +24,7 @@ type DriftInfo struct { Reason string `json:"reason"` } -// TestReport captures the full assertion results of a regression run. +// TestReport captures the full assertion results of a regression or recording run. type TestReport struct { CassetteID string `json:"cassette_id"` TotalExpected int `json:"total_expected"` @@ -48,30 +51,30 @@ func (r *TestReport) FormatSummary() string { if !r.Passed { status = "FAILED (REGRESSION DETECTED)" } - sb.WriteString(fmt.Sprintf(" Status: %s\n", status)) - sb.WriteString(fmt.Sprintf(" Exit Code: %d\n", r.CommandExitCode)) - sb.WriteString(fmt.Sprintf(" Interactions Replayed: %d / %d (%.1f%%)\n", + sb.WriteString(fmt.Sprintf(" Status: %s\n", status)) + sb.WriteString(fmt.Sprintf(" Exit Code: %d\n", r.CommandExitCode)) + sb.WriteString(fmt.Sprintf(" Interactions Replayed: %d / %d (%.1f%%)\n", r.MatchedCount, r.TotalExpected, float64(r.MatchedCount)/float64(max(1, r.TotalExpected))*100)) sb.WriteString(fmt.Sprintf(" Estimated Tokens Saved: %d\n", r.TokensSaved)) sb.WriteString(fmt.Sprintf(" Estimated Cost Saved: $%.4f USD\n", r.CostSavedUSD)) sb.WriteString(fmt.Sprintf(" Replay Run Duration: %v\n", r.Duration)) if len(r.UnmatchedExpected) > 0 { - sb.WriteString("\n [!] Missing Expected Interactions (Cassette items not called):\n") + sb.WriteString("\n [WARN] Missing Expected Interactions (Cassette items not called):\n") for _, id := range r.UnmatchedExpected { sb.WriteString(fmt.Sprintf(" - %s\n", id)) } } if len(r.UnexpectedCalls) > 0 { - sb.WriteString(fmt.Sprintf("\n [!] Unexpected Calls (%d unmatched requests):\n", len(r.UnexpectedCalls))) + sb.WriteString(fmt.Sprintf("\n [FAIL] Unexpected Calls (%d unmatched requests):\n", len(r.UnexpectedCalls))) for i, req := range r.UnexpectedCalls { sb.WriteString(fmt.Sprintf(" %d. %s %s %s\n", i+1, req.Method, req.Endpoint, req.ToolName)) } } if len(r.DriftDetails) > 0 { - sb.WriteString("\n [!] Semantic Drift Warnings:\n") + sb.WriteString("\n [WARN] Semantic Drift Warnings:\n") for _, d := range r.DriftDetails { sb.WriteString(fmt.Sprintf(" - %s (%s, sim: %.2f): %s\n", d.InteractionID, d.ToolOrPath, d.Similarity, d.Reason)) } @@ -81,19 +84,24 @@ func (r *TestReport) FormatSummary() string { return sb.String() } -// RunOptions configures the regression test execution. +// RunOptions configures the regression or recording command execution. type RunOptions struct { - Cassette *cassette.Cassette - Matcher matcher.Matcher - Command []string - Env map[string]string - StrictMode bool - Timeout time.Duration + Cassette *cassette.Cassette + Matcher matcher.Matcher + Command []string + Env map[string]string + StrictMode bool + Timeout time.Duration + Mode proxy.Mode + UpstreamURL string + Server *proxy.Server + SSETiming cassette.SSETimingConfig } -// Run executes the command against a deterministic replay proxy and asserts cassette expectations. +// Run executes the target command against a deterministic replay or live recording proxy +// and asserts cassette expectations. func Run(opts RunOptions) (*TestReport, error) { - if opts.Cassette == nil { + if opts.Cassette == nil && opts.Server == nil { return nil, fmt.Errorf("cassette cannot be nil") } if len(opts.Command) == 0 { @@ -102,42 +110,83 @@ func Run(opts RunOptions) (*TestReport, error) { if opts.Matcher == nil { opts.Matcher = matcher.NewStrictMatcher() } - if opts.Timeout == 0 { + if opts.Timeout <= 0 { opts.Timeout = 60 * time.Second } - // 1. Initialize deterministic replay proxy - srv, err := proxy.NewServer(proxy.Config{ - Mode: proxy.ModeReplay, - Cassette: opts.Cassette, - Matcher: opts.Matcher, - SimulateDelay: false, // 0 delay for instant CI runs - }) - if err != nil { - return nil, fmt.Errorf("failed to start replay server: %w", err) + mode := opts.Mode + if mode == "" { + if opts.Server != nil { + mode = opts.Server.Mode() + } else { + mode = proxy.ModeReplay + } + } + + // 1. Acquire or start proxy server + srv := opts.Server + ownsServer := false + if srv == nil { + var err error + srv, err = proxy.NewServer(proxy.Config{ + Mode: mode, + UpstreamURL: opts.UpstreamURL, + Cassette: opts.Cassette, + Matcher: opts.Matcher, + SimulateDelay: false, + SSETiming: opts.SSETiming, + }) + if err != nil { + return nil, fmt.Errorf("failed to create %s server: %w", mode, err) + } + if err := srv.Start(); err != nil { + return nil, fmt.Errorf("failed to bind %s server: %w", mode, err) + } + ownsServer = true + } else if srv.Addr() == "" { + if err := srv.Start(); err != nil { + return nil, fmt.Errorf("failed to bind provided %s server: %w", mode, err) + } + ownsServer = true } - if err := srv.Start(); err != nil { - return nil, fmt.Errorf("failed to bind replay server: %w", err) + if ownsServer { + defer srv.Close() } - defer srv.Close() - // 2. Prepare environment variables + // 2. Prepare deduplicated environment variables (case-insensitive on Windows!) proxyURL := srv.URL() - envVars := os.Environ() - envVars = append(envVars, - "AGENTFIXTURE_PROXY_URL="+proxyURL, - "ANTHROPIC_BASE_URL="+proxyURL, - "OPENAI_BASE_URL="+proxyURL, - "HTTP_PROXY="+proxyURL, - "http_proxy="+proxyURL, - ) + overrides := map[string]string{ + "AGENTFIXTURE_PROXY_URL": proxyURL, + "ANTHROPIC_BASE_URL": proxyURL, + "OPENAI_BASE_URL": proxyURL, + "HTTP_PROXY": proxyURL, + "NO_PROXY": "127.0.0.1,localhost,::1", + } + if runtime.GOOS != "windows" { + overrides["http_proxy"] = proxyURL + overrides["no_proxy"] = "127.0.0.1,localhost,::1" + } for k, v := range opts.Env { - envVars = append(envVars, fmt.Sprintf("%s=%s", k, v)) + overrides[k] = v + } + + var fallbacks map[string]string + if mode == proxy.ModeReplay { + // Provide dummy SDK keys in replay mode when unset so agent SDKs initialize cleanly in zero-secret CI. + fallbacks = map[string]string{ + "ANTHROPIC_API_KEY": "af-replay-offline-key", + "OPENAI_API_KEY": "af-replay-offline-key", + } } - // 3. Execute target command + envVars := buildChildEnv(os.Environ(), overrides, fallbacks, runtime.GOOS == "windows") + + // 3. Execute target command with context timeout + ctx, cancel := context.WithTimeout(context.Background(), opts.Timeout) + defer cancel() + startTime := time.Now() - cmd := exec.Command(opts.Command[0], opts.Command[1:]...) + cmd := exec.CommandContext(ctx, opts.Command[0], opts.Command[1:]...) cmd.Env = envVars var outputBuf bytes.Buffer @@ -149,37 +198,77 @@ func Run(opts RunOptions) (*TestReport, error) { exitCode := 0 if runErr != nil { - if exitErr, ok := runErr.(*exec.ExitError); ok { + var exitErr *exec.ExitError + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + exitCode = 124 + } else if errors.As(runErr, &exitErr) { exitCode = exitErr.ExitCode() } else { exitCode = 1 } } - // 4. Collect assertions - cas := opts.Cassette - cas.ComputeMetrics() + // 4. Drain any in-flight proxy handler goroutines before reading cassette state + srv.WaitIdle() + activeCas := srv.GetCassette() + activeCas.ComputeMetrics() + + if mode == proxy.ModeRecord { + if opts.Cassette != nil { + *opts.Cassette = *activeCas + } + return &TestReport{ + CassetteID: activeCas.Metadata.CassetteID, + TotalExpected: len(activeCas.Interactions), + MatchedCount: len(activeCas.Interactions), + CommandExitCode: exitCode, + CommandOutput: outputBuf.String(), + TokensSaved: activeCas.Metadata.Metrics.EstimatedTokensSaved, + CostSavedUSD: activeCas.Metadata.Metrics.EstimatedCostSavedUSD, + Duration: duration, + Passed: exitCode == 0, + }, nil + } + + matchedIDs := srv.MatchedIDs() + var unmatchedExpected []string + for _, inter := range activeCas.Interactions { + if !matchedIDs[inter.ID] { + unmatchedExpected = append(unmatchedExpected, inter.ID) + } + } + + rawDrifts := srv.DriftRecords() + var driftDetails []DriftInfo + for _, d := range rawDrifts { + driftDetails = append(driftDetails, DriftInfo{ + InteractionID: d.InteractionID, + ToolOrPath: d.ToolOrPath, + Similarity: d.Similarity, + Reason: d.Reason, + }) + } report := &TestReport{ - CassetteID: cas.Metadata.CassetteID, - TotalExpected: len(cas.Interactions), - MatchedCount: srv.MatchedCount(), - CommandExitCode: exitCode, - CommandOutput: outputBuf.String(), - TokensSaved: cas.Metadata.Metrics.EstimatedTokensSaved, - CostSavedUSD: cas.Metadata.Metrics.EstimatedCostSavedUSD, - Duration: duration, - Passed: exitCode == 0, - } - - // Check for unexpected calls + CassetteID: activeCas.Metadata.CassetteID, + TotalExpected: len(activeCas.Interactions), + MatchedCount: srv.MatchedCount(), + UnmatchedExpected: unmatchedExpected, + DriftDetails: driftDetails, + CommandExitCode: exitCode, + CommandOutput: outputBuf.String(), + TokensSaved: activeCas.Metadata.Metrics.EstimatedTokensSaved, + CostSavedUSD: activeCas.Metadata.Metrics.EstimatedCostSavedUSD, + Duration: duration, + Passed: exitCode == 0, + } + unmatchedCalls := srv.UnmatchedRequests() if len(unmatchedCalls) > 0 { report.UnexpectedCalls = unmatchedCalls report.Passed = false } - // In strict mode, verify all cassette interactions were exercised if opts.StrictMode && report.MatchedCount < report.TotalExpected { report.Passed = false } @@ -187,6 +276,66 @@ func Run(opts RunOptions) (*TestReport, error) { return report, nil } +type envEntry struct { + key string + val string +} + +// buildChildEnv merges base OS environment variables with overrides and fallbacks, +// performing case-insensitive deduplication on Windows so duplicate keys (e.g. Path/PATH, http_proxy/HTTP_PROXY) never collide. +func buildChildEnv(base []string, overrides map[string]string, fallbacks map[string]string, caseInsensitive bool) []string { + normKey := func(k string) string { + if caseInsensitive { + return strings.ToUpper(k) + } + return k + } + + indexByNorm := make(map[string]int, len(base)+len(overrides)+len(fallbacks)) + entries := make([]envEntry, 0, len(base)+len(overrides)+len(fallbacks)) + + setVar := func(k, v string) { + nk := normKey(k) + if idx, exists := indexByNorm[nk]; exists { + entries[idx] = envEntry{key: k, val: v} + } else { + indexByNorm[nk] = len(entries) + entries = append(entries, envEntry{key: k, val: v}) + } + } + + for _, kv := range base { + eq := strings.IndexByte(kv, '=') + if eq <= 0 { + // Preserve special Windows drive vars like "=C:=C:\..." + entries = append(entries, envEntry{key: "", val: kv}) + continue + } + setVar(kv[:eq], kv[eq+1:]) + } + + for k, v := range overrides { + setVar(k, v) + } + + for k, v := range fallbacks { + nk := normKey(k) + if idx, exists := indexByNorm[nk]; !exists || strings.TrimSpace(entries[idx].val) == "" { + setVar(k, v) + } + } + + result := make([]string, 0, len(entries)) + for _, e := range entries { + if e.key == "" { + result = append(result, e.val) + } else { + result = append(result, e.key+"="+e.val) + } + } + return result +} + func max(a, b int) int { if a > b { return a diff --git a/pkg/runner/runner_test.go b/pkg/runner/runner_test.go index 1a5d019..e626aaf 100644 --- a/pkg/runner/runner_test.go +++ b/pkg/runner/runner_test.go @@ -1,15 +1,18 @@ package runner import ( + "net/http" + "net/http/httptest" "strings" "testing" "time" "github.com/aloneangel45/AgentFixture/pkg/cassette" "github.com/aloneangel45/AgentFixture/pkg/matcher" + "github.com/aloneangel45/AgentFixture/pkg/proxy" ) -func TestRunner_BasicExecution(t *testing.T) { +func TestRunner_BasicExecutionAndStrictUnmatched(t *testing.T) { cas := cassette.NewCassette("cas_runner_test") cas.Interactions = append(cas.Interactions, cassette.Interaction{ ID: "int_001", @@ -32,24 +35,118 @@ func TestRunner_BasicExecution(t *testing.T) { Command: []string{"go", "version"}, Timeout: 10 * time.Second, }) - if err != nil { t.Fatalf("unexpected error running regression runner: %v", err) } - if !report.Passed { - t.Errorf("expected report to pass for valid command, got failed") + t.Errorf("expected report to pass when StrictMode=false, got failed") } - if report.CommandExitCode != 0 { - t.Errorf("expected exit code 0, got %d", report.CommandExitCode) + if len(report.UnmatchedExpected) != 1 || report.UnmatchedExpected[0] != "int_001" { + t.Errorf("expected UnmatchedExpected=[int_001], got %+v", report.UnmatchedExpected) } if !strings.Contains(report.CommandOutput, "go version") { t.Errorf("expected command output to contain go version, got %q", report.CommandOutput) } - summary := report.FormatSummary() - if !strings.Contains(summary, "PASSED") { - t.Errorf("expected summary to contain PASSED, got: %s", summary) + // With StrictMode=true, unexercised int_001 fails report.Passed + strictReport, err := Run(RunOptions{ + Cassette: cas, + Matcher: matcher.NewStrictMatcher(), + Command: []string{"go", "version"}, + StrictMode: true, + Timeout: 10 * time.Second, + }) + if err != nil { + t.Fatal(err) + } + if strictReport.Passed { + t.Errorf("expected StrictMode=true to fail when cassette interaction is uncalled") + } +} + +func TestRunner_BuildChildEnv_CaseInsensitiveWindowsDeduplication(t *testing.T) { + base := []string{ + "=C:=C:\\Work", + "Path=C:\\Windows\\System32", + "http_proxy=http://old-proxy:8080", + "OPENAI_API_KEY=", + } + overrides := map[string]string{ + "HTTP_PROXY": "http://127.0.0.1:5555", + "NO_PROXY": "127.0.0.1,localhost", + } + fallbacks := map[string]string{ + "ANTHROPIC_API_KEY": "af-dummy", + "OPENAI_API_KEY": "af-dummy-openai", + } + + merged := buildChildEnv(base, overrides, fallbacks, true) + proxyCount := 0 + hasAnthropic := false + hasOpenAI := false + for _, kv := range merged { + upper := strings.ToUpper(kv) + if strings.HasPrefix(upper, "HTTP_PROXY=") { + proxyCount++ + if kv != "HTTP_PROXY=http://127.0.0.1:5555" { + t.Errorf("unexpected HTTP_PROXY entry: %s", kv) + } + } + if kv == "ANTHROPIC_API_KEY=af-dummy" { + hasAnthropic = true + } + if kv == "OPENAI_API_KEY=af-dummy-openai" { + hasOpenAI = true + } + } + if proxyCount != 1 { + t.Errorf("expected case-insensitive deduplication of http_proxy/HTTP_PROXY to 1 entry, got %d (%v)", proxyCount, merged) + } + if !hasAnthropic || !hasOpenAI { + t.Errorf("expected fallback dummy API keys to be populated when missing/empty: %v", merged) + } +} + +func TestRunner_RecordModeWithLiveServer(t *testing.T) { + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"recorded":true}`)) + })) + defer upstream.Close() + + cas := cassette.NewCassette("rec_runner") + srv, err := proxy.NewServer(proxy.Config{ + Mode: proxy.ModeRecord, + UpstreamURL: upstream.URL, + Cassette: cas, + }) + if err != nil { + t.Fatal(err) + } + if err := srv.Start(); err != nil { + t.Fatal(err) + } + defer srv.Close() + + // Make a request during command execution via Go test helper or verify Server wiring + resp, err := http.Get(srv.URL() + "/ping") + if err != nil { + t.Fatal(err) + } + _ = resp.Body.Close() + + rep, err := Run(RunOptions{ + Cassette: cas, + Mode: proxy.ModeRecord, + Server: srv, + Command: []string{"go", "version"}, + }) + if err != nil { + t.Fatal(err) + } + if !rep.Passed || rep.MatchedCount != 1 || len(cas.Interactions) != 1 { + t.Fatalf("expected recorded interaction preserved in ModeRecord runner, got report=%+v len=%d", rep, len(cas.Interactions)) } } @@ -65,6 +162,14 @@ func TestTestReport_FormatSummary_RegressionDetected(t *testing.T) { Endpoint: "/v1/unexpected", }, }, + DriftDetails: []DriftInfo{ + { + InteractionID: "int_001", + ToolOrPath: "/v1/messages", + Similarity: 0.88, + Reason: "fuzzy match", + }, + }, CommandExitCode: 1, Passed: false, } @@ -79,4 +184,7 @@ func TestTestReport_FormatSummary_RegressionDetected(t *testing.T) { if !strings.Contains(summary, "/v1/unexpected") { t.Errorf("expected unexpected call listed, got: %s", summary) } + if !strings.Contains(summary, "0.88") { + t.Errorf("expected drift details listed, got: %s", summary) + } } diff --git a/test/sse_determinism_test.go b/test/sse_determinism_test.go new file mode 100644 index 0000000..2ce358c --- /dev/null +++ b/test/sse_determinism_test.go @@ -0,0 +1,204 @@ +package test + +import ( + "bytes" + "errors" + "io" + "net/http" + "net/http/httptest" + "path/filepath" + "reflect" + "strings" + "sync/atomic" + "testing" + + "github.com/aloneangel45/AgentFixture/pkg/cassette" + "github.com/aloneangel45/AgentFixture/pkg/matcher" + "github.com/aloneangel45/AgentFixture/pkg/proxy" +) + +func TestSSEDeterminism_10xReplayAndZeroOutboundNetwork(t *testing.T) { + var upstreamHits atomic.Int64 + + // 1. Live mock Anthropic Messages API SSE upstream emitting thinking, citations, text, and tool_use JSON deltas + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + upstreamHits.Add(1) + w.Header().Set("Content-Type", "text/event-stream; charset=utf-8") + w.WriteHeader(http.StatusOK) + flusher, _ := w.(http.Flusher) + + frames := []string{ + ": keep-alive\n\n", + "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_det_001\",\"type\":\"message\",\"role\":\"assistant\",\"model\":\"claude-sonnet-4-20250514\",\"content\":[],\"stop_reason\":null,\"usage\":{\"input_tokens\":512,\"cache_creation_input_tokens\":64,\"cache_read_input_tokens\":128,\"output_tokens\":1}}}\n\n", + "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"thinking\",\"thinking\":\"\",\"signature\":\"\"}}\n\n", + "event: ping\ndata: {\"type\":\"ping\"}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"thinking_delta\",\"thinking\":\"Checking repository files \"}}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"thinking_delta\",\"thinking\":\"for deterministic replay.\"}}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"signature_delta\",\"signature\":\"sig_deterministic_v1\"}}\n\n", + "event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\n", + "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":1,\"content_block\":{\"type\":\"text\",\"text\":\"\"}}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":1,\"delta\":{\"type\":\"text_delta\",\"text\":\"Executing search_code tool now.\"}}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":1,\"delta\":{\"type\":\"citations_delta\",\"citation\":{\"type\":\"char_location\",\"cited_text\":\"ReplaySSEStream\",\"document_index\":0,\"start_char_index\":0,\"end_char_index\":15}}}\n\n", + "event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":1}\n\n", + "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":2,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_det_99\",\"name\":\"search_code\",\"input\":{}}}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":2,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"pattern\\\": \\\"ReplaySSE\"}}\n\n", + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":2,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"Stream\\\", \\\"max_results\\\": 25}\"}}\n\n", + "event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":2}\n\n", + "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null},\"usage\":{\"output_tokens\":118}}\n\n", + "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n", + } + for _, f := range frames { + _, _ = w.Write([]byte(f)) + if flusher != nil { + flusher.Flush() + } + } + })) + + // 2. Record the streaming interaction into a YAML cassette + casPath := filepath.Join(t.TempDir(), "anthropic_sse_determinism.yaml") + recordSrv, err := proxy.NewServer(proxy.Config{ + Mode: proxy.ModeRecord, + UpstreamURL: upstream.URL, + }) + if err != nil { + t.Fatalf("failed to create record server: %v", err) + } + if err := recordSrv.Start(); err != nil { + t.Fatalf("failed to start record server: %v", err) + } + + reqPayload := `{"model":"claude-sonnet-4-20250514","max_tokens":1024,"stream":true,"messages":[{"role":"user","content":"Search for ReplaySSEStream"}]}` + recReq, _ := http.NewRequest("POST", recordSrv.URL()+"/v1/messages?beta=true", strings.NewReader(reqPayload)) + recReq.Header.Set("Content-Type", "application/json") + recReq.Header.Set("X-Api-Key", "sk-ant-api03-liveSecretKeyForRecording123456") + + recResp, err := http.DefaultClient.Do(recReq) + if err != nil { + t.Fatalf("record request failed: %v", err) + } + liveWireBytes, err := io.ReadAll(recResp.Body) + _ = recResp.Body.Close() + if err != nil { + t.Fatalf("failed reading live stream: %v", err) + } + + if err := cassette.Save(casPath, recordSrv.GetCassette()); err != nil { + t.Fatalf("failed saving cassette: %v", err) + } + _ = recordSrv.Close() + + // Permanently shut down upstream so any accidental network leak fails immediately + upstream.Close() + if upstreamHits.Load() != 1 { + t.Fatalf("expected exactly 1 upstream hit during recording, got %d", upstreamHits.Load()) + } + + // 3. Reload cassette and verify zero unredacted secrets + loadedCas, err := cassette.Load(casPath) + if err != nil { + t.Fatalf("failed loading cassette: %v", err) + } + if hasSec, reason := cassette.CheckForUnredactedSecrets(loadedCas); hasSec { + t.Fatalf("unredacted secret found in saved SSE cassette: %s", reason) + } + + // 4. Start deterministic offline replay server + replaySrv, err := proxy.NewServer(proxy.Config{ + Mode: proxy.ModeReplay, + Cassette: loadedCas, + Matcher: matcher.NewStrictMatcher(), + SSETiming: cassette.SSETimingConfig{ + Mode: cassette.SSETimingInstant, + }, + }) + if err != nil { + t.Fatalf("failed creating replay server: %v", err) + } + if err := replaySrv.Start(); err != nil { + t.Fatalf("failed starting replay server: %v", err) + } + defer replaySrv.Close() + + offlineClient := proxy.NewOfflineHTTPClient(replaySrv.URL()) + + // 5. Execute 10 consecutive replays and assert 100% byte-for-byte & semantic determinism + var baselineBytes []byte + var baselineSummary cassette.StreamSummary + + for run := 1; run <= 10; run++ { + // Reorder JSON keys and pass dummy CI API key to verify canonical secret-safe matching simultaneously + reorderedPayload := `{"stream":true,"messages":[{"content":"Search for ReplaySSEStream","role":"user"}],"max_tokens":1024,"model":"claude-sonnet-4-20250514"}` + req, _ := http.NewRequest("POST", replaySrv.URL()+"/v1/messages?beta=true", strings.NewReader(reorderedPayload)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-Api-Key", "af-ci-dummy-key") + + resp, err := offlineClient.Do(req) + if err != nil { + t.Fatalf("replay #%d failed: %v", run, err) + } + if resp.StatusCode != http.StatusOK { + t.Fatalf("replay #%d status=%d, expected 200", run, resp.StatusCode) + } + if resp.Header.Get("X-AgentFixture-Replay") != "true" { + t.Fatalf("replay #%d missing X-AgentFixture-Replay header", run) + } + + bodyBytes, err := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if err != nil { + t.Fatalf("replay #%d body read failed: %v", run, err) + } + + if !bytes.Equal(bodyBytes, liveWireBytes) { + t.Fatalf("replay #%d wire bytes diverged from live recorded stream", run) + } + + events, decErr := cassette.DecodeSSE(bytes.NewReader(bodyBytes)) + if decErr != nil { + t.Fatalf("replay #%d DecodeSSE failed: %v", run, decErr) + } + summary := cassette.ReconstructStreamSummary(events) + + if run == 1 { + baselineBytes = bodyBytes + baselineSummary = summary + } else { + if !bytes.Equal(bodyBytes, baselineBytes) { + t.Fatalf("replay #%d bytes differed from replay #1", run) + } + if !reflect.DeepEqual(summary, baselineSummary) { + t.Fatalf("replay #%d StreamSummary differed from replay #1", run) + } + } + } + + // Verify reconstructed fields + if baselineSummary.Text != "Executing search_code tool now." { + t.Errorf("unexpected reconstructed text: %q", baselineSummary.Text) + } + if len(baselineSummary.ThinkingBlocks) != 1 || baselineSummary.ThinkingBlocks[0] != "Checking repository files for deterministic replay." { + t.Errorf("unexpected thinking blocks: %+v", baselineSummary.ThinkingBlocks) + } + if len(baselineSummary.ToolCalls) != 1 || baselineSummary.ToolCalls[0].Name != "search_code" || + baselineSummary.ToolCalls[0].Input["pattern"] != "ReplaySSEStream" || + baselineSummary.ToolCalls[0].Input["max_results"] != int64(25) { + t.Errorf("unexpected reconstructed tool call: %+v", baselineSummary.ToolCalls) + } + if baselineSummary.Usage.InputTokens != 512 || baselineSummary.Usage.OutputTokens != 118 || + baselineSummary.Usage.CacheCreationInputTokens != 64 || baselineSummary.Usage.CacheReadInputTokens != 128 { + t.Errorf("unexpected reconstructed usage: %+v", baselineSummary.Usage) + } + + // 6. Verify Zero Outbound Network Dials during replay + directOffline := proxy.NewOfflineHTTPClient("") + if _, err := directOffline.Get("http://api.anthropic.com/v1/messages"); !errors.Is(err, proxy.ErrOutboundNetworkBlocked) { + t.Fatalf("expected direct external dial to be blocked with ErrOutboundNetworkBlocked, got: %v", err) + } + if _, err := offlineClient.Get("https://api.anthropic.com/v1/messages"); err == nil { + t.Fatalf("expected HTTPS CONNECT tunnel through replay proxy to be rejected") + } + if upstreamHits.Load() != 1 { + t.Fatalf("expected zero upstream hits during 10x replay, total=%d", upstreamHits.Load()) + } +}