feat(curl): 自研 curl bash 工具 —— 出站自然化 + 证据一等公民 - #110
Open
M09Ic wants to merge 6 commits into
Open
Conversation
Reimplement curl as an in-process aiscan bash command (shadowing the system binary) so HTTP probing is evidence-first and does not announce itself as automated tooling. - curl-shaped flag surface: -X/-H/-d(+--data-raw/binary,@file)/-G/-b/-c/ -L(--max-redirs)/-A/-e/-u/-o/-i/-s/-S/-w/-v/-k/--connect-timeout/--max-time; unsupported flags are rejected, never silently ignored. - routes through the runner MITM hub read from execution.Env (ALL_PROXY carries the tool-call id; CURL_CA_BUNDLE trusts intercepted HTTPS), so flows attribute by tool_id and are captured as http.exchange evidence. - browser-shaped User-Agent + header set (Chrome) applied only where the caller did not set them; -A/-H always win. - cookie round-trip (-b/-c) in Netscape format; stateless per-call execution to stay safe under concurrent invocations. Registered in the aiscan CLI and runner binaries; 22 unit tests cover the parser and the client against httptest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One model for one request/response pair: flat ordered header Pairs in memory, the Flow proto as its wire view, and the http.exchange.v1 flow element as its persisted JSON (headers as name→values map, byte compatible). Consumers that mirrored the flow shape on both sides of the wire can now alias this type instead. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The hub's stored Flow was a parallel HTTP exchange model with http.Header maps and *Snip body fields, converted to the wire Flow by hand. It now embeds aop/traffic.Exchange (headers as ordered Pairs, bodies as RequestBody/ResponseBody) and keeps only hub-only metadata (ToolID, Timestamp, Host, ContentType, Duration, TLS). Capture flattens http.Header to Pairs once; flowToProto is Exchange.Proto with attribution stamped on top, and headersToProto is gone. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
-F builds a real multipart/form-data body (name=value, name=@file with ;type= override or extension sniffing, name=<file for text), implies POST, and rejects mixing with -d/-G like curl does. --data-urlencode percent- encodes only the content half of name=content (unreserved set, space as %20), with @file/name@file forms. -x overrides the egress proxy for one invocation without bypassing the hub by default. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The flat Exchange could not say "request sent, no answer" — a refused or
timed-out probe was a status_code=0 smudge, and consumers (vuln pairing,
frontend rendering) had to special-case it. Exchange is now
{ID, Request, Response *Response, Error, Complete}: the response half is a
pointer that stays nil for request-only captures, and Complete requires a
response in the 1xx-9xx range with no error. The Flow proto mirrors this
(request=15, response=16; flat fields 3-11 reserved), and the
http.exchange.v1 JSON reshapes in place — headers stay a name→values map
projection, now inside request/response.
The hub captures natively in this shape: RequestError leaves Response nil,
the protocol version is recorded from the request line, and mitm query/
formatting handles response-less flows. Fixtures pin the nested v1 bytes
and the request-only round-trip.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
Issue: chainreactors/cairn-platform#84。
Runner 出站流量"自报家门"（系统 curl 的
curl/xUA、缺浏览器头），一眼是自动化工具。本 PR 用 Go 重写一个curl工具（commands.Command，名字就叫curl，shadow 系统 curl，用户/模型无感），把出站规整成贴近真实浏览器，并作为证据一等公民的受控 HTTP 主路径，替代 cairn 侧自研的http_request（其移除 + 提示词迁移在 cairn 侧后续 PR）。改动
新增
tools/curl/：parse.go— 手写 curl flag 解析：短 flag 捆绑（-sSL）、--k=v/--k v、-d @file、-H "Name:"（删默认头）/"Name;"（空值）。未识别 flag 明确报错，不静默忽略、不回退系统 curl。client.go— 从execution.Env读 hub 代理（ALL_PROXY带 tool-call id）+CURL_CA_BUNDLE（信任被拦截的 HTTPS）；自建 RootCAs；-L+--max-redirs重定向；-G原样拼 query；-i/-o/-w；浏览器 UA + 头集合仅补缺（-A/-H优先）。cookies.go—-b/-cNetscape cookie 往返。curl.go—Command{toolargs.Base}+Run；每调用无可变状态，并发安全。register.go—capability.Register+RegisterFactory。skills/aiscan/okf/easm/curl.md— 工具 playbook。接线：
cmd/aiscan/imports.go+cmd/runner/main.goblank-import。可观测性
curl 出站经 hub →
aop/traffic按tool_id捕获 → runner 侧规整为http.exchange.v1证据（脱敏复用）。curl 本身不建证据通道，复用现有机制。支持的 flag（v1）
-X -H -d --data-raw --data-binary -G -b -c -L --max-redirs -A -e -u -o -i -s -S -w -v -k --connect-timeout --max-time --url推迟 v1.1：
-F（multipart）、--data-urlencode、-x（未识别即报错；这些走真 shell 时仍经 env→hub 捕获，证据不丢）。测试
go build ./tools/curl、go vet ./tools/curl✅go test ./tools/curl— 22 单测（parser + httptest 验证方法/体/默认头/覆盖/重定向/cookie/basic-auth）✅go test ./tools（注册）✅go build ./cmd/aiscan ./cmd/runner✅🤖 Generated with Claude Code