Skip to content

cli-contract.md

Latest commit

 

History

History
332 lines (273 loc) · 14.5 KB

cli-contract.md

File metadata and controls

332 lines (273 loc) · 14.5 KB
Raw

CLI contract

xprobe exposes exactly four public commands: doctor, discover, validate, and measure.

Common behavior

All commands support --json --non-interactive --no-color. JSON mode writes one versioned document to stdout. Diagnostics, including the online-injection warning, go to stderr. Commands never prompt.

All success and error records carry schema_version: "2.0". Unknown JSON fields and unsupported schema versions are rejected. xprobe maintains only the current schema during the pre-1.0 protocol transition.

Exit Meaning
0 Command completed and emitted a result
1 Validation, collection, decode, export, cleanup, or internal failure
2 Invalid CLI syntax from Clap
3 Target missing, exited, or reused
4 Permission denied for inspection, eBPF, or ptrace

Runtime failures use schemas/error.schema.json and include a stable code, message, recoverability, details, and hints.

doctor

xprobe doctor --json --non-interactive --no-color

Reports Linux/kernel identity, BTF, BPF/perf permissions, lockdown, ptrace policy, NVIDIA driver, CUDA toolkit/driver, CUPTI, namespaces, and conservative capability booleans. ok: true means diagnosis completed, not that every capability is available. The CUPTI check lists each supported installed major and its resolved library path.

discover

xprobe discover --pid 4242 [--limit 200] \
  --json --non-interactive --no-color

Treats the PID as a process-tree root and queries NVML for active CUDA compute processes without attaching. It returns only root/descendant context holders, including PID plus procfs start time, parent PID, executable, command line, and GPU UUIDs. limit must be positive. total_candidates and truncated describe bounded output. The caller chooses one candidate and passes its PID to validate and measure; xprobe does not guess among workers.

This process-candidate response is discovery schema 2.0. xprobe does not emit or maintain the pre-0.3 event-discovery shape.

An unavailable or failed NVML query is an explicit command error, not an empty candidate list.

Selectors

Selectors are supplied by the caller and checked by validate before any attachment. Host selector forms are:

uprobe:<binary>:<symbol>:entry
uprobe:<binary>:<symbol>:return
uprobe:<binary>:symbol=<full-demangled-c++-signature>:entry
uprobe:<binary>:symbol=<full-demangled-c++-signature>:return
uprobe:<binary>:+0x<file-offset>:entry
uprobe:<binary>:+0x<file-offset>:return
syscall:<name>:entry
syscall:<name>:exit
tracepoint:<category>:<name>
python:gc_start
python:gc_end

The symbol= form allows :: and other punctuation in a full C++ signature. Resolution returns the attachable mangled ELF name in symbol and its readable signature in symbol_demangled; captured host events retain both. It works for mapped CPython, native extension, and framework libraries, but does not resolve Python frames or module.qualname names.

Named syscall selectors are Linux x86_64 endpoints resolved by validate. Their eBPF path filters PID and syscall number before reserving an event, records the six scalar syscall ABI registers on entry, and records the scalar return value on exit. It never dereferences pointer arguments. Named tracepoints record identity and timestamp only; they do not copy the tracepoint payload. Unsupported syscall names and unavailable tracepoints fail explicitly.

Python GC selectors are capability-based, not process-name based. Validation reads the target's mapped CPython executable and libpython ELF metadata and requires both python:gc__start and python:gc__done USDT markers. Captured events contain process/thread identity, timestamp, mapped binary, provider, and marker name without copying generation or object payloads.

CUDA forms include Runtime/Driver API entry/exit, kernel/memcpy/memset activity start/end, and NVTX range boundaries. Kernel selectors accept name~REGEX; memcpy selectors accept kind=<HtoD|DtoH|DtoD|HtoH|PtoP>. NVTX selectors are cuda:nvtx_range_start:name~REGEX and cuda:nvtx_range_end:name~REGEX; their name expression must reduce to one nonempty exact, prefix, suffix, or contains filter shorter than 128 bytes.

validate

xprobe validate --pid 4242 \
  --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match exact --json --non-interactive --no-color

Validation is read-only. It verifies target identity, resolves host and Python USDT selectors, parses CUDA filters, and checks collection and correlation requirements. validate --cpu-sample checks target threads, perf-event access, and Python symbolization status. validate --syscall-aggregate checks the bounded eBPF aggregate path. Inventory modes do not take event-pair selectors. Results conform to schemas/validate.schema.json.

Supported policies are exact, first-after, nearest, stack-nested, and stream-order. Exact uses deterministic CUPTI correlation IDs, NVTX range kind/ID, one named syscall's per-thread entry/exit lifecycle, or one CPython GC start/end lifecycle on the same thread. Nested requires entry/return of the same host function. Stream order requires GPU activity endpoints. Temporal policies always warn that they are heuristic. policy_recommendation reports the strongest compatible policy, a stable machine-readable reason, and all compatible alternatives. The caller still chooses the policy; xprobe does not silently replace or retry it.

requirements.agent_activation is not_required, already_loaded, injection_required, or startup_required. Online injection handles ordinary CUDA callbacks and activity; injection_required sets target_mutation: true and emits TARGET_PROCESS_WILL_BE_MODIFIED. NVTX dispatch is fixed by its first API initialization, so an unmapped Agent produces startup_required and an invalid result. Restart the target with NVTX_INJECTION64_PATH set to the matching xprobe Agent before its first NVTX call. A mapped Agent is verified again at ARM through the capture feature flag. Callers must stop on valid: false. Mapped CUDA/CUPTI majors that conflict or fall outside 12 and 13 produce UNSUPPORTED_CUDA_VERSION. CUDA major support does not imply host-clock alignment: measure accepts GPU-only pairs in the CUPTI clock and rejects CPU/GPU subtraction when the capture omits the host-monotonic feature flag.

measure

Live target:

xprobe measure --pid 4242 \
  --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match exact --samples 100 --timeout-ms 30000 \
  --events-out /tmp/xprobe-events.jsonl \
  --json --non-interactive --no-color

Linux syscall duration:

xprobe measure --pid 4242 \
  --from 'syscall:mmap:entry' --to 'syscall:mmap:exit' \
  --match exact --samples 100 --max-events 1000 \
  --json --non-interactive --no-color

CPU/Python hotspot inventory:

xprobe measure --pid 4242 --cpu-sample --duration-ms 1000 \
  --frequency-hz 99 --max-samples 10000 --max-groups 256 \
  --stack-depth 64 --max-threads 1024 \
  --json --non-interactive --no-color

Syscall lifecycle inventory:

xprobe measure --pid 4242 --syscall-aggregate --duration-ms 1000 \
  --max-groups 256 --max-inflight 1024 \
  --json --non-interactive --no-color

CPython garbage-collection duration:

xprobe measure --pid 4242 \
  --from python:gc_start --to python:gc_end \
  --match exact --samples 100 --max-events 1000 \
  --json --non-interactive --no-color

NVTX application range:

xprobe measure --pid 4242 \
  --from 'cuda:nvtx_range_start:name~^forward$' \
  --to 'cuda:nvtx_range_end:name~^forward$' \
  --match exact --samples 100 --max-events 1000 \
  --json --non-interactive --no-color

Completed captures:

xprobe measure --input host.json --input cupti.bin \
  --from 'uprobe:/srv/app/server:request:entry' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match first-after --samples 100 \
  --json --non-interactive --no-color

Versioned live spec:

xprobe measure --spec measurement.json \
  --json --non-interactive --no-color

Broad GPU activity inventory:

xprobe measure --pid 4242 \
  --from 'cuda:kernel_start' --to 'cuda:kernel_end' \
  --match exact --aggregate --duration-ms 1000 --max-groups 4096 \
  --json --non-interactive --no-color

Exactly one source mode is used: --pid, one or more --input, or --spec. At least one positive --samples or --duration-ms bound is required in direct mode. --timeout-ms defaults to 30 seconds and --max-events to 100,000.

--cpu-sample is live-only and requires --duration-ms. It opens bounded userspace callchain sampling events for the target's current threads. Defaults are 99 Hz, 10,000 observed samples, 256 retained stack groups, depth 64, and 1,024 threads. The result uses schemas/cpu-sample-inventory-result.schema.json: exact raw stack counts are grouped deterministically, hotspots report inclusive/exclusive counts and proportions, and attachable native frames include uprobe selector hints. Native ELF and /tmp/perf-<pid>.map Python symbols are resolved independently. python_status and resolved/unresolved frame counts prevent native fallback from being mislabeled as Python semantic coverage. Lost samples, truncated stacks, thread coverage, and capacity pressure remain visible.

--syscall-aggregate is live-only and requires --duration-ms. PID-filtered raw syscall entry/exit programs retain at most --max-inflight thread starts and --max-groups syscall groups in BPF maps, then emit one bounded summary. Known x86_64 numbers include names and exact entry/exit selector hints; unknown numbers remain numeric. schemas/syscall-aggregate-result.schema.json exposes entries, matched and unmatched exits, inflight-at-end, drops, occupancy, errors, and total/min/max/mean duration. It contains no pointer values or per-call event stream.

--aggregate is live-only, duration-bounded, and accepts one matching kernel, memcpy, or memset activity start/end pair. It uses --max-groups (default 4,096), does not accept --samples or --events-out, and emits schemas/aggregate-inventory-result.schema.json. The result is a coarse inventory, not exact event evidence: it reports count, total/min/max/mean duration, optional transferred bytes, table occupancy, and drop completeness. Aggregate kernel regex must be reducible to an exact, prefix, suffix, or contains filter because this mode intentionally has no Rust-side event pass. Each kernel group reports name_complete. CUPTI names that fill the fixed 127-byte observed prefix are marked false; their selector hints use that prefix instead of claiming an exact full name, so the next capture can still filter in the Agent hot path.

The three inventory modes have distinct result contracts and contain no exact event timeline. They cannot be passed to measure --input or exported with --events-out. Their groups and hotspots are evidence for choosing selectors; read-only validation still precedes the resulting exact measurement.

Live host endpoints attach PID-scoped eBPF probes. Linux syscall endpoints use raw tracepoints so they do not depend on tracingfs event IDs; ordinary named tracepoints use their kernel category and name. CPython GC endpoints attach to the exact mapped ELF selected by validation. Every endpoint link is attached before collection is armed. A samples-bound Linux capture allows bounded startup slack for a target already inside an event boundary. A duration capture that fills --max-events returns EVENT_RATE_TOO_HIGH rather than reporting partial success. CUDA endpoints automatically activate the CUPTI agent. If it is absent, --agent or XPROBE_CUPTI_AGENT_PATH selects the shared object; otherwise xprobe searches ../lib/xprobe/cuda12 or ../lib/xprobe/cuda13 according to the target and then the matching development build path. An unobservable target major is accepted only when exactly one supported CUPTI major is installed. Injection requires Linux x86_64, a shared mount namespace, and ptrace permission. It emits a stderr warning and CUPTI_AGENT_INJECTED. Each live call arms a fresh --max-events-bounded capture with endpoint filters. Final stop disables CUPTI; automatic injection also removes its private socket while leaving the .so mapped.

NVTX thread ranges preserve nesting through the thread ID and zero-based push level. Process ranges preserve the NVTX range ID and may end on a different thread; evidence reports both boundary threads plus start_tid. Names are copied into the fixed record bound with name_complete. ASCII nvtxRangePushA/nvtxRangeStartA and ASCII messages in their Ex forms are supported; wide and registered-string messages are not collected. This is timeline boundary collection only. It does not enable CUPTI Range Profiling, metric replay, or kernel performance-counter passes.

Completed inputs may be CUPTI ABI binary, bounded host-capture JSON, or Event JSONL. Repeated inputs are merged after target-PID checks. Unknown records, mixed targets, malformed captures, and excessive event counts fail explicitly.

Results conform to schemas/measurement-result.schema.json and contain:

  • status and latency statistics;
  • matched, unmatched, ambiguous, and dropped counts;
  • method, confidence, and score;
  • clock alignment and optional estimated error;
  • collection completeness, host/CUDA totals, and CUPTI capacity, observed, retained, dropped, and utilization values when an ABI envelope provides them;
  • full start/end events plus latency for each evidence pair;
  • structured warnings.

--events-out PATH atomically writes the complete bounded capture with mode 0600. The default format is JSONL; --format chrome writes Chrome Trace Event Format. Once collection has produced a capture, correlation, clock, drop, or capacity failures still write it and identify its path, format, and event count in the JSON error. Artifact failure returns TRACE_EXPORT_FAILED and preserves the original measurement error code in details.

No matched pairs return NO_MATCHED_SAMPLES. Cross-clock subtraction without declared alignment returns CLOCK_ALIGNMENT_FAILED. Drops and unknown clock error are never silently converted to high-quality results. Error details and hints carry selectors, policies, counters, clock domains, and explicit next actions where applicable; callers must not infer these fields from messages.

Compatibility

Earlier low-level command spellings remain hidden during the pre-1.0 transition so existing integration fixtures can decode captures. They are not public API and must not be used by Skills or new automation.