xprobe

AI-native heterogeneous profiling tool for CPU and GPU.

xprobe is an AI harness for measuring latency between two observable events in a process, on the CPU, NVIDIA GPU, or across both. Its bounded native profiler combines sampled native/Python stacks, eBPF function, syscall, tracepoint and CPython runtime evidence with NVIDIA CUPTI, an agent-friendly CLI, strict JSON contracts, explicit correlation quality, and no daemon or server lifecycle.

Install

Install the version-matched Skill with the open Agent Skills CLI. This is the only installation action required from the user:

npx skills@1 add \
  https://github.com/itdevwu/xprobe/tree/v0.5.0/skills/xprobe-measure-latency \
  --global

The installer detects Codex, Claude Code, Cursor, and other compatible agents. When invoked, the Skill checks for the matching xprobe CLI and installs or repairs it under a writable prefix before profiling. It can then diagnose and adjust path, permission, NVIDIA, CUDA, or CUPTI problems from live evidence. Node.js is only needed for Skill installation, not for xprobe itself. See Installation for direct CLI use and archive verification.

Measure

For an unknown CPU workload, start with one representative sampled inventory:

xprobe validate --pid 4242 --cpu-sample \
  --json --non-interactive --no-color

xprobe measure --pid 4242 \
  --cpu-sample --duration-ms 1000 \
  --json --non-interactive --no-color

Use its hotspot and selector evidence, or a GPU aggregate inventory, to state a narrow hypothesis. Then validate and measure that boundary directly:

xprobe validate --pid 4242 \
  --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match exact --json --non-interactive --no-color

xprobe measure --pid 4242 \
  --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match exact --samples 100 --timeout-ms 30000 \
  --events-out /tmp/xprobe-events.jsonl \
  --json --non-interactive --no-color

Kernel launch latency is only one event pair. xprobe can first inventory sampled CPU/Python hotspots, syscall lifecycles, or GPU activity, then measure host function spans, CPython GC, syscall latency, named Linux events, CUDA API calls, GPU operation durations, transfers, NVTX ranges, and CPU/GPU paths after selecting the correct process.

measure also accepts completed --input captures and versioned live --spec files. Evidence can be exported as jsonl or chrome. JSON results contain every matched start/end event, latency statistics, unmatched and ambiguous counts, collection completeness, CUPTI buffer usage, clock quality, correlation confidence, and warnings. With --events-out, the bounded capture is also preserved when correlation or clock validation fails.

Public CLI

Command	Purpose
doctor	Report local eBPF, ptrace, NVIDIA, CUDA, and CUPTI capabilities
discover	List NVML-confirmed CUDA context holders under a process-tree root
validate	Check an event pair or inventory mode and report requirements without attaching
measure	Inventory bounded work or correlate bounded event evidence

measure --pid automatically loads the matching CUDA 12 or CUDA 13 CUPTI Agent when a selected endpoint requires it. It reports the target mutation on stderr and in JSON, disables collection afterward, and leaves the shared object mapped for safe reactivation. NVTX ranges are the exception: set NVTX_INJECTION64_PATH before the target's first NVTX call because online attach cannot retrofit an initialized NVTX dispatch.

Support

Surface	Current support
OS/architecture	Linux x86_64, glibc 2.34 or newer
CPU inventory	PID-scoped sampled user stacks with native ELF and CPython perf-map symbols
Host events	ELF functions, named syscall/tracepoint boundaries, and CPython GC USDT
Coarse aggregation	Bounded syscall lifecycle and CUDA kernel/memcpy/memset groups
CUDA callbacks	Runtime and Driver API entry/exit
GPU activity	Kernel, memcpy, and memset start/end
Application ranges	Bounded ASCII NVTX thread and process ranges
CUDA/CUPTI	12.x and 13.x with automatic major selection
Correlation	exact, first-after, nearest, stack-nested, stream-order
Online injection	same mount namespace; ptrace permission required

GPU-to-GPU durations remain available across the supported majors. Cross CPU/GPU subtraction requires the Agent's runtime alignment check to pass; otherwise measure returns CLOCK_ALIGNMENT_FAILED rather than treating an offset CUPTI clock as host monotonic.

Documentation

• Installation
• Architecture
• CLI contract
• CUPTI agent and injection
• Development and hardware tests
• Agent integration
• Public JSON schemas

Source builds and hardware tests are documented under Development. Implemented behavior is defined by code, tests, and schemas. Licensed under Apache-2.0.