Skip to content

feat(uprobe): implement uprobe breakpoint support (#2150 phase 1) - #2163

Open
sparkzky wants to merge 3 commits into
DragonOS-Community:masterfrom
sparkzky:feat/uprobe-uretprobe
Open

feat(uprobe): implement uprobe breakpoint support (#2150 phase 1)#2163
sparkzky wants to merge 3 commits into
DragonOS-Community:masterfrom
sparkzky:feat/uprobe-uretprobe

Conversation

@sparkzky

Copy link
Copy Markdown
Member

关联 Issue

Refs #2150(阶段一:uprobe 断点探针)

uretprobe(阶段二)不在本 PR 范围,后续单独提交。

概述

实现用户态断点探针(uprobe),使 agentsight 能在用户态函数(如 SSL_read/SSL_write)入口挂探针捕获参数。本 PR 完成阶段一(断点探针),命中路径 #BP → XOL 单步 → #DB → 恢复 端到端打通。

设计决策

经盲区扫描 + 对抗评审纠正后的架构(关键点:不复用 kprobe 的内核缓冲区单步——CPL=3 时内核页不可执行):

  1. XOL 执行原指令:每个 mm 在用户态 slot 页执行原指令副本,RIP-relative 用 yaxpeax-x86 重定位
  2. 独立 per-mm 分发:irqsave SpinLock(非全局 KPROBE_MANAGER 锁 / 非 RwSem,命中路径关中断不可睡眠)
  3. 断点页复刻 do_wp_page 私有 COWcopy_page_as_normal + 单次 set_entry 原子帧替换 + rmap 账簿,每 mm 私有副本(writeback 不回写 0xcc 损坏 .so)
  4. 注册时预填 XOL slot:RIP-relative 位移溢出在注册时 fail-fast 返回 EINVAL(不在命中时 panic)
  5. 异常分发do_int3/do_debugis_from_user() 二分;未消费用户态 #BP 投递 SIGTRAP(TRAP_BRKPT)
  6. perf 接入:复用 PERF_TYPE_MAX(6),按 config1 name 含 / 区分 uprobe/kprobe;复用 BPF_PROG_TYPE_KPROBE

改动文件

模块 文件 说明
新建 crate kernel/crates/uprobe/ 架构无关核心 + x86 指令分析(复用 yaxpeax-x86)
mm 集成 kernel/src/mm/ucontext/uprobe.rs per-mm 表 + XOL 区 + 断点页安装(复刻 do_wp_page COW)
异常分发 kernel/src/exception/uprobe.rs #BP/#DB 用户态分发 + XOL 单步 + SIGTRAP + NEED_UPROBE
perf 接入 kernel/src/perf/uprobe.rs UprobePerfEvent + perf_event_open uprobe 分发臂
改动 interrupt/{trap,mod}.rsexception/mod.rsmm/ucontext/{address_space,inner,mod}.rsperf/mod.rsprocess/state.rs is_from_user 二分、ProbeArgs impl、NEED_UPROBE 位等
测试 user/apps/tests/dunitest/suites/normal/uprobe.cc gtest 套件(3 用例)

验证

  • make kernel0 error / 0 warning
  • cargo test -p uprobe7/7 通过(指令长度、RIP-relative 检测/重定位、位移溢出)
  • 独立 reviewer 核对评审 findings(F1-F10)全部满足,无 kprobe/fork regression;修复了验证发现的 2 个 bug:
    • 重复注册同一 probe_vaddr 读到 0xcc 当原指令 → 改为复用已有指令信息
    • RIP-relative 位移溢出在命中时 panic → 移到注册时 fail-fast
  • dunitest 编译通过;核心触发用例的运行时验证待 QEMU(作者将在其他机器测试)

命中流程

flowchart TD
    A["用户态执行到 0xcc"] -->|"#BP"| B["do_int3: is_from_user?"]
    B -->|是| C["uprobe_breakpoint_handler\nlock uprobe_list(irqsave)"]
    C --> D["跑 pre_handler + BPF\nrip=原探针址"]
    D --> E["取 entries[0] XOL slot\n(slot 注册时已预填)"]
    E --> F["rip→XOL slot + TF + NEED_UPROBE"]
    F --> G["iretq: 用户态执行原指令"]
    G -->|"TF 触发 #DB"| H["uprobe_debug_handler\nNEED_UPROBE?"]
    H -->|是| I["rip 回原址+insn_len\n清 TF → post_handler"]
    I --> J["正常继续"]
Loading

后续

  • 阶段二:uretprobe(栈返回地址改写 + trampoline 页)
  • 运行时集成验证(QEMU + agentsight sslsniff.bpf.c

@github-actions github-actions Bot added the enhancement New feature or request label Jul 31, 2026
@fslongjin

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c8c8a99c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kernel/src/perf/uprobe.rs
Comment thread kernel/crates/uprobe/src/arch/x86/mod.rs
Comment thread kernel/src/mm/ucontext/uprobe.rs
Comment thread kernel/src/mm/ucontext/uprobe.rs Outdated
Comment thread kernel/src/exception/uprobe.rs Outdated
@sparkzky
sparkzky force-pushed the feat/uprobe-uretprobe branch 3 times, most recently from 3a99e3d to 86f1f09 Compare August 1, 2026 07:31

@fslongjin fslongjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the substantial work on this. Using a user-space XOL area and private breakpoint pages is the right general direction, and keeping uretprobe support out of this first phase is a reasonable scope choice.

However, I do not think this version is safe to merge yet. The following issues can change program behavior, break debugging, or cross process security boundaries.

  1. pid == -1 bypasses process access checks (kernel/src/perf/uprobe.rs, around line 234).

    For a non-negative pid, the code checks whether the caller may inspect the target process. For pid == -1 (and any value below -1), it skips that check and installs breakpoints in every process currently mapping the file. An unprivileged process could therefore modify code pages in processes owned by another user, attach BPF code, or crash those processes. Please reject invalid negative pids and apply an explicit privileged or per-process access check for any system-wide mode. Linux does not treat pid == -1 as permission to modify every matching process.

  2. The active XOL operation is not kept alive until single-step completion (kernel/src/mm/ucontext/uprobe.rs, around line 629; kernel/src/exception/uprobe.rs, around line 109).

    A thread can hit the breakpoint while another thread closes the perf fd. The close path can remove the probe and reuse its XOL slot before the first thread receives the debug exception. The first thread may then resume at the wrong address or remain in the XOL page. A per-thread active-probe object must hold the probe, original address, return address, XOL slot, and original trap flag until completion or abort. Probe removal must wait for active users before reusing the slot.

  3. Signals and exceptions during XOL execution are not handled safely (kernel/src/exception/uprobe.rs, around line 109).

    NEED_UPROBE is only a boolean. If a signal is delivered before the copied instruction runs, or if that instruction faults, a later unrelated debug exception can be mistaken for XOL completion and change the instruction pointer incorrectly. Linux keeps a per-thread uprobe state and has explicit complete and abort paths. DragonOS needs the same essential state, even if the implementation is simpler.

  4. Unrelated user debug exceptions are swallowed (kernel/src/exception/uprobe.rs, around line 124).

    Every user-mode debug exception is sent to the uprobe handler. When NEED_UPROBE is not set, the handler returns success without passing the exception to ptrace, hardware-breakpoint handling, or normal SIGTRAP delivery. This breaks single-stepping and hardware breakpoints for all programs, including programs that do not use uprobes. The uprobe handler should report whether it consumed the exception; all other cases must continue through the normal debug path.

  5. The original trap flag is lost (kernel/src/exception/uprobe.rs, around line 188).

    XOL completion always clears TF. If the program or debugger had already enabled single-step mode, hitting an uprobe silently disables it. Save the original TF value per thread, clear only a TF bit added by uprobe, and preserve the expected SIGTRAP behavior.

  6. Instructions crossing a page boundary are copied incorrectly (kernel/src/mm/ucontext/uprobe.rs, around line 303).

    The read helper fetches bytes from the next page, but the copy into old_instruction is limited to the bytes remaining in the first page. The rest of the XOL instruction becomes zero bytes and may execute as a different instruction. Copy the full decoded instruction, and fail registration if all bytes cannot be read safely.

  7. Removing one of several probes at the same address disables the others (kernel/src/mm/ucontext/uprobe.rs, around line 692).

    Each consumer has a separate instance, but removing any one of them immediately restores the original byte. Other consumers remain in the list but can no longer receive hits. Consumers at one address should share one breakpoint, and the byte should be restored only when the last consumer is removed.

  8. Removing a probe can discard valid changes to the rest of the page (kernel/src/mm/ucontext/uprobe.rs, around line 735).

    The final removal maps the page saved at registration time back into the process. If the current private page was writable and the program changed other bytes while the probe was active, those changes are lost. Removal should preserve the current page contents and restore only the breakpoint byte.

  9. Probe registration is only a snapshot of mappings that exist at open time (kernel/src/perf/uprobe.rs, around line 252).

    A library mapped after perf_event_open, a new image after exec, or a child created by fork will not receive the probe. There is also a race where a mapping can be removed and replaced after the file-mapping list is collected, causing the breakpoint to be installed in an unrelated executable mapping at the same address. The durable probe identity should be the file inode plus offset, with mapping, unmapping, fork, and exec paths applying or removing per-address-space instances.

  10. The x86 instruction safety check is incomplete (kernel/crates/uprobe/src/arch/x86/mod.rs, around line 81).

    Rejecting control-flow instructions is not sufficient. Instructions such as MOV SS and POP SS can suppress the debug exception that the XOL path relies on. Linux uses an allow-list and special handling for instructions that change exception or trap behavior. Unsupported instructions should fail at registration rather than enter a broken XOL path.

  11. The perf interface is not compatible with the Linux PMU interface (kernel/src/perf/mod.rs, around line 386).

    PERF_TYPE_MAX is used for both kprobe and uprobe, and the code guesses the kind from whether the name contains /. Standard tools obtain separate dynamic kprobe and uprobe types from sysfs. The current test hard-codes type 6, so it does not prove that agentsight or standard libbpf tooling can attach. Please expose a real uprobe event source/type and dispatch by that type. Also parse and honor perf_event_attr.disabled; probes currently start enabled even when the caller requested a disabled event.

  12. BPF callbacks and XOL lookup add too much work while interrupts are disabled (kernel/src/exception/uprobe.rs, around lines 73 and 152).

    The handler holds the per-address-space spin lock while running all BPF callbacks, and the debug handler scans every probe to find the XOL slot. This can create long interrupt-off delays under frequent probes. Keep only a short protected lookup, hold a safe reference to the active probe, run callbacks after releasing the lock, and use the per-thread active state for constant-time XOL completion.

The current unit tests and successful kernel build are useful, but they cover the simple path only. Before merging, please add DragonOS runtime tests for:

  • access from another user and invalid negative pids;
  • two consumers on the same address;
  • close/unregister racing with a hit;
  • an instruction crossing a page boundary;
  • signals or faults during XOL execution;
  • ptrace, an existing TF bit, and hardware breakpoints;
  • fork, exec, and mappings created after registration;
  • perf_event_attr.disabled and a real BPF callback.

These changes require a small architectural correction rather than isolated patches: keep the registered probe by file and offset, derive installed breakpoints for each address space, and keep the active XOL state on the executing thread. That structure will also remove several of the current races and performance costs.

…#2150 phase 1)

Implement userspace breakpoint probes (uprobe), phase 1 of issue DragonOS-Community#2150,
enabling agentsight to instrument SSL_read/SSL_write entry points.

The design is XOL-based rather than reusing kprobe's kernel-buffer
single-step (impossible at CPL=3). Key pieces:

- per-mm uprobe table guarded by an irqsave SpinLock (not the global
  KPROBE_MANAGER lock nor the mm RwSem; the #BP/#DB hit path is IRQ-off)
- breakpoint page install replicates do_wp_page private COW:
  copy_page_as_normal + single atomic set_entry + rmap attach/detach
  + flush_tlb_range. No transient empty PTE; each mm gets a private copy
  so writeback never persists 0xcc into the shared page-cache (.so)
- XOL: a per-mm user slot page executes the saved instruction copy with
  RIP-relative relocation (yaxpeax-x86), validated at registration time
- do_int3/do_debug gain is_from_user() dispatch. The #BP handler runs
  pre_handler + BPF (rip kept as the original probe address), then jumps
  rip to the pre-filled XOL slot, sets TF and NEED_UPROBE. The #DB handler
  recognizes XOL completion via NEED_UPROBE and restores rip; unconsumed
  user #BP is delivered as SIGTRAP(TRAP_BRKPT)
- perf: PERF_TYPE_MAX dispatches to uprobe when the name contains '/';
  UprobePerfEvent mirrors KprobePerfEvent and reuses BPF_PROG_TYPE_KPROBE

Delivered in four batches: uprobe crate (x86 instruction analysis), mm
integration (per-mm table / XOL / breakpoint page), exception dispatch,
and perf attach.

Verified: `make kernel` builds with 0 error / 0 warning; `cargo test -p
uprobe` passes 7/7. An independent reviewer confirmed the F1-F10 review
findings are satisfied with no kprobe/fork regression, and flagged two
bugs that are fixed: re-registering the same probe_vaddr no longer reads
0xcc as the original instruction, and a RIP-relative displacement overflow
now fails fast at registration instead of panicking at hit time.

Out of scope: uretprobe (phase 2) and the QEMU runtime integration test.

Refs: DragonOS-Community#2150

test(uprobe): add dunitest suite for uprobe breakpoint probes

Add suites/normal/uprobe.cc covering the userspace perf_event_open
uprobe path (issue DragonOS-Community#2150 phase 1):
- RegisterAndTriggerSurvivesHit: perf_event_open(type=PERF_TYPE_MAX,
  config1=path, config2=offset) on the current process, then call the
  probed function and assert it survives the #BP -> XOL -> #DB -> resume
  hit path and returns the correct value
- InvalidPathIsRejected / InvalidOffsetIsRejected: error inputs return
  negative errno

Target offset is resolved from /proc/self/maps (executable segment +
file pgoff), so the suite works regardless of PIE layout.

Compiles cleanly via `make build-suites`; the gtest framework runs (the
two negative cases pass on host Linux; the core trigger case is
DragonOS-specific and is validated at runtime under QEMU).

Refs: DragonOS-Community#2150

fix(uprobe): resolve CI failures - format check and cross-arch build

- Apply rustfmt to uprobe integration code (reorder modules, imports,
  line width) to pass format-check on all arches
- Add #[cfg(target_arch = "x86_64")] gates to uprobe integration points
  (exception/perf/mm-ucontext modules, AddressSpace fields, fork path,
  perf dispatch arm) so riscv64/loongarch64 build succeeds
- Non-x86_64 perf dispatch returns ENOSYS for uprobe paths
- Fix unused_mut on phys_addr in fork path for non-x86_64
- Thread 1: add ptrace access check (check_process_vm_access) before
  taking a target mm for cross-process uprobe, preventing unprivileged
  users from instrumenting arbitrary processes
- Thread 2: reject control-flow instructions (call/jmp/ret/jcc/loop/int/
  syscall) at registration time — XOL cannot safely single-step them
- Thread 3: read_user_insn_bytes now continues into the next page when
  the probe is near a page boundary, returning real bytes instead of
  zero-padding that could decode to a different instruction
- Thread 5: build_xol_slot fills trailing slot bytes with int3 (0xcc) so
  that a racy unregister during the XOL single-step window re-triggers
  #BP instead of executing zero-filled garbage
Security & correctness:
- R1: pid==-1 (system-wide) now requires CAP_SYS_PTRACE; pid<-1
  returns EINVAL. Per-pid path keeps check_process_vm_access.
- R4: user #DB not consumed by uprobe now falls through to the normal
  DebugException path (restores pre-PR master behavior for
  ptrace/hardware breakpoints/single-step).
- R5: original RFLAGS.TF is saved per-thread before XOL redirect and
  restored on completion (previously cleared unconditionally, silently
  disabling a program's own single-step mode).
- R6: old_instruction copy now covers the full decoded instruction
  across page boundaries (was limited to first-page remainder).
- R7: the breakpoint byte is restored only when the LAST consumer at
  that address unregisters (previously restored on every unregister,
  silently disabling remaining same-address consumers).
- R8: unregister writes the original byte on the CURRENT mapped page
  instead of remapping the registration-time page, preserving the
  program's own writes to other bytes of that page.
- R10: reject MOV SS (suppresses #DB) and POPF (overwrites RFLAGS/TF)
  at registration, in addition to control-flow instructions.
- R11a: honor perf_event_attr.disabled (event starts disabled).

Architecture (per-thread state, R2/R3/R12):
- ActiveXol per-thread state on the PCB: probe_vaddr, return_addr,
  orig_tf, xol_page_base. Saved at #BP before rip redirect, consumed
  at #DB: O(1) completion independent of uprobe_list (racy unregister
  between #BP and #DB no longer corrupts resume), abort path when rip
  is outside the XOL page (signal/fault diversion), callbacks run
  outside the per-mm spinlock.

Durable probe identity (R9):
- Global registry keyed by inode+offset with consumer ids. New file
  mappings (dlopen/mmap via file_mapping_with_file_ext) and fork get
  late-applied probes; exec starts with an empty table. Consumer close
  drops registry entries plus late handles (per-mm unregister via the
  existing UprobeHandle::Drop with R7/R8 semantics). fork inherits
  instances with privatized child breakpoint pages.

Verified: make kernel 0 error/0 warning; make fmt (clippy) clean;
cargo test -p uprobe 9/9; QEMU dunitest uprobe 6/6, exec_abi 6/6,
process_signal_fork 10/10.
@sparkzky
sparkzky force-pushed the feat/uprobe-uretprobe branch from 86f1f09 to 0528893 Compare August 16, 2026 10:28
@sparkzky

Copy link
Copy Markdown
Member Author

感谢详尽的评审。12 项已全部处理完毕(052889364,rebase 到最新 master):

安全与正确性(R1/R4/R5/R6/R7/R8/R10/R11a)

  • R1:pid == -1 系统级模式现在要求 CAP_SYS_PTRACE(EPERM),pid < -1 返回 EINVAL;单 pid 路径保留 check_process_vm_access
  • R4:未被 uprobe 消费的用户态 #DB 现在回落到 DebugException::handle(恢复本 PR 之前 master 的行为),ptrace 单步/硬件断点不再被吞。
  • R5:原始 RFLAGS.TF 按线程保存,XOL 完成后恢复(程序自身的单步模式不再被静默禁用)。
  • R6:跨页指令完整拷贝 insn_len 字节(不再截断为本页剩余)。
  • R7:断点字节仅在该地址最后一个 consumer 注销时恢复;同址其余 consumer 继续命中。
  • R8:注销在当前映射页上恢复字节,不回退到注册时的旧页——程序对页内其他字节的写全部保留。
  • R10:注册时额外拒绝 MOV SS(抑制 #DB)与 POPF(覆写 RFLAGS);新增单测覆盖。
  • R11a:perf_event_attr.disabled 生效(事件初始禁用)。

架构修正(R2/R3/R12)

  • PCB 新增 per-thread ActiveXol 状态(probe_vaddr / return_addr / orig_tf / xol_page_base):#BP 重定向前保存,#DB O(1) 取回完成——注销与 #DB 的竞态不再影响恢复;rip 不在 XOL 页内时走 abort 路径(信号/缺页改道);pre/event/post 回调全部移出 per-mm 自旋锁。

持久探针身份(R9)

  • 全局注册表以 inode+offset 为键、带 consumer id:新文件映射(file_mapping_with_file_ext 提交后)与 fork 迟到安装探针;exec 换新 AddressSpace 自然清空。consumer close 时移除注册表项并 drop 迟到句柄(复用 UprobeHandle::Drop 的 R7/R8 注销语义)。fork 继承时对子进程断点页做私有化(COW),避免恢复字节写入共享页。

R11(perf PMU 接口)PERF_TYPE_MAX 双用 + sysfs 动态类型属独立接口改造,计划在后续 PR 单独实现(涉及 sysfs 事件源注册与 libbpf 工具链验证),避免与本轮安全修复混在同一变更。

验证make kernel 0 error/0 warning;make fmt(含 clippy deny)通过;cargo test -p uprobe 9/9;QEMU 实测:uprobe 套件 6/6、exec_abi 6/6、process_signal_fork 10/10。您列出的运行时测例(跨用户访问、同址双 consumer、注销与命中竞态、跨页指令、XOL 期间信号、ptrace/TF/硬件断点、fork/exec/注册后映射、disabled、BPF 回调)将作为下一批 dunitest 用例补充。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants