Skip to content

Commit 3b5dd95

Browse files
smuppandsmuppand
committed
test(module-reload): harden fastrpc profile daemon quiesce flow
Update the fastrpc profile to stop and mask rpc daemons, verify remaining daemon processes, and provide clearer stdout context when quiesce or unload paths fail. Signed-off-by: Srikanth Muppandam <smuppand@qti.qualcomm.com>
1 parent da4ee85 commit 3b5dd95

5 files changed

Lines changed: 552 additions & 0 deletions

File tree

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/Module_Reload_Validation.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
metadata:
2+
name: Module_Reload_Validation
3+
format: "Lava-Test Test Definition 1.0"
4+
description: "Generic profiled kernel module unload/reload regression validation"
5+
maintainer:
6+
- Srikanth kumar
7+
os:
8+
- linux
9+
scope:
10+
- functional
11+
12+
params:
13+
PROFILE: ""
14+
ITERATIONS: "3"
15+
MODE: ""
16+
TIMEOUT_UNLOAD: "30"
17+
TIMEOUT_LOAD: "30"
18+
TIMEOUT_SETTLE: "20"
19+
ENABLE_SYSRQ_HANG_DUMP: "1"
20+
21+
run:
22+
steps:
23+
- REPO_PATH=$PWD
24+
- cd Runner/suites/Kernel/Baseport/Module_Reload_Validation
25+
- SYSRQ_ARG=""
26+
- if [ "${ENABLE_SYSRQ_HANG_DUMP}" = "0" ]; then SYSRQ_ARG="--disable-sysrq-hang-dump"; fi
27+
- ./run.sh --module "${PROFILE}" --iterations "${ITERATIONS}" --mode "${MODE}" --timeout-unload "${TIMEOUT_UNLOAD}" --timeout-load "${TIMEOUT_LOAD}" --timeout-settle "${TIMEOUT_SETTLE}" ${SYSRQ_ARG} || true
28+
- $REPO_PATH/Runner/utils/send-to-lava.sh Module_Reload_Validation.res

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/Module_Reload_Validation_README.md‎

Lines changed: 227 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,227 @@
1+
# Module_Reload_Validation
2+
3+
## Overview
4+
`Module_Reload_Validation` is a generic, profile-driven kernel module unload/reload regression suite.
5+
6+
It is intended to catch issues such as:
7+
- module unload hangs,
8+
- failed reloads,
9+
- service/device rebind regressions after reload,
10+
- issues that reproduce on the 1st, 2nd, or later reload iteration.
11+
12+
The suite uses:
13+
- a **generic engine** in `run.sh`,
14+
- shared helper logic in `Runner/utils/lib_module_reload.sh`,
15+
- **module-specific profiles** under `profiles/`.
16+
17+
## Folder layout
18+
19+
```text
20+
Runner/suites/Kernel/Baseport/Module_Reload_Validation/
21+
├── run.sh
22+
├── Module_Reload_Validation.yaml
23+
├── profiles/
24+
│ ├── enabled.list
25+
│ └── fastrpc.profile
26+
27+
Runner/utils/
28+
└── lib_module_reload.sh
29+
```
30+
31+
## Main components
32+
33+
### `run.sh`
34+
Thin orchestration layer that:
35+
- parses CLI arguments,
36+
- resolves the selected profile(s),
37+
- invokes the generic library engine,
38+
- writes `Module_Reload_Validation.res`.
39+
40+
### `lib_module_reload.sh`
41+
Shared module reload engine that handles:
42+
- module state checks,
43+
- timeout-controlled unload/load execution,
44+
- per-iteration evidence collection,
45+
- timeout-path hang evidence,
46+
- profile hook dispatch,
47+
- result handling.
48+
49+
### `profiles/*.profile`
50+
Each profile provides module-specific metadata and optional hook logic.
51+
52+
Example profile fields:
53+
- `PROFILE_NAME`
54+
- `PROFILE_DESCRIPTION`
55+
- `MODULE_NAME`
56+
- `PROFILE_MODE_DEFAULT`
57+
- `PROFILE_REQUIRED_CMDS`
58+
- `PROFILE_SERVICES`
59+
- `PROFILE_DEVICE_PATTERNS`
60+
- `PROFILE_SYSFS_PATTERNS`
61+
62+
Optional hooks:
63+
- `profile_prepare`
64+
- `profile_warmup`
65+
- `profile_quiesce`
66+
- `profile_post_unload`
67+
- `profile_post_load`
68+
- `profile_smoke`
69+
- `profile_finalize`
70+
71+
## Current starter profile
72+
73+
### `fastrpc.profile`
74+
Current profile covers FastRPC unload/reload validation and supports service lifecycle-based testing.
75+
76+
Default mode:
77+
- `daemon_lifecycle`
78+
79+
Relevant services:
80+
- `adsprpcd.service`
81+
- `cdsprpcd.service`
82+
83+
## Execution flow
84+
For each selected profile:
85+
86+
1. Validate the profile.
87+
2. Ensure the module is loaded before starting iteration work.
88+
3. Run warmup hook.
89+
4. Capture pre-state logs.
90+
5. Run quiesce hook.
91+
6. Attempt module unload with timeout.
92+
7. Validate module absence.
93+
8. Run post-unload hook.
94+
9. Attempt module reload with timeout.
95+
10. Validate module presence.
96+
11. Run post-load hook.
97+
12. Run smoke hook.
98+
13. Capture post-load state.
99+
14. Repeat for all iterations.
100+
15. Run finalize hook.
101+
102+
## Hang handling policy
103+
Sysrq dump is **not** triggered on normal passing iterations.
104+
105+
It is triggered only when an unload/load action actually times out.
106+
107+
Current behavior:
108+
- normal pass -> no sysrq dump
109+
- quick non-timeout failure -> normal failure evidence only
110+
- timeout / hang -> hang evidence bundle + optional sysrq dump
111+
112+
Default behavior in current suite:
113+
- sysrq hang dump enabled
114+
- but only used on timeout paths
115+
116+
## Evidence collected
117+
Per profile / iteration, the suite can capture:
118+
- command logs,
119+
- `lsmod`,
120+
- `modinfo`,
121+
- `ps`,
122+
- `dmesg`,
123+
- service status and recent journal,
124+
- profiled device path presence,
125+
- profiled sysfs path presence,
126+
- `/sys/module/<module>/holders`,
127+
- timeout PID `/proc` snapshots,
128+
- optional sysrq task/block dumps.
129+
130+
Results are stored under:
131+
132+
```text
133+
results/Module_Reload_Validation/<profile>/iter_XX/
134+
```
135+
136+
## CLI usage
137+
138+
### Run one profile
139+
```sh
140+
./run.sh --module fastrpc
141+
```
142+
143+
### Run one profile with more iterations
144+
```sh
145+
./run.sh --module fastrpc --iterations 5
146+
```
147+
148+
### Override mode
149+
```sh
150+
./run.sh --module fastrpc --mode daemon_lifecycle
151+
./run.sh --module fastrpc --mode basic
152+
```
153+
154+
### Override timeouts
155+
```sh
156+
./run.sh --module fastrpc --timeout-unload 60 --timeout-load 60 --timeout-settle 30
157+
```
158+
159+
### Disable sysrq timeout-path dumps
160+
```sh
161+
./run.sh --module fastrpc --disable-sysrq-hang-dump
162+
```
163+
164+
### Run all enabled profiles
165+
```sh
166+
./run.sh
167+
```
168+
169+
If `--module` is empty or not given, the suite runs all profiles listed in `profiles/enabled.list`.
170+
171+
If `--mode` is empty or not given, the profile default mode is used.
172+
173+
## YAML usage in LAVA
174+
Current YAML is generic and takes profile input from the test plan.
175+
176+
Important params:
177+
- `PROFILE`
178+
- `ITERATIONS`
179+
- `MODE`
180+
- `TIMEOUT_UNLOAD`
181+
- `TIMEOUT_LOAD`
182+
- `TIMEOUT_SETTLE`
183+
- `ENABLE_SYSRQ_HANG_DUMP`
184+
185+
Behavior:
186+
- `PROFILE="fastrpc"` -> runs only `fastrpc.profile`
187+
- `PROFILE=""` -> runs all enabled profiles
188+
- `MODE=""` -> uses the profile default mode
189+
190+
## How to add a new profile
191+
1. Create a new file under `profiles/`, for example:
192+
- `profiles/ath11k_pci.profile`
193+
2. Define the required metadata:
194+
- `PROFILE_NAME`
195+
- `MODULE_NAME`
196+
3. Add hooks only if module-specific lifecycle handling is needed.
197+
4. Add the profile basename to `profiles/enabled.list`.
198+
5. Run locally with:
199+
200+
```sh
201+
./run.sh --module ath11k_pci
202+
```
203+
204+
No YAML duplication is needed for new profiles.
205+
206+
## Result policy
207+
208+
### PASS
209+
- all requested iterations for a profile pass successfully.
210+
211+
### FAIL
212+
- unload/load timeout,
213+
- unload/load command failure,
214+
- module state validation failure,
215+
- profile hook failure,
216+
- smoke validation failure.
217+
218+
### SKIP
219+
- module not present,
220+
- module built into the kernel,
221+
- required commands not available,
222+
- profile explicitly not reloadable.
223+
224+
## Notes
225+
- This suite is intended for **profiled, supported modules**, not blind reload of every loaded kernel module.
226+
- The current structure avoids hidden run.sh-to-library globals as much as possible by passing explicit arguments and hook context.
227+
- Profile hooks receive context arguments from the engine, so module-specific logic can store logs in the correct iteration directory without depending on hidden globals.

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/profiles/enabled.list‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
fastrpc

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/profiles/fastrpc.profile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/sh
2+
# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
3+
# SPDX-License-Identifier: BSD-3-Clause-Clear
4+
5+
PROFILE_NAME="fastrpc"
6+
PROFILE_DESCRIPTION="FastRPC module reload validation"
7+
MODULE_NAME="fastrpc"
8+
9+
MODULE_RELOAD_SUPPORTED="yes"
10+
PROFILE_MODE_DEFAULT="daemon_lifecycle"
11+
12+
PROFILE_REQUIRED_CMDS="modprobe rmmod systemctl ps sed"
13+
PROFILE_SERVICES="adsprpcd.service cdsprpcd.service"
14+
PROFILE_PROC_PATTERNS="/usr/bin/adsprpcd /usr/bin/cdsprpcd"
15+
PROFILE_DEVICE_PATTERNS="/dev/fastrpc-* /dev/*dsp*"
16+
PROFILE_SYSFS_PATTERNS="/sys/module/fastrpc /sys/class/remoteproc/*"

0 commit comments

Comments
 (0)