DAOS-17519 test: Automate dlck testing (basic/fault_injection) by rpadma2 · Pull Request #17307 · daos-stack/daos

rpadma2 · 2025-12-23T18:53:19Z

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

github-actions · 2025-12-23T18:53:37Z

Ticket title is 'Automation - Basic dlck test: scan the DAOS system by running the dlck tool.'
Status is 'In Progress'
Labels: 'test_2.8,testp2'
https://daosio.atlassian.net/browse/DAOS-17519

Test-tag: DlckBasicTest DlckBasicFaultTest Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

janekmi · 2025-12-29T15:52:39Z

+        'id': '131584',
+        'probability_x': '100',
+        'probability_y': '100',
+        'interval': '2'},


IMHO without a comment it is hard to understand why the interval value is as it is.

Suggested change

'interval': '2'},

'interval': '2'}, # skip sys_db

janekmi · 2025-12-29T15:56:11Z

+        'id': '131584',
+        'probability_x': '100',
+        'probability_y': '100',
+        'interval': '2'},


Why you decided to skip the max_faults parameter for a number of faults?

Suggested change

'interval': '2'},

'interval': '2',

'max_faults': '1'},

janekmi · 2025-12-29T16:06:52Z

+        'id': '131586',
+        'probability_x': '100',
+        'probability_y': '100',
+        'interval': '2'},


The same here. Please apply to all instances below.

Suggested change

'interval': '2'},

'interval': '2'}, # skip sys_db

janekmi · 2025-12-29T16:09:55Z

+        'probability_x': '100',
+        'probability_y': '100',


As far as I know these are defaults so adding them to all of the records just bloats the code with no added value.

Suggested change

'probability_x': '100',

'probability_y': '100',

janekmi · 2025-12-29T16:18:10Z

+    'DAOS_FAULT_BTREE_OPEN_INV_CLASS': {
+        'id': '131590',
+        'probability_x': '100',
+        'probability_y': '100',
+        'interval': '28',
+        'max_faults': '1'},


This fault can be fine-tuned to hit either the containers tree (interval = 28) or the gc tree (interval = 29). Hence it seems we need two records with distinctive keys. I am sorry it was not obvious from the fault_injection_dlck.yaml file.

And IMHO we really need to give comments here explaining where 28 and 29 come from. Otherwise you won't reverse-engineer their meaning and considering these values are fine-tuned it could be crucial to adjust them in the future.

Suggested change

'DAOS_FAULT_BTREE_OPEN_INV_CLASS': {

'id': '131590',

'probability_x': '100',

'probability_y': '100',

'interval': '28',

'max_faults': '1'},

'DAOS_FAULT_BTREE_OPEN_INV_CLASS_28': {

'id': '131590',

'probability_x': '100',

'probability_y': '100',

'interval': '28', # containers tree fine-tuned

'max_faults': '1'},

'DAOS_FAULT_BTREE_OPEN_INV_CLASS_29': {

'id': '131590',

'probability_x': '100',

'probability_y': '100',

'interval': '29', # gc tree fine-tuned

'max_faults': '1'},

janekmi · 2025-12-29T17:56:25Z

+            if self.server_managers[0].manager.job.using_control_metadata:
+                dlck_cmd = DlckCommand(host, self.bin, pool_uuids[0], nvme_conf=nvme_conf,
+                                      storage_mount=scm_mount, env_str=env_str)
+            else:
+                dlck_cmd = DlckCommand(host, self.bin, pool_uuids[0], storage_mount=scm_mount,
+                                       env_str=env_str)


The same as above. There is no need to have an if just to provide nvme_conf=None for some cases. Please reduce.

janekmi · 2025-12-29T17:57:49Z

+            result = dlck_cmd.run()
+            if not result.passed:
+                errors.append(f"dlck failed on {result.failed_hosts}")
+            self.log.info("dlck basic test output:\n%s", result)


The same as with dumping the contents of fault injection file you are processing and printing the command result twice in this code. Please write a helper function.

janekmi · 2025-12-29T18:02:36Z

+            self.log.info("dlck basic test output:\n%s", result)
+        dmg.system_start()
+        if not errors:
+            self.fail("No Errors detected:\n{}".format("\n".join(errors)))


It is a very elaborate way of printing an empty list. 😉

Suggested change

self.fail("No Errors detected:\n{}".format("\n".join(errors)))

self.fail("No Errors detected.")

janekmi · 2025-12-29T18:03:22Z

+    1:
+      targets: 4
+      storage: auto


The same here. A single engine should be enough.

Oh, I see... I can reduce it to single engine testing.

janekmi · 2025-12-29T20:03:32Z

+        result = dlck_cmd.run()
+        if not result.passed:
+            errors.append(f"dlck failed on {result.failed_hosts}")
+        self.log.info("dlck basic test output:\n%s", result)


Suggested change

self.log.info("dlck basic test output:\n%s", result)

self.log.info(f"dlck basic test output:\n{result}")

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest

daosbuild3 · 2026-01-08T03:54:18Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/3/execution/node/932/log

daosbuild3 · 2026-01-08T04:09:12Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17307/3/testReport/

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

daosbuild3 · 2026-03-12T21:34:15Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/4/execution/node/966/log

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

daosbuild3 · 2026-03-31T22:08:14Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/5/execution/node/988/log

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

daosbuild3 · 2026-05-23T06:32:25Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/6/execution/node/867/log

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

daosbuild3 · 2026-05-23T19:58:48Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/7/execution/node/866/log

DAOS-17519 test: Automate dlck testing (basic/fault_injection)

e44ee36

Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

DAOS-17519 test: Fix linting issues.

1dfe0ef

Test-tag: DlckBasicTest DlckBasicFaultTest Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

janekmi requested changes Dec 29, 2025

View reviewed changes

DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519

a376c98

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest

rpadma2 added 2 commits March 12, 2026 11:26

DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519

1f16a51

DAOS-17519 test: Some of the review updates.

aa8ede9

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

rpadma2 added 3 commits March 31, 2026 10:49

DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519

5213ba6

DAOS-17519 test: Fix Linting issues.

0cea354

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

rpadma2 added 2 commits May 22, 2026 13:18

DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519

593f473

DAOS-17519 test: Update the control_meta_dir.

e4508aa

Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-unit-test: true Skip-fault-injection-test: true Skip-nlt: true Test-tag: DlckBasicFaultTest DlckBasicTest Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

	self.fail("No Errors detected:\n{}".format("\n".join(errors)))
	self.fail("No Errors detected.")

	self.log.info("dlck basic test output:\n%s", result)
	self.log.info(f"dlck basic test output:\n{result}")

Conversation

rpadma2 commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 8, 2026

Uh oh!

daosbuild3 commented Jan 8, 2026

Uh oh!

daosbuild3 commented Mar 12, 2026

Uh oh!

daosbuild3 commented Mar 31, 2026

Uh oh!

daosbuild3 commented May 23, 2026

Uh oh!

daosbuild3 commented May 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

rpadma2 commented Dec 23, 2025 •

edited

Loading

github-actions Bot commented Dec 23, 2025 •

edited

Loading