DAOS-18235 test: pool/verify_space.py - Remove system_ram_reserved

[shimizukko]

The test intermittently fails during storage format due to "Available memory (RAM) insufficient" error. This is because too little memory is allocated to non-DAOS system processes due to "system_ram_reserved: 1" (unit is GiB) in server config (test yaml). If we increase it, the system would have more space. The error message indicated that "want 159 GiB RAM but only have 158 GiB", increasing it by 1 should resolve, but use the default 64GiB by removing the field because that's the common real-world use case.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_verify_pool_space
Test-repeat: 5

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'pool/verify_space.py:VerifyPoolSpace.test_verify_pool_space - Available memory (RAM) insufficient for configured 177 GiB ram-disk size'
Status is 'In Review'
Labels: '2.8.0tb1,ci_2.8_weekly,ci_master_weekly,weekly_test'
https://daosio.atlassian.net/browse/DAOS-18235

[shimizukko]

Removing system_ram_reserved field would allocate the default 64GiB to system, but we just need 1 GiB more, so use 5 instead.

[phender]

Won't increasing the reserved memory make the "Available memory (RAM) insufficient" error worse? We should also target testing on the hdr-[142-145] cluster - where this failure has been seen a few times.

[phender]

I brought this up in the MD on SSD WG and the recommendation is to not specify a system_ram_reserved value for HW tests. Since this will result in the default value being used, it should also be a better representation of a real-world test.

We have passing results in https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18298/1/artifact/Functional%20Hardware%20Medium%20MD%20on%20SSD/pool/verify_space.py with system_ram_reserved removed that ran on the hdr-[142-145] cluster.

[shimizukko]

I talked to @tanabarr and it sounds like "Available memory (RAM) insufficient" error is coming from the system (i.e., non-DAOS), so if we increase system_ram_reserved, the system would have more RAM and it'll be resolved. I removed the field. Thanks.

[daltonbohning]

I agree with Phil that we should use the default if that works