Feature Request: Add Support for Sharded Test Runs (Shard Count and Shard Index)

[roxell]

Running large sub-suites like syscalls on a single device can be very time-consuming (e.g., ~60 minutes on hardware X).
I would like kirk to support sharding, meaning:

• The full test set can be split into multiple shards (e.g., 10 shards).
• Each boot (device) would be responsible for running one shard.
• Each boot is given two parameters:
  • shard_count: the total number of shards (e.g., 10)
  • shard_index: which specific shard this boot should run (from 1 to 10)

Behavior Example:

• Configure shard_count=10.
• Boot 10 devices, each launched with a unique shard_index (1, 2, ..., 10).
• Each device runs its assigned subset of tests.

Benefits:

• Huge reduction in total test time (e.g., from 60 minutes down to ~6 minutes).
• Better utilization of available hardware.
• Much faster CI/CD feedback for developers and testers.

[pevik]

This might get quite complex, I'm not sure if we find it interesting enough to work on it (better question for @acerv and @metan-ucw).

FYI we planned to get rid of runtest files: #13 (long term plan, it takes long time). That's why it might be worth of considering. IMHO it'd not save time in our normal testing, when we run whole LTP (many runtests). But maybe for testing just single big runtest file (syscalls). We usually test just specific tests (filter by --run-pattern), when we want to verify something. What you propose could be implemented with it. But sure have an option would be much easier to use.

Other plan (probably implemented faster) is: #12.

FYI @acerv present kirk plans on our regular public LTP meeting (https://lore.kernel.org/ltp/63e8ded5-9aff-4e47-9bb7-13c5c1fb80e9@suse.com/), feel free to join. Or we could come to Thursday OATS monthly call.

[metan-ucw]

@pevik this is not actually tied to runtest files, this feature request is about partitioning tests into several groups and running them in parallel on different machines, which is rather clever way how to run the tests fast without worrying about which tests cannot be run at the same time. This sounds like a great solution as long as you have enough machines or can run enough VMs.

Also we want to have some kind of multi-machine support in kirk eventually, this shouldn't be hard as long as we have proper abstraction for a SUT and can spin a few of them. I'm not sure how this can be added to the scheduler though, this is something that we need to discuss with @acerv.

[acerv]

This is an interesting feature. It will probably require the following actions:

• to provide a configuration file defining nodes and the number of shard_count
• to use a SUT class implementation for non-local testing (asyncssh, LTX, maybe Qemu)
• to have a layer in between SUT and Scheduler that abstract multiple targets

Configuration file must be a python native configuration format, such as JSON or ini (this second one is the most human-readable). Yaml would be nice, but then we would need a new external library dependency we don't really want.

In general, I think it would be more beneficial than parallel testing, but I don't have time for this at the moment. Let's keep it as a nice-to-have for future implementations.

[metan-ucw]

@acerv maybe we can do it slightly differently because partitioning the work in advance is going to be wasteful, you cannot really split the work in advance so that the workers finish at the same time. Optimal solution would be to provide more than one SUT to the kirk and the scheduler will schedule work on workers as needed. This is not that much different than parallel runs, but instead of running tests in parallel on single machine you run tests on N machines in parallel.

[acerv]

@metan-ucw you still need an abstraction class between SUT and Scheduler. Which means to handle multiple SUT at the same time.
It's not hard, it's just a matter of doing it properly. Once the architecture is ready, to run tests and to wait for the results is as easy as await async.gather().

[metan-ucw]

@acerv I'm not sure if abstraction is the right word, the scheduler has to understand that there are more machines in order to schedule the tests properly, because unlike the parallel runs that we do on a single machine there are no constraints in this case.

[acerv]

Actually, the right approach would be to implement a new MultiMachineScheduler which is using multiple SuiteScheduler implementations initialized with proper SUT and configurations.

[metan-ucw]

@acerv I think that we want a single suite (i.e. queue) but multiple consumers.

[acerv]

I finally had time to implement this. Please take a look at #107