one_to_all and all_to_one memcpy measurement operations are launched before "warmup" iterations are finished

[matthewpopescu]

In one_to_all_read and all_to_one_write testcases (both sm and ce), when stream 0 is much faster than the other streams, memcpy operations are not launched at the same time, which taints the metric collected due to overlapping "warmup" and "measurement" from separate streams.

This is due to how "warm up" runs are done in doMemcpyCore. Stream 0, if it is fastest, does not "wait" on any of the warmups to finish. Any stream other than "stream 0" has to "wait" for stream 0 to start.

So in the case that stream 0 is fastest, it finishes its warmup and immediately starts the measurement phase before stream 1,2,3... finish their warmup.

In the case that stream 0 is NOT fastest, every other stream's warmup finishes first, and then "waits" for stream 0 measurement to start, correctly aligning the measurement.

I've profiled the testcase with nsys to show how this is happening.

Incorrect alignment when stream0 is fastest and doesn't wait for 1 or 2




Correct Alignment when stream 0 and 1 are slowest, so stream 2 waits after warmup

memcpy SM GPU(row) <- GPU(column) bandwidth (GB/s)
          0         1         2         3
0     40.87     40.84     44.39     44.36    
## GPUs 0 and 1 show poor performance due to stream0 launching before others are done with warmup
## GPUs 2 and 3 show good performance, because all streams are waiting for warmup to end before launching

I've attempted my own fix by creating "warmupEndEvents" in memcpy.cpp for all streams to wait before beginning the measurement, and this fixes the issue in both cases (stream 0 fastest or slowest)
Stream 0 fastest, now correctly aligned measurement start



Stream 2 fastest, but still waits for others after warmup


Here are the results from my modified version:

memcpy SM GPU(row) <- GPU(column) bandwidth (GB/s)
           0         1         2         3
 0     44.37     44.33     44.39     44.37
## All GPUs show identical performance

[esitaridi]

@matthewpopescu Thanks for the detailed investigation and nsys traces. We will be fixing this issue in an upcoming release.