dataProcess() function, when using numOfCores > 1, was copying the input data to all the worker processes. That becomes a real problem when working with a very large input dataset.
In addition, the parallel processing loop was done over individual proteins, instead of over chunks of data.
The proposed change, using packages foreach and doParallel, splits the data into chunks and only sends one chunk to each of the workers.
dataProcess()function, when usingnumOfCores > 1, was copying the input data to all the worker processes. That becomes a real problem when working with a very large input dataset.In addition, the parallel processing loop was done over individual proteins, instead of over chunks of data.
The proposed change, using packages foreach and doParallel, splits the data into chunks and only sends one chunk to each of the workers.