R Parallel processing terminating prematurely

advertisements

I am having an issue with parallel processing and the foreach function in R. I am registering 6 clusters immediately before running my foreach code and monitoring the progress. I see intially my CPUs spike to 70% (100% expected since im registering all 6 cores), but after a few seconds it decreases and seems to be finishing the rest of the job on a single core. I have run the same bit of code on my linux machine with the doMC package where I get 100% of cpu for about 10 seconds and then it decreases and finishes on one core. This leads me to believe the problem is how I have written my code.

I have a data set of locations with beginning and end lat/lon columns. I am using the geophere package to calculate a great cirlce path from beginning to end for each row which expands each original row in the data set to 12 rows after which I am rbinding the results back into a data frame for plotting.

My entire data set should be around 35M rows when complete so I would really like to take advantage of the multiple cores.

library(foreach)
library(doParallel)

cl = makeCluster(6)
registerDoParallel(cl) 

arc = foreach (i = temp$id, .combine = rbind, .packages = "geosphere") %dopar% {
    data.frame(
    id = rep(temp[i, 9], 12),
    supplier = rep(temp[i, 1], 12),
    receiving_facility = rep(temp[i, 2], 12),
    commodity = rep(temp[i, 3], 12),
    weight = rep(temp[i, 4], 12),
    time_period = rep(temp[i, 5], 12),
    lat = gcIntermediate(p1 = c(temp$supp_lon[i], temp$supp_lat[i]),
                         p2 = c(temp$rec_lon[i], temp$rec_lat[i]), n = 10,
                         addStartEnd = TRUE)[,2],
    lon = gcIntermediate(p1 = c(temp$supp_lon[i], temp$supp_lat[i]),
                         p2 = c(temp$rec_lon[i], temp$rec_lat[i]), n = 10,
                         addStartEnd = TRUE)[,1])
    }
stopCluster(cl)


I think this would be better as a comment but I do not have the reputation for it yet.

To my understanding, and from my experience, it is working as intended. Running the code in parallel. However, the row binding function (i.e. putting it all back together) is done on a single core which is why you see that extra single-core processing time at the end.