Error in check for remote errors (val): 5 nodes produced an error: object not found

Question

Im trying to do a 10-fold cross validation and estimate the model performance of a joint model by using parallel processing (parLapply). Im trying to find out why I receive the error message: "Error in checkForRemoteErrors(val): five nodes produced an error: object 'Week' not found"

The code looks as follows:

# Validation using 10-fold CV
    library("parallel")
    set.seed(123)
    V <- 10
    n <- nrow(dfC)
    splits <- split(seq_len(n), sample(rep(seq_len(V), length.out = n)))
    CrossValJM <- function (i) {
        library("JM")
        library("nlme")
        trainingData <- dfL[!dfL$ID %in% i, ]
        trainingData_ID <- trainingData[!duplicated(trainingData$ID), ]
        testingData <- dfL[dfL$ID %in% i, ]

        lmeFit <- lme(DA ~ ns(Week, 2), data = trainingData,
                           random = ~ ns(Week, 2) | ID)
        coxFit <- coxph(Surv(TT_event, Event) ~ Gender * Age, data = 
                           trainingData_ID, 
                             x = TRUE)

        jointFit <- jointModel(lmeFit, coxFit, timeVar = "Week")

        pe <- prederrJM(jointFit, newdata = testingData, Tstart = 10, 
                                                  Thoriz = 20)
        auc <- aucJM(jointFit, newdata = testingData, Tstart = 10, 
                                                  Thoriz = 20)
        list(pe = pe, auc = auc)
    }

    cl <- makeCluster(5)
    res <- parLapply(cl, splits, CrossValJM)
    stopCluster(cl)

The function itself gets accepted but when running the Cluster commands I run into this error that mentions that it cannot recognize objects given within the function.. should they be defined within the function itself?? Or am I not using the parLapply function correctly?

P.S.: data looks as follows (dfL is a dataframe of length ~ 1000 and dfC ~ 200):

dfL <- data.frame(ID = c(1, 1, 1, 2, 2, 3), DA = c(0.4, 1.8, 1.2, 3.2, 3.6, 2.8), Week = c(0, 4, 16, 4, 20, 8), Event = c(1, 1, 1, 0, 0, 1), TT_Event = c(16, 20, 8), Gender = c(0, 0, 0, 1, 1, 0), Age = c(24, 24, 24, 56, 56, 76))

dfC <- data.frame(ID = c(1, 2, 3, 4, 5, 6), DA = c(1.2, 3.6, 2.8, 2.4, 1.9, 3.4), Week = c(16, 20, 8, 36, 24, 32), Event = c(1, 0, 1, 1, 1, 0), TT_Event = c(16, 20, 8, 36, 24, 32), Gender = c(0, 1, 0, 0, 1, 1), Age = c(24, 56, 76, 38, 44, 50))

Thnx :)

Answer 1

Very related questions have already been answered on Stack Overflow. Basically, you have three solutions:

• use clusterExport() to export the variables you need to the clusters (the most common method)
• pass all variables as arguments of your function CrossValJM() so that they are automatically exported to the clusters (the solution I prefer, the most programmatically correct one)
• use R package {future} which should detect automatically variables to export (the lazy solution, but seems to work well also)

See for example this.