Prediction Warnings when predicting a fitted caret SVM model

Question

I'm hoping to get some pointers as to why I'm getting :

Warning message: In method$predict(modelFit = modelFit, newdata =
newdata, submodels = param) :   kernlab class prediction calculations
failed; returning NAs

When I print out the prediction:

svmRadial_Predict
  [1] <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>....

The code I wrote to perform the SVM fitting:

    #10-fold cross validation in 3 repetitions 
    control = trainControl(seeds = s, method="repeatedcv", number=10,
repeats=3, savePredictions = TRUE, classProbs =  TRUE)

The SVM model for the fitting is like this:

svmRadial_model = train(y=modelTrain$Emotion,
                        x=modelTrain[c(2:4)],
                        method ='svmRadial',
                        trControl = control,
                        data=modelTrain,
                        tuneLength = 3
                        )

And the code I wrote to perform the prediction looks like this:

    svmRadial_Predict <- predict(svmRadial_model, 
newdata = modelTest[c(2:4)], probability = TRUE )

I've checked the data, and there's no NA values in the training or testing set. The y value is a factor and the x values are numeric if that makes a difference? Any tips to debug this would be very much appreciated!

As the model trains I can see warnings like this:

line search fails -1.407938 -0.1710936 2.039448e-05

which I had assumed was just the model being unable to fit a hyperplane for particular observations in the data. I'm using the svmRadial kernel The data I'm trying to fit was already centred and scaled using the R scale() function.

Further work leads me to believe it's something to do with the classProbs = TRUE flag. If I leave it out, no warnings are printed. I've kicked off another run of my code, SVM seems to take ages to complete on my laptop for this task but I'll report the results as soon as that completes.

As a final edit, the model fitting completed without error, and I can use that model just fine for prediction/calculating the confusion matrix etc. I don't understand why including classProbs = TRUE breaks it, but maybe it's related to the combination of the cross validation that does, with the cross validation I had requested in my trainControl

Answer 1

Your problem is part of the peculiarities of the caret package.

There are two potential reasons why your prediction fails with kernlab svm methods called by caret:

1. The x, y interface returns a caret::train object which the predict function cannot use.

Solution: Simply replace by the formula interface.

train(form = Emotion ~ . , data = modelTrain, ...

---

2. The iterative search within the support vector machine algorithm doesn't converge.

Solution 2a)

Set different seeds before the train() call until it converges.

set.seed(xxx)
train(form = Emotion ~ . , data = modelTrain, ...)

Solution 2b)

Decrease parameter minstep as suggested by @catastrophic-failure here. For this solution, there is no parameter in the ksvm function, so you need to change the source code in line 2982: minstep <- 1e-10 to a lower value. Then compile the source code yourself. No guarantee it will help though.

Try out Solution 1 first, as it is the most likely!

Answer 2

My solution to this was just to leave out the classProbs = TRUE parameter of the trainControl function. Once I did that, everything worked. I'd guess it's related to what's happening with cross validation under the hood but I'm not certain of that.