Improving SVC prediction performance on single samples

Question

I have large-ish SVC models (~50Mb cPickles) for text classification and I am trying out various ways to use them in a production environment. Classifying batches of documents works very well (about 1k documents per minute using both predict and predict_proba). However, prediction on a single document is another story, as explained in a comment to this question:

Are you doing predictions in batches? The SVC.predict method, unfortunately, incurs a lot of overhead because it has to reconstruct a LibSVM data structure similar to the one that the training algorithm produced, shallow-copy in the support vectors, and convert the test samples to a LibSVM format that may be different from the NumPy/SciPy formats. Therefore, prediction on a single sample is bound to be slow. – larsmans

I am already serving the SVC models as Flask web-applications, so a part of the overhead is gone (unpickling) but the prediction times for single docs are still on the high side (0.25s). I have looked at the code in the predict methods but cannot figure out if there is a way to "pre-warm" them, reconstructing the LibSVM data structure in advance at server startup... any ideas?

def predict(self, X):
    """Perform classification on samples in X.

    For an one-class model, +1 or -1 is returned.

    Parameters
    ----------
    X : {array-like, sparse matrix}, shape = [n_samples, n_features]

    Returns
    -------
    y_pred : array, shape = [n_samples]
        Class labels for samples in X.
    """
    y = super(BaseSVC, self).predict(X)
    return self.classes_.take(y.astype(np.int))

Hi, I see what you mean, but I should have specified that it is a multiclass sentiment classification (very different class sizes). For the time being, I'm trying to reach the highest accuracy. So far, SVC with RBF kernels has outperformed every other classifier, although by a small margin (e.g. SVC 0.898, PassiveAggressiveClassifier 0.868, MultinomialNB 0.837). However, SVC largely outperforms the competition with the smallest classes (e.g. F1 SVC 0.84, PAC 0.76, MNB 0.68). If SVC were just a little faster with a single document, I would not see any reason not to use it with my current data. — emiguevara, Jan 30 '14 at 12:32

score 3 · Accepted Answer · edited Jun 20 '20 at 09:12

I can see three possible solutions.

Custom server

It is not the matter of "warming" anything up. Simply - libSVM is the C library, and you need to pack/unpack data into correct format. This process is more efficient on the whole matrices than on each row separately. The only way to overcome this would be to write more efficient wrapper between your production env and the libSVM (you could write a libsvm based server, which would use some kind of shared memory with your service). Unfortunately, this is to custom problem to be solvable by existing implementations.

Batches

Naive approach like buffering the queries is an option (if it is "high performance" system with thousands of queries, you can simply store them in N-element batches, and send them to libSVM in such packs).

Own classification

Lastly - classification using SVM is really simple task. You don't need libSVM to perform classification. Only training is a complex problem. Once you get all the support vectors (SV_i), kernel (K), lagragian multipliers (alpha_i) and intercept term (b), you classify using:

cl(x) = sgn( SUM_i y_i alpha_i K(SV_i, x) + b)

You can code this operation directly in your app, without the need to actualy pack/unpack/send anything to libsvm. This can speed things up by the order of magnitude. Obviously - probability is more complex to retrieve, as it requires the Platt's scaliing, but it is still possible.

Very helpful, thanks. I think that batches will be the solution for now, but I'll try classification as soon as I get the time :-) — emiguevara, Jan 29 '14 at 23:32

score 1 · Answer 2 · answered Jan 29 '14 at 10:39

1

You can't construct the LibSVM data structure in advance. When a request to classify a document arrives, you get the text of the document, make a vector out of if and only then convert to LibSVM format so you can get a decision.

LinearSVC should be considerably faster than a SVC with a linear kernel as it uses liblinear. You could try using a different classifier if that does not decrease performance too much.

answered Jan 29 '14 at 10:39

mbatchkarov

15,487
9
60
79

Of course, you cannot avoid processing the one document you get on request. However, the difference in performance depending on number of samples is so great that I still wonder if something can be done in advance. For example, calling both `predict` and `predict_proba` on each document: 100 docs 5.6157s, 10 docs 0.9705s, 2 docs 0.4969s, 1 doc 0.4551s – emiguevara Jan 29 '14 at 12:58
Changing classifier was not a part of the question. – emiguevara Jan 29 '14 at 12:59
`LinearSVC` is just an optimised version of `SVC`, so you are not really changing the classifier. http://stackoverflow.com/questions/11508788/whats-the-difference-between-libsvm-and-liblinear – mbatchkarov Jan 29 '14 at 13:48
Are you aware of the differences between the options of `LinearSVC` and `SVC` (read: non-linear kernels)??? I repeat: this was not a part of my question, no need to suggest changing the classifier. – emiguevara Jan 29 '14 at 14:58

Improving SVC prediction performance on single samples

2 Answers2

Custom server

Batches

Own classification