5

I have text as shown :

 list1 = ["My name is xyz", "My name is pqr", "I work in abc"]

The above will be training set for clustering text using kmeans.

list2 = ["My name is xyz", "I work in abc"]

The above is my test set.

I have built a vectorizer and the model as shown below:

vectorizer = TfidfVectorizer(min_df = 0, max_df=0.5, stop_words = "english", charset_error = "ignore", ngram_range = (1,3))
vectorized = vectorizer.fit_transform(list1)
km=KMeans(n_clusters=2, init='k-means++', n_init=10, max_iter=1000, tol=0.0001, precompute_distances=True, verbose=0, random_state=None, copy_x=True, n_jobs=1)
km.fit(vectorized)

If I try to predict the cluster for my test set of "list2":

km.predict(list2)

I get the error below:

ValueError: Incorrect number of features. Got 2 features, expected 5

I was told to use Pipeline to solve this issue. So I wrote the code below:

pipe = Pipeline([('vect', vectorizer), ('vectorized', vectorized), ('kmeans',km )])

But I get the error:

TypeError                                 Traceback (most recent call last)
/mnt/folder/Text_Mining/<ipython-input-14-321cabc3bf35> in <module>()
----> 1 pipe = Pipeline([('vect', vectorizer), ('vectorized', vectorized), ('kmeans',km )])
/usr/local/lib/python2.7/dist-packages/scikit_learn-0.13-py2.7-linux-x86_64.egg/sklearn/pipeline.pyc in __init__(self, steps)
     87                 raise TypeError("All intermediate steps a the chain should "
     88                                 "be transforms and implement fit and transform"
---> 89                                 "'%s' (type %s) doesn't)" % (t, type(t)))
     90
     91         if not hasattr(estimator, "fit"):
TypeError: All intermediate steps a the chain should be transforms and implement fit and transform'  (0, 2)     1.0
(1, 4)        0.57735026919
(1, 3)        0.57735026919
(1, 1)        0.57735026919
(2, 0)        1.0' (type <class 'scipy.sparse.csr.csr_matrix'>) doesn't)

I think that maybe the output of vectorized does not implement a fit and transform, but how do I do that in this particular case? I'm new to Machine Learning. Also, how to get the labels from the kmeans model? When I run kmeans, I can access the cluster labels by using km.labels_. How to do something similar in Pipeline?

Alexander L. Hayes
  • 3,892
  • 4
  • 13
  • 34
user1452759
  • 8,810
  • 15
  • 42
  • 58
  • 1
    The `('vectorized', vectorized)` part is not a valid part of a pipeline. In a pipeline you only want objects that have a `fit` and for all but the last a `transform` method. You don't call `fit` on the constituents like you have done, you should call it on `pipe`: `pipe.fit(list1)` – eickenberg Nov 13 '14 at 09:47

2 Answers2

3

What you need to do is to train a vectorizer using list1, then with the same vectorizer, transform list1 and list2. This will solve the issue. Demo:

>>> list1 = ["My name is xyz", "My name is pqr", "I work in abc"]
>>> list2 = ["My name is xyz", "I work in abc"]
>>> vectorizer = TfidfVectorizer(min_df = 0, max_df=0.5, stop_words = "english", charset_error = "ignore", ngram_range = (1,3))
>>> vec=vectorizer.fit(list1)   # train vec using list1
>>> vectorized = vec.transform(list1)   # transform list1 using vec
>>> km=KMeans(n_clusters=2, init='k-means++', n_init=10, max_iter=1000, tol=0.0001, precompute_distances=True, verbose=0, random_state=None, cpy_x=True, n_jobs=1)
>>> km.fit(vectorized)
>>> list2Vec=vec.transform(list2)  # transform list2 using vec
>>> km.predict(list2Vec)
array([0, 0], dtype=int32)
Irshad Bhat
  • 8,479
  • 1
  • 26
  • 36
  • 1
    I'd still love to see this implemented via Pipeline. Doesn't that make it easier (a one-line process) to run new data thru the same steps, e.g. for prediction, and easier to add new steps in the middle as needed? – nealmcb Apr 26 '17 at 00:10
  • 1
    @nealmcb Me too! So... see below. :-) – G__ Jul 24 '20 at 21:47
2

You were very close! Skip the explicit vectorizer.fit() step in the middle, and do it all in the pipeline:

list1 = ["My name is xyz", "My name is pqr", "I work in abc"]
list2 = ["My name is xyz", "I work in abc"]
vectorizer = TfidfVectorizer(min_df = 0, max_df=0.5, stop_words = "english", ngram_range = (1,3))
km = KMeans(n_clusters=2, init='k-means++', n_init=10, max_iter=1000, tol=0.0001, precompute_distances=True, verbose=0, random_state=None, copy_x=True, n_jobs=1)
pipe = Pipeline([('vect', vectorizer), ('kmeans', km)])
pipe.fit(list1)
pipe.transform(list2)

Result:

array([[0.70710678, 1.41421356], [0.70710678, 1.41421356]])

  • Note: tweaked a couple parameters so it'll run on my current version of sklearn (0.22.1)... it has been a few years :-)
G__
  • 7,003
  • 5
  • 36
  • 54