0

I wanted to cluster my data-set using carrot2 workbench. I have an input xml file with 65536 documents. I am using Lingo clustering algorithm.

But when I start the process, the workbench returns the result within few seconds having all the documents in the "other topics" cluster.

I have checked the clustering with smaller data-sets and I am getting the results.

vishalaksh
  • 2,054
  • 5
  • 27
  • 45

1 Answers1

0

Carrot2 Lingo algorithm was designed for small data sets, up to a thousand or so of documents. For larger data sets, you may want to try STC, which scales better.

Regardless of the algorithm, Carrot2 processes all data in-memory, so it will not scale to millions of documents. In the latter case you may want to look at Apache Mahout, for example.

Stanislaw Osinski
  • 1,231
  • 1
  • 7
  • 9