clustering on very large sparse matrix?

Question

I am trying to do some (k-means) clustering on a very large matrix.

The matrix is approximately 500000 rows x 4000 cols yet very sparse (only a couple of "1" values per row). I want to get around 2000 clusters.

I got two questions: - Can someone recommend an open source platform or tool for doing that (maybe using k-means, maybe with something better)? - How can I best estimate the time the algorithm will need to finish? I tried weka once, but aborted the job after a couple of days because I couldn't tell how much time it would take.

Thanks!

Is this for a real time application or for your own self development? — nsivakr, Aug 26 '10 at 23:01
Reasked in http://stackoverflow.com/questions/3039646/k-means-clustering-in-r-on-very-large-sparse-matrix . — Andrew Dalke, Dec 20 '11 at 20:04

score 1 · Answer 1 · answered Jun 16 '13 at 21:17

For your case, I guess your problem is only in the size of the input.

I would suggest "cluto" as a good tool for large and sparse dataset. It is written in C. I have tried around 17 millions of rows with around 400 cols. And it works fast.

Link of the Cluto library

score 1 · Answer 2 · answered Aug 26 '10 at 23:00

1

http://lucene.apache.org/mahout/

answered Aug 26 '10 at 23:00

Doug

1,446
4
14
26

Please don't post just links as answers, explain why this answers the question of the OP – Gabriel Jun 02 '22 at 21:13

score 0 · Answer 3 · answered Aug 08 '14 at 17:00

0

You can try sparcl package in R, it implements sparse k-means and hierarchical clustering. Not so easy to understand tough

answered Aug 08 '14 at 17:00

svural

961
1
9
17

1

be careful, sparcl is 'sparse' in feature selection and does not address the n^2 storage for the similarity matrix. – Chris Dec 03 '14 at 20:47

clustering on very large sparse matrix?

3 Answers3

Linked