Is it possible to run a clustering algorithm with chunked distance matrices?

Question

I have a distance/dissimilarity matrix (30K rows 30K columns) that is calculated in a loop and stored in ROM.

I would like to do clustering over the matrix. I import and cluster it as below:

Mydata<-read.csv("Mydata.csv")
Mydata<-as.dist(Mydata)
Results<-hclust(Mydata)

But when I convert the matrix to dist object, I get RAM limitation error. How can I handle it? Can I run hclust algorithm in a loop/chunking? I mean I divide the distance matrix into chunks and run them in a loop?

In [ROM](https://en.m.wikipedia.org/wiki/Read-only_memory)? Seriously? Why? — Has QUIT--Anony-Mousse, Oct 29 '18 at 00:58
Also it certainly is possible. The question is how *fast* it will be. Maybe you can save some memory if you would *not* use R... or **choose an algorithm that does not need O(n²) distance matrixes**. — Has QUIT--Anony-Mousse, Oct 29 '18 at 01:00

score 0 · Answer 1 · answered Nov 03 '18 at 20:17

You may try the following:

Mydata<-read.csv("Mydata.csv")
Mydata<-as.matrix(Mydata)
Mydata<-as.dist(Mydata)
Results<-hclust(Mydata)

Read the following to track what's happening in your session: http://adv-r.had.co.nz/memory.html

This might be helpful in general: https://cran.r-project.org/web/packages/fastcluster/ And also this question: hclust() in R on large datasets

It also depends on your OS, but maybe you can change the RAM limit (or just run this code on someone else's computer with more RAM, store the object using saveRDS and then read it in your own computer using readRDS).

Is it possible to run a clustering algorithm with chunked distance matrices?

1 Answers1

Linked