1

I was checking out this post (Representing a 100K X 100K matrix in Java) and implemented the solution regarding Maps and floats. However, the heap still runs out of memory if I fill the map with every entry in a 100K by 100K matrix. Is there a better solution without having to resort to 3rd party solutions (and also besides just increasing the heap allocation)?

I would like to implement this for a graph, by the way. I work with huge graphs.

bayesianpower
  • 93
  • 2
  • 8
  • 100k x 100k = 10 billions items ; if you don't have enough memory to store that in the first place, that sounds difficult. – Déjà vu Feb 21 '20 at 05:30
  • 1
    Does it need to be loaded into memory at the same time? Could you use a database? – ernest_k Feb 21 '20 at 05:30
  • Can you physically fit all the items into heap irrespective of structure? If not then you'll need to look at a paging mechanism to disk or compression mechanisms. Neither of those are simple to implement efficiently. – sprinter Feb 21 '20 at 05:34
  • If the matrix has lots of repetitive sequences of numbers, you could "format" it like e.g. `5x(123,4x(456))` for `123,456,456,456,456,123,456,456,456,456,123,456,456,456,456,123,456,456,456,456,123,456,456,456,456`. Access will be slow and more difficult compared to a direct [x][y] access. Trade-off between cpu and memory... (and algorithmics!) But you could have a separate index that points to the beginning of a row (or col). – Déjà vu Feb 21 '20 at 05:46
  • The most memory-efficient Java data structure for a dense matrix is a 2d array. It sounds like you want `float`s, so it's `float [][] a = new float[100_000][100_000];` This data structure is 40 gigabytes in size. That means your machine had better have at least 60 gigabyes of RAM or so. Also make sure you're using 64-bit Java. You'll probably need to fiddle with JVM heap size parameters. If the data are mostly a single value or have other patterns, then a sparse matrix data structure is a much more efficient choice as others have said, but the way youv'e asked implies that's untrue. – Gene Feb 21 '20 at 06:26

0 Answers0