Java: Serialization of large Guava Table object fails with out of memory error

Question

I have a large Guava Table object that I am trying to serialize:

import com.google.common.collect.Table;
Table<Integer, Integer, Double> pTable;
//
// Put lots of stuff into pTable
//
FileOutputStream fos = new FileOutputStream("pTable.ser");
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(pTable);

At the last line I got:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.io.ObjectOutputStream$HandleTable.growEntries(ObjectOutputStream.java:2346)
    at java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2275)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at java.util.HashMap.writeObject(HashMap.java:1129)
    at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at java.util.HashMap.writeObject(HashMap.java:1129)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)

Is there a fix/workaround for this? Would it also work for deserialization?

By the way, pTable represents a dense matrix of size about 30000 X 30000 where any pair of integers, each of which is from the set of 30000, maps into a float "distance" between them. If there is a more compact way of representing this matrix than Guava's Table that could be created in a reasonable time and allow for quick lookup, I would be interested.

Why not just use a `double[30000][30000]`? – Louis Wasserman Apr 03 '16 at 01:36 — Louis Wasserman, Apr 03 '16 at 01:36

score 1 · Accepted Answer · edited May 23 '17 at 12:32

1

You need to Increase heap size in Java.

e.g.

java -Xmx16g pTableProgram.jar

However, if you are using [0..30000) as your row keys and column keys then a double[][] would use less memory and if you don't need the precision of a double then a float[][] would use even less.

e.g.

double[][] pTable = new double[30000][];
for (int i = 0; i < pTable.length; i++) {
    pTable[i] = new double[30000];
}
//
// Put lots of stuff into pTable
//
FileOutputStream fos = new FileOutputStream("pTable.ser");
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(pTable);

edited May 23 '17 at 12:32

Community

1
1

answered Apr 01 '16 at 20:10

mfulton26

29,956
6
64
88

1

Even if the keys are not directly the 0..30000 range, I'd still go with the arrays and use a `Map` to give the real values an index in the 0..30000 range. You can use a primitive `Map` from fastutil, Trove, HPPC, etc. to avoid the boxing and reduce the memory usage. – Frank Pavageau Apr 02 '16 at 21:35

Java: Serialization of large Guava Table object fails with out of memory error

1 Answers1