0

I have a simple method that creates a tar file with a collection of compressed files inside:

private byte[] gzipCompress() throws IOException {
    File source = super.getPath();
    
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    try (GZIPOutputStream gos = new GZIPOutputStream(baos)) {
        Files.copy(source.toPath(), gos);
    }
    
    return baos.toByteArray();
}

When the file is large (not sure of exact size 1GB maybe) - this method causes an error:

Caused by: java.lang.OutOfMemoryError: Java heap space

Seems similar to this question - java nio Files.copy thowing Java heap space out of memory, but is there some other alternative to the IOUtils.copyLarge solution.

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
user1472672
  • 313
  • 1
  • 9
  • 3
    A `ByteArrayOutputStream` is a memory backed, growing, that is even array copying, data structure. No wonder. You could copy to a temporary file. Alternatively you could assume a 10% compression, and do `new ByteArrayOutputStream(100_000)` with a sufficient initial capacity reserved. – Joop Eggen Nov 04 '22 at 12:40
  • 2
    @JoopEggen Presizing the BytArrayOutputStream will likely just move the OutOfMemoryError to baos.toByteArray(). – Mark Rotteveel Nov 04 '22 at 14:02
  • @MarkRotteveel likely as then the memory is duplicated too, however many small reallocations (much slower too), will reallocate to _more_ than needed, so instead of double memory it might be triple memory – Joop Eggen Nov 04 '22 at 17:02
  • Don't try to load entire large files into memory. At some point it *must* fail. Rethink. – user207421 Apr 04 '23 at 10:12

0 Answers0