How to fix java.lang.OutOfMemoryError: Java heap space error?

Question

I have a file with size of 32 MB, I have downloaded it from DocuShare server to DocuShare temp folder. I am trying to read the file content from it to create a file. I get error when I URL encode my base64 content. I am not getting any exception when I run the same code a simple java application. But when I use the same code in DocuShare service to get document content I get Exception. HTTP Status 500 - org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space

org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space

File file = new File(filePath);
FileInputStream fileInputStreamReader = new FileInputStream(file);
byte[] bytes = new byte[(int)file.length()];
fileInputStreamReader.read(bytes);
String encodedBase64 = String encodedBase64 = java.util.Base64.getEncoder().encodeToString(bytes);
String urlEncoded = URLEncoder.encode(encodedBase64);

How to fix this error? Do I need to increase my tomcat heap size?

https://stackoverflow.com/questions/1565388/increase-heap-size-in-java — Akash Shah, Mar 25 '19 at 05:43
First you should determine what your heap usage actually is when you get this error. It seems you're storing several copies of this file's contents. (Additionally, you should probably just use the URL-safe Base64 character set in the first place.) — chrylis -cautiouslyoptimistic-, Mar 25 '19 at 05:44

score 2 · Answer 1 · answered Mar 25 '19 at 06:02

2

There are two ways in which you can fix the issue.

You can increase the heap size, but IMO this is a bad solution, because you will hit the same issue if you get several parallel requests or when you try to process a bigger file.

You can optimize your algorithm - instead of storing several copies of your file in-memory, you can process it in a streaming fashion, thus not holding more than several KBs in memory:

import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Base64;

public class Launcher {
    public static void main(String[] args) throws Exception {
        final Path input = Paths.get("example");
        final Path output = Paths.get("output");

        try (InputStream in = Files.newInputStream(input); OutputStream out = Base64.getUrlEncoder().wrap(Files.newOutputStream(output))) {
            final byte[] buffer = new byte[1024 * 8];

            for (int read = in.read(buffer); read > 0; read = in.read(buffer)) {
                out.write(buffer, 0, read);
            }
        }
    }
}

PS: If you really need the URL encoder, you'll have to create a streaming version of it, but I think a URL-safe base64 would be more than enough

answered Mar 25 '19 at 06:02

Svetlin Zarev

14,713
4
53
82

Increasing heap size! Weird! Why would you suggest something you yourself stated as bad solution? Let me point out, this is no programmatic solution at all. You should remove your point 1. – Mar 25 '19 at 06:13
3

@MohammadRakibAmin I did not recommend it - just the opposite. I have only mentioned it as a possible solution. – Svetlin Zarev Mar 25 '19 at 06:19
2

Good answer. It's important to mention increasing the heap size, because OP mentioned it. Increasing -Xmx will make the OOM error go away at small scale. Solution 2 here is a better answer because it reduces the application's peak heap requirement. Jersey in a servlet container, returning a 32MB file as a single chunk can also end up allocating a big non-heap Direct Buffer for each request thread that serves a big file. Some containers hold these after completing the request until the request thread is killed, which could be never. – joshp Mar 25 '19 at 06:53
@MohammadRakibAmin it's an option. It may not be (generally) a good option, but it is an option. And in some cases it may well be essential, for example if you have a fixed amount of resources you MUST keep in memory and the total allocated isn't enough to hold that. – jwenting Mar 25 '19 at 08:12
For your information, you are ignoring the detailed Answer in the [Duplicate](https://stackoverflow.com/a/37349/2453382), utilizing a finite storage. On top of it, OP asked if he needs to increase the allotted heap size. He didn't ask how to increase heap size. Otherwise I won't go barking at the wrong tree. Just turning off OutOfMemory error can not be a programmatic solution in a question titled "How to Fix." – Mar 25 '19 at 10:49
I came across this question, not duplicate, may be my google-ing is bad. But I decided to read this page anyway. And found a not so good answer followed by a good answer. I think that is all. Not really sure how "options" == "right answers". May be you can add a comment suggesting an "option". Not add an option as an answer. – Mar 25 '19 at 10:52

score 2 · Accepted Answer · answered Mar 25 '19 at 06:13

Base64 converts each 3 bytes into 4 letters. That means you can read your data in chunks and decode it in the same way as you would decode the whole file.

Try this:

       File file = new File(filePath);
       FileInputStream fileInputStreamReader = new FileInputStream(file);
       StringBuilder sb = new StringBuilder();
       Base64.Encoder encoder = java.util.Base64.getEncoder();
       int bufferSize = 3 * 1024; //3 mb is the size of a chunk
       byte[] bytes = new byte[bufferSize]; 
       int readSize = 0;

       while ((readSize = fileInputStreamReader.read(bytes)) == bufferSize) {
            sb.append(encoder.encodeToString(bytes));
       }

       if (readSize > 0) {
            bytes = Arrays.copyOf(bytes, readSize);
            sb.append(encoder.encodeToString(bytes) );
       }

       String encodedBase64  = sb.toString();

score 0 · Answer 3 · answered Mar 25 '19 at 05:44

If you have large files, you will always run into OOM errors depending on size of file. If your goal is to base64 encoding using Apache Commons Base64 Streams.

https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html

How to fix java.lang.OutOfMemoryError: Java heap space error?

3 Answers3