"java.lang.OutOfMemoryError: Java heap space" while uploading 1+ GB file in spring boot application

Question

I built spring boot app exactly as explained in this tutorial for uploading single file. The changes I have made are as explained below.

I have added following properties to application.properties:

spring.http.multipart.max-file-size=2048MB
spring.http.multipart.max-request-size=2048MB

Then I added following jvm parameters to projects run configuration to limit jvm heap size to 2GB as explained in this answer:

-Xmx2048m

Next I also ensured that my spring boot app is running against 64 bit java as it seems that 32 bit java needs "contiguous" heap space to reserve.

But still when I was trying to upload big file, it gives me same error.

First I tried uploading 20MB file. It worked. Next I kept trying ubuntu iso worth 1GB, but it kept giving me following exception:

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_74]
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_74]
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_74]
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_74]
    at org.springframework.util.StreamUtils.copy(StreamUtils.java:128) ~[spring-core-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109) ~[spring-core-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156) ~[spring-core-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest$StandardMultipartFile.getBytes(StandardMultipartHttpServletRequest.java:291) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at com.digitate.ignio.spring_boot_hdfs_file_upload.controller.UploadController.singleFileUpload(UploadController.java:73) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_74]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_74]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_74]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_74]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:220) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]

Currently I have 8GB RAM installed on machine and 1.5GB of it free. I also tried running spring boot app when there was 2+ GB free RAM.

During debugging I can see that the size of the uploaded file is indeed 1+GB:

But on executing line 73, file.getBytes(), it throws above exception.

What I am missing here?

Also streams and chunking the file on the client side would help you resolve this. — akortex, Aug 17 '18 at 10:44
any link about discussing exactly what you are trying to say? — Mahesha999, Aug 17 '18 at 10:46
Have a look here: https://stackoverflow.com/questions/10326460/how-to-avoid-outofmemoryerror-when-uploading-a-large-file-using-jersey-client — akortex, Aug 17 '18 at 10:47
"Also streams and chunking the file on the client side" means what? The client is browser right? — Mahesha999, Aug 17 '18 at 10:50
Plz remember there is a lot going on in your heap space. You could use visual VM and check out how it already starts... — Desorder, Aug 17 '18 at 10:51

score 4 · Answer 1 · edited Apr 15 '19 at 05:40

for large file it would be better to read bytes of 1024 to a buffer using while loop then write the buffer to the server.This code has being tested with a 5GB file upload.You need to adjust apllication proprties accordingly i.e atleast 5GB if you wanna try the test

spring.http.multipart.max-file-size=6000MB
spring.http.multipart.max-request-size=6000MB

Here is the code snippet written using spring framework.

@Override
public ResponseEntity<?> uploadFile(@RequestParam("file") MultipartFile multipartfile) {
    ResponseEntity<String> response;
    multipartfile.getOriginalFilename();
    byte [] bufferedbytes= new byte[1024];
    File file= new File("/home/david/Music/"+multipartfile.getOriginalFilename());
    FileOutputStream outStream = null;
    int count=0;
    try {
        BufferedInputStream fileInputStream= new BufferedInputStream(multipartfile.getInputStream());
        outStream=new FileOutputStream(file);
        while((count=fileInputStream.read(bufferedbytes))!=-1) {
         outStream.write(bufferedbytes,0,count);

        }
         outStream.close();
    } catch (IOException e) {
         response= new ResponseEntity<String>("File failed to upload"+multipartfile.getOriginalFilename(),HttpStatus.PAYLOAD_TOO_LARGE);

        return response;

    }

     response= new ResponseEntity<String>("File uploaded sucessifully"+multipartfile.getOriginalFilename(),HttpStatus.OK);

    return response;
}

score 2 · Accepted Answer · answered Aug 17 '18 at 10:50

When handling big files it is never a good idea to load an entire file into the memory. Instead just read a little and write that on your OutputStream. Here is a little example that can upload your file while only ever using 4KB of memory at a time.

File source = new File("mySourceFile.txt");
File target = new File("myTargetFile.txt");
int readByteCount = 0;
byte[] buffer = new byte[4096];

try(FileInputStream in = new FileInputStream(source);
    FileOutputStream out = new FileOutputStream(target)) {

    while((readByteCount = in.read(buffer)) != -1) {

        out.write(buffer, 0, readByteCount);
    }
}

"java.lang.OutOfMemoryError: Java heap space" while uploading 1+ GB file in spring boot application

2 Answers2