0

I need to upload a large file to the server, preferably in streaming mode, so that it does not load completely into RAM. Since now, when trying to upload a large file, an exception is thrown

/manager/api/server/import/upload java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) at org.eclipse.jetty.util.MultiPartInputStreamParser$MultiPart.write(MultiPartInputStreamParser.java:173) at org.eclipse.jetty.util.MultiPartInputStreamParser.parse(MultiPartInputStreamParser.java:797) at org.eclipse.jetty.util.MultiPartInputStreamParser.getParts(MultiPartInputStreamParser.java:482) at org.eclipse.jetty.server.MultiParts$MultiPartsUtilParser.getParts(MultiParts.java:112) at org.eclipse.jetty.server.Request.getParts(Request.java:2326) at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:529) at org.eclipse.jetty.server.Request.getParameters(Request.java:430) at org.eclipse.jetty.server.Request.getParameter(Request.java:1059) at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:194) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:167) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:536) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1592) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

On the frontend for file uploading I use the react and antd

            <Upload.Dragger style={{paddingLeft: '10px', paddingRight: '10px'}} action="api/server/import/upload" name={name} onChange={this.changeFileState} showUploadList={false} accept=".zip">
            <p className="ant-upload-drag-icon">
                <Icon type={icon}/>
            </p>
            <p className="ant-upload-text">{label}</p>

            {this.fileName ? (
                <div>
                    <p className="ant-upload-hint">{this.fileName}</p>
                    <Progress size="small" status={this.status} percent={this.progress}/>
                </div>
            ) : (
                <p className="ant-upload-hint">{hint}</p>
            )}
        </Upload.Dragger>

The request is received by the next controller

    @PostMapping("import/upload")
public void importUpload(
        @RequestParam("file")
        @ApiParam("Файл с данными для импорта") MultipartFile file) throws Exception {
    try (OutputStream outputStream = new FileOutputStream(zipArchive.getFile());
         InputStream inputStream = file.getInputStream();) {
        byte[] buffer = new byte[8192];
        int bytesRead;
        while ((bytesRead = inputStream.read(buffer)) > 0) {
            outputStream.write(buffer, 0, bytesRead);
        }
    } catch (Exception ex) {
        throw new RuntimeException("Error" + ex.getMessage(), ex);
    }

An exception occurs at the stage of mapping parameters. What can I do to download large files (5 Gb) without heavy memory load?

Lukyanov Mikhail
  • 500
  • 7
  • 17
  • The Jetty unit tests upload 10GB files via multipart, using the standard multipart configurations within the Servlet spec. Something else is going on. What is your standard servlet MultipartConfig set to? – Joakim Erdfelt Nov 20 '19 at 12:28
  • 2
    Thank you, I added spring.http.multipart.file-size-threshold=200MB and the memory stopped loading using the default org.springframework.web.multipart.MultipartFile – Lukyanov Mikhail Nov 20 '19 at 13:28

0 Answers0