MultipartFile issue, unable to convert to File

Question

I am trying to to upload above 1 GB file, I am using Spring Boot.

I've tried with below code, but I am getting as Out of Memory error.

public void uploadFile(MultipartFile file) throws IOException {
        try {       
            HttpHeaders headers = new HttpHeaders();
            headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);

            SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
            requestFactory.setBufferRequestBody(false);
            restTemplate.setRequestFactory(requestFactory);

            String uploadFile= restTemplate.exchange(url, HttpMethod.POST,
                new HttpEntity<>(new FileSystemResource(convert(file)), headers), String.class).getBody();

        } catch (Exception e) {
            throw new RuntimeException("Exception Occured", e);
        }
    }


    private static File convert(MultipartFile file) throws IOException {
        File convFile = new File(file.getOriginalFilename());
        convFile.createNewFile();
        FileOutputStream fos = new FileOutputStream(convFile);
        fos.write(file.getBytes());
        fos.close();
        return convFile;    
    }

The Main problem I am facing is, I am unable to convert MultipartFile to java.io.File.

I've even tried replacing FileSystemResource with ByteArrayResource, but still getting OOM error.

I've even tried using below code too:

private static File convert(MultipartFile file) throws IOException {
        CommonsMultipartFile commonsMultipartFile = (CommonsMultipartFile) multipartFile;
        FileItem fileItem = commonsMultipartFile.getFileItem();
        DiskFileItem diskFileItem = (DiskFileItem) fileItem;
        String absPath = diskFileItem.getStoreLocation().getAbsolutePath();
        File file = new File(absPath);  
    }

But I am getting below exception for above snippet:

org.springframework.web.multipart.commons.CommonsMultipartFile cannot be cast to org.springframework.web.multipart.MultipartFile

Could anyone please tell me on how to convert MultipartFile to java.io.File?
And also is there any other approach better than FileSystemResource bcoz I will have to create new file in server everytime before uploading. If file is more than 1GB, another 1 GB new file has to be created on server side, and has to manually delete that file again, which I personally didn't like this approach.

locus2k · Accepted Answer · 2019-11-05T20:17:35.073

4

getBytes() tries to load the whole byte array into memory which is causing your OOM what you need to do is stream the file and write it out.

Try the following:

private static Path convert(MultipartFile file) throws IOException {
  Path newFile = Paths.get(file.getOriginalFilename());
  try(InputStream is = file.getInputStream();
     OutputStream os = Files.newOutputStream(newFile))) {
     byte[] buffer = new byte[4096];
     int read = 0;
     while((read = is.read(buffer)) > 0) {
       os.write(buffer,0,read);
     }
  }
  return newFile;  
}

I changed your method to return a Path instead of File which is part of the java.nio package. The package is preferred over java.io as its been optimized more.

If you do need a File object you can call newFile.toFile()

Since it returns a Path object you can use the java.nio.file.Files class to relocate the file to your preferred directory once it has been written out

private static void relocateFile(Path original, Path newLoc) throws IOException {
  if(Files.isDirectory(newLoc)) {
    newLoc = newLoc.resolve(original.getFileName());
  }
  Files.move(original, newLoc);
}

edited Nov 05 '19 at 20:17

answered Nov 05 '19 at 20:08

locus2k

2,802
1
14
21

2

Note you could also use `Files.copy(is, newFile)` or, if using Java 9+, `is.transferTo(os)`. You do lose control over the buffer size, however. – Slaw Nov 05 '19 at 20:27
Those are good suggestions too. I'm not sure how `Files.copy` works under the hood (haven't looked at the source) but I tend to like controlling the buffer size especially on a machine that may have limited resources – locus2k Nov 05 '19 at 20:39
@locus2k one doubt: why this code still getting OOM, I mean: try(InputStream is = new BufferedInputStream(file.getInputStream()); ByteArrayOutputStream os = new ByteArrayOutputStream()) { byte[] buffer = new byte[4096]; int read = 0; while((read = is.read(buffer)) > 0) { os.write(buffer,0,read); } – john Nov 06 '19 at 04:38
2

ByteArrayOutputStream is still storing the bytes in memory as you're not writing it to file as you read thus the `OOM` – locus2k Nov 06 '19 at 12:09
@locus2k okay, got it. thnq – john Nov 06 '19 at 13:58
@locus2k 2 questions: In your code, can I use BufferedInputStream and write it to file 4kb each time? and currently we have kepted size as 4096, how do you decide this size? why cannot I use more bytesize to read? – john Nov 06 '19 at 14:00
For what you're doing you do not have to wrap it into a `BufferedInputStream` unless you want to take advantage of some of the higher level options. You can increase your buffer size as you see fit. I just use `4096` for the example. You can play with it to see what is optimal for your solution – locus2k Nov 06 '19 at 14:42
@locus2k sorry, I ddin't get , whats the problem wrapping with BufferedInputStream ? – john Nov 06 '19 at 17:46
There is no issue wrapping it around a `BufferedInputStream` but it is unnecessary for the function as it will provide no significant improvements. – locus2k Nov 06 '19 at 18:02
@locus2k Reading from buffer is much better than InputStream, right? I mean in case of InputStream, OS has to involve to read bytes, in case of BufferedInputStream , we read from buffer(ie., memory), right? – john Nov 06 '19 at 18:50
2

No, `InputStream` is just an abstraction. `BufferedInputStream` implements `InputStream` under the hood and then adds some more functionality that you can take advantage of. Both read a section of information and stores it into memory. Take a look at this answer for a better explanation https://stackoverflow.com/questions/2964044/should-i-always-wrap-an-inputstream-as-bufferedinputstream – locus2k Nov 06 '19 at 19:11
@locus2k okay, thanks for the link. one last thing: I've actually tried using different buffer sizes (4096 which is 4KB, 1048576 which is 1MB), but I found that 4Kb took less time than 1MB (I've tested with 1.2GB file), any idea? – john Nov 06 '19 at 20:12
2

@john See https://stackoverflow.com/questions/10143501/java-io-performance-issue and https://stackoverflow.com/questions/236861/how-do-you-determine-the-ideal-buffer-size-when-using-fileinputstream – Slaw Nov 07 '19 at 00:25

MultipartFile issue, unable to convert to File

1 Answers1