0

I am facing a Out of memory issue while downloading multipart file in Java.

I am using HttpUrlConnection to download file getting issue in fist line:

ByteArrayDataSource ds = new ByteArrayDataSource(con.getInputStream(),
                "multipart/mixed");           //Line 1
MimeMultipart multipart = new MimeMultipart(ds);          //Line 2
        javax.mail.BodyPart jsonPart = multipart.getBodyPart(1); //Line 3
        javax.mail.BodyPart videoPart = multipart.getBodyPart(2);  //Line 4

Multipart response contains 2 different file, one is simple text file and one is video file.

Whole multipart response size is more than 1 GB.

CroMagnon
  • 1,218
  • 7
  • 20
  • 32
pash
  • 47
  • 1
  • 10
  • Which line is throwing the error? You'll probably need to process the stream incrementally instead of trying to read it all at once. – shmosel Feb 09 '17 at 09:12
  • Do you specify the memory allocation `Xmx` value? You may refer to: http://stackoverflow.com/questions/14763079/what-are-the-xms-and-xmx-parameters-when-starting-jvms – Pavan Kumar Feb 09 '17 at 09:23
  • @shmosel ByteArrayDataSource ds = new ByteArrayDataSource(con.getInputStream(), "multipart/mixed"); This line give error. Yes i know but i am not getting proper solution to read stream incrementally – pash Feb 09 '17 at 09:28
  • @PavanKumar yes i have specify Xms memory allocation but still it gives me out of memory error. – pash Feb 09 '17 at 09:29
  • @pash you may please specify `-Xmx2g` which is the higher limit. `Xms` might not cause an issue here. – Pavan Kumar Feb 09 '17 at 09:31
  • @PavanKumar i don't want to increase it because, if file size increases in future (say 4 GB) then again same issue will come. – pash Feb 09 '17 at 09:35
  • Then the issue should be with the design. If the input file size is unknown and we're keeping the whole file in memory, we definitely fail all the times when we cross the environment limits. May be you can try persisting it on file system or so. – Pavan Kumar Feb 09 '17 at 09:39
  • @PavanKumar Yes you are correct. But that response file contains 2 different file. One is text and another is Video. I want to extract that video file from the response. Is there any idea how to parse the multipart response? – pash Feb 09 '17 at 09:44
  • May be you can try something like `multipart.getBodyPart(0).getInputStream()` and use the stream to write the file etc? Note: You're using 1 and 2 in line 3 and 4, it should be 0 and 1. – Pavan Kumar Feb 09 '17 at 10:14

1 Answers1

0

You should not load the whole input stream into a byte array unless you make sure it is small. You can do so by temporarily saving the file to the local file system. For large files, you need to find an alternate solution. Either not allow it, or stream it to a destination that supports streaming. i.e. JMS queue.

vstrom coder
  • 297
  • 1
  • 8