0

I am using Jersey as my REST implementation and JAXB to represent my classes. I have done a tiny file transfer mechanism. At the client side the file is converted to a BASE64 string before I set it to the JAXB object. The JAXB object is then sent to the server with Jersey.

It works with files not larger than 50 MB or so, but when I try to send a 500 MB file I get OutOfMemoryError on my client. I have set the -Xms and -Xmx to 2048m but it does not help, I still get the error.

What can I do to get it work with very large files?

Rox
  • 2,647
  • 15
  • 50
  • 85

2 Answers2

1

Rewrite the entire thing to stream the file. Don't base64 encoded it, don't use JAXB, don't embed it in XML. Make it work even if the file is 900 TB. Implement this outside of Jersey if you have to. HTTP GET results in HTTP response with file bytes streamed in body. You can find example servlets that implement this in many intro to servlet tutorials, etc.

Keith
  • 4,144
  • 1
  • 19
  • 14
  • I will try using a `HttpUrlConnection` to upload/download a file. I guess it Content-Type should be `application/octet-stream` in the HTTP header? – Rox Jun 26 '13 at 18:23
  • On the client side you *may* still be able to use the same API. Some WS client APIs have a mechanism to get the underlying request, or a getInputStream() method. From there the client can just read the stream. Yep, that is the right content-type to set on the server side. If you want to make it nice to test with a browser, consider also setting the content disposition header: http://stackoverflow.com/questions/1012437/uses-of-content-disposition-in-an-http-response-header That way a web browser will prompt you where to save the file, instead of trying to display it. – Keith Jun 26 '13 at 18:27
  • I forgot to mention that I am using a rich application and not a web form to send the file. I will try `HttpUrlConnection` tomorrow and return back here if I fail. :-) – Rox Jun 26 '13 at 18:42
  • When you say "same API", are you referring to `HttpUrlConnection`? – Rox Jun 26 '13 at 18:43
  • No, whatever you were using to retrieve the XML. Maybe that *was* HttpUrlConnection. Anyway, as an asside, apache's HTTP API is much nicer than that in the JDK (HttpUrlConnection). See this: http://hc.apache.org/httpcomponents-client-ga/index.html – Keith Jun 26 '13 at 19:02
  • I was using Jersey for both sending / receiving file contents. I will check out Apache´s HTTP api. – Rox Jun 26 '13 at 19:14
1

Webservices aren't designed to transport large amounts of data; they are designed for inter-process communication. File transfers you do using specific protocols like FTP. If I'd have to design something like this I'd do it in two steps.

  1. upload file(s) using FTP (apache commons.net for FTP or JSCH for SFTP) List item
  2. invoke webservice call to tell that the files are there

However you can try the this example to upload the large files using Jersey Client.

Community
  • 1
  • 1
Velu
  • 1,681
  • 1
  • 22
  • 26