How to avoid OutOfMemoryError when uploading a large file using Jersey client

Question

I am using Jersey client for http-based request. It works well if the file is small but run into error when I post a file with size of 700M:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2786)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
    at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
    at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:90)
    at com.sun.jersey.core.util.ReaderWriter.writeTo(ReaderWriter.java:115)
    at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:76)
    at com.sun.jersey.core.impl.provider.entity.FileProvider.writeTo(FileProvider.java:103)
    at com.sun.jersey.core.impl.provider.entity.FileProvider.writeTo(FileProvider.java:64)
    at com.sun.jersey.multipart.impl.MultiPartWriter.writeTo(MultiPartWriter.java:224)
    at com.sun.jersey.multipart.impl.MultiPartWriter.writeTo(MultiPartWriter.java:71)
    at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568)
    at TestHttpRequest.main(TestHttpRequest.java:42)

here is my code:

ClientConfig cc = new DefaultClientConfig();
        Client client = Client.create(cc);
        WebResource resource = client.resource("http://localhost:8080/JerseyWithServletTest/helloworld");
        FormDataMultiPart form = new FormDataMultiPart();
        File file = new File("E:/CN_WXPPSP3_v312.ISO");
        form.field("username", "ljy");
        form.field("password", "password");
        form.field("filename", file.getName());
        form.bodyPart(new FileDataBodyPart("file", file, MediaType.MULTIPART_FORM_DATA_TYPE));
        ClientResponse response = resource.type(MediaType.MULTIPART_FORM_DATA).post(ClientResponse.class, form);

So, no matter what server used, there is no request reached up to server. — Mr rain, May 02 '12 at 01:52
possible duplicate of [Large file transfer from HTTP server running Java Jersey Rest API](http://stackoverflow.com/questions/14864939/large-file-transfer-from-http-server-running-java-jersey-rest-api) — Raedwald, May 01 '15 at 22:06

score 36 · Accepted Answer · answered Apr 27 '12 at 12:43

36

You could use streams.Try something like this on the client:

InputStream fileInStream = new FileInputStream(fileName);
String sContentDisposition = "attachment; filename=\"" + fileName.getName()+"\"";
WebResource fileResource = a_client.resource(a_sUrl);       
ClientResponse response = fileResource.type(MediaType.APPLICATION_OCTET_STREAM)
                        .header("Content-Disposition", sContentDisposition)
                        .post(ClientResponse.class, fileInStream);

with resource like this on the server:

@PUT
@Consumes("application/octet-stream")
public Response putFile(@Context HttpServletRequest a_request,
                         @PathParam("fileId") long a_fileId,
                         InputStream a_fileInputStream) throws Throwable
{
    // Do something with a_fileInputStream
    // etc

answered Apr 27 '12 at 12:43

Martin Wilson

3,386
1
24
29

But, I want to post some parameters such as file's name. – Mr rain Apr 28 '12 at 08:39
My example shows how to include the filename in the header. You can get the identifier for the file resource from the URL- shown as fileId in my example. – Martin Wilson Apr 28 '12 at 09:33
7

Why is this not in the examples for jersey?? Such an obvious use case! – PiersyP Aug 14 '14 at 11:55
What if I want to POST form with several text fields and large file? It was feasible using plain Servlet and [FilePart](http://stackoverflow.com/a/2424824/660408). But I get HeapSpace error in jersey. – gkiko Jun 29 '15 at 12:08

sikrip · Answer 2 · 2013-05-30T13:43:06.940

In order for your code not to depend on the size of the uploaded file, you need:

Use streams
Define the chuck size of the jersey client. For example: client.setChunkedEncodingSize(1024);

Server:

    @POST
    @Path("/upload/{attachmentName}")
    @Consumes(MediaType.APPLICATION_OCTET_STREAM)
    public void uploadAttachment(@PathParam("attachmentName") String attachmentName, InputStream attachmentInputStream) {
        // do something with the input stream
    }

Client:

    ...
    client.setChunkedEncodingSize(1024);
    WebResource rootResource = client.resource("your-server-base-url");
    File file = new File("your-file-path");
    InputStream fileInStream = new FileInputStream(file);
    String contentDisposition = "attachment; filename=\"" + file.getName() + "\"";
    ClientResponse response = rootResource.path("attachment").path("upload").path("your-file-name")
            .type(MediaType.APPLICATION_OCTET_STREAM).header("Content-Disposition", contentDisposition)
            .post(ClientResponse.class, fileInStream);

score 5 · Answer 3 · edited Oct 07 '16 at 13:41

Below is the code for uploading a (potentially large) file with chunked transfer encoding (i.e. streams) using Jersey 2.11.

Maven:

<properties>
    <jersey.version>2.11</jersey.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-client</artifactId>
        <version>${jersey.version}</version>
    </dependency>
    <dependency>
    <groupId>org.glassfish.jersey.media</groupId>
        <artifactId>jersey-media-multipart</artifactId>
        <version>${jersey.version}</version>
    </dependency>
<dependencies>

Java:

Client client = ClientBuilder.newClient(clientConfig);
client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED");

WebTarget target = client.target(SERVICE_URI); 
InputStream fileInStream = new FileInputStream(inFile);
String contentDisposition = "attachment; filename=\"" + inFile.getName() + "\"";
System.out.println("sending: " + inFile.length() + " bytes...");
Response response = target
            .request(MediaType.APPLICATION_OCTET_STREAM_TYPE)
            .header("Content-Disposition", contentDisposition)
            .header("Content-Length", (int) inFile.length())
            .put(Entity.entity(fileInStream, MediaType.APPLICATION_OCTET_STREAM_TYPE));
System.out.println("Response status: " + response.getStatus());

In Jersey 1 just setting (using the appropriate jersey 1 APIs of course) client.property(ClientProperties.CHUNKED_ENCODING_SIZE, chunkSize1mb); seemed to do the trick. However it seems that it no longer works for Jersey 2. I have set In addition the property suggested in this answer (client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED"); ) and it worked! So thanks a bunch for that! Do you have any idea if this could break something else? — Svilen, May 30 '16 at 10:44
In my case this solution working properly only adding `System.setProperty("sun.net.http.allowRestrictedHeaders", "true")`. I added an answer with an alternative solution using `StreamingOutput`. Anyway, setting the `CHUNKED` client property makes things working, so thank you! — xonya, Oct 07 '16 at 10:18

score 1 · Answer 4 · edited May 23 '17 at 12:02

In my case (Jersey 2.23.2) rschmidt13's solution gave this warning:

WARNING: Attempt to send restricted header(s) while the [sun.net.http.allowRestrictedHeaders] system property not set. Header(s) will possibly be ignored.

This can be solved adding the following line:

System.setProperty("sun.net.http.allowRestrictedHeaders", "true");

However I think a cleaner solution can be obtained using the StreamingOutput interface. I post a complete example hoping it could be useful.

Client (File upload)

WebTarget target = ClientBuilder.newBuilder().build()
            .property(ClientProperties.CHUNKED_ENCODING_SIZE, 1024)
            .property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED")
            .target("<your-url>");

StreamingOutput out = new StreamingOutput() {

    @Override
    public void write(OutputStream output) throws IOException, 
            WebApplicationException {

        try (FileInputStream is = new FileInputStream(file)) {

            int available;
            while ((available = is.available()) > 0) {
                // or use a buffer
                output.write(is.read());
            }
        }
    }
};

Response response = target.request().post(Entity.text(out));

Server

@Path("resourcename")
public class MyResource {

    @Context
    HttpServletRequest request;

    @POST
    @Path("thepath")
    public Response upload() throws IOException, ServletException {

        try (InputStream is = request.getInputStream()) {
            // ...
        }
    }
}

JScoobyCed · Answer 5 · 2012-04-26T02:46:00.133

0

If possible, can you split the file you send into smaller parts? This will reduce memory usage, but you need to change the code on both sides of the uploading/downloading code.

If you can't, then your heap space is too low, try increasing it with this JVM parameter. In your application server add/change the Xmx JVM options. For example

-Xmx1024m

to set Max Heap Space to 1Gb

edited Apr 26 '12 at 02:46

answered Apr 26 '12 at 02:35

JScoobyCed

10,203
6
34
58

Thank you for answering, what my project do is to provide service to external system. – Mr rain Apr 26 '12 at 02:41
I see, I want to know if we can flush the memory during the file upload. And how to do it!! – Mr rain Apr 26 '12 at 02:44
By splitting the file into smaller parts you will have a better control, and predictable memory usage per file. This need changes in your server side and client side – JScoobyCed Apr 26 '12 at 02:47
yeah, that's a very good idea, thank you for your advise, JScoobyCed. – Mr rain Apr 26 '12 at 02:49
I had split File into smaller parts with size of 10M and upload all the parts within a request. But it still doesn't work, I am going to crazy !!! – Mr rain Apr 26 '12 at 05:56
If all the file parts are in the same request then sun.net.www.http.PosterOutputStream is going to buffer them all (see http://stackoverflow.com/questions/2082057/java-outputstream-java-lang-outofmemoryerror-java-heap-space) hence you will still get the OOM error – Martin Wilson May 02 '12 at 13:57
BTW, the suggestion above to up the heap space available would need to be done on your client to have any effect on the error you are seeing. Although, if this post is correct (https://issues.alfresco.com/jira/browse/ETHREEOH-974) the underlying sun JDK impl of the HTTP POST connection stream is flawed and so you'll need a lot of heap for a 700MB file. – Martin Wilson May 02 '12 at 14:01

score 0 · Answer 6 · edited Apr 02 '16 at 22:17

@Consumes("multipart/form-data")
@Produces(MediaType.TEXT_PLAIN + ";charset=utf-8")
public String upload(MultipartFormDataInput input,  @QueryParam("videoId") String  videoId,
        @Context HttpServletRequest a_request) {

    String fileName = "";
    for (InputPart inputPart : input.getParts()) {
        try {

            MultivaluedMap<String, String> header = inputPart.getHeaders();
            fileName = getFileName(header);
            // convert the uploaded file to inputstream
            InputStream inputStream = inputPart.getBody(InputStream.class, null);               
            // write the inputStream to a FileOutputStream
            OutputStream outputStream = new FileOutputStream(new File("/home/mh/Téléchargements/videoUpload.avi"));
            int read = 0;
            byte[] bytes = new byte[1024];
            while ((read = inputStream.read(bytes)) != -1) {
                outputStream.write(bytes, 0, read);
            }
            System.out.println("Done!");
        } catch (IOException e) {
            e.printStackTrace();
            return "ko";
        }

    }

How to avoid OutOfMemoryError when uploading a large file using Jersey client

6 Answers6

Client (File upload)

Server

Linked