How to upload large files to S3 using presigned URL?

Question

I have been able to upload successully small files ~1kb but when I try uploading larger files > 1Mb I get this exception:

java.io.IOException: Error writing to server
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:699)
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:711)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1585)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
    at com.red5.VimeoTechRojoIntegration.uploadFromEFSToS3(VimeoTechRojoIntegration.java:209)
    at com.red5.Red5ProLive$1.run(Red5ProLive.java:128)
    at java.lang.Thread.run(Thread.java:748)

To upload I am suing this code

OutputStream out = null;
InputStream in = null;
HttpURLConnection connection = null;
try {
    URL url = new URL(signedURLS3);
    in = new FileInputStream(videoEFSPath);

    System.out.println("establish connection");
    connection=(HttpURLConnection) url.openConnection();
    connection.setDoOutput(true);
    connection.setRequestMethod("PUT");
    connection.setRequestProperty("Content-Type", "application/octet-stream"); 


    System.out.println("get output stream");
    out = (OutputStream) connection.getOutputStream();
    System.out.println("copy");
    IOUtils.copyLarge(in,out);
    System.out.println("copy finished");

    int responseCode = connection.getResponseCode();
    if (responseCode < 300)
        return "";

    return "{\"error\":\"The upload to S3 failed. AWS server returned response code "+responseCode+"\"}";   
}

What is the problem?

How about the AWS SDK? https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/examples-s3-objects.html#upload-object — , Apr 19 '18 at 19:06
@mrblewog using presigned urls is a way how to decouple the application from the direct use of AWS, or provide the upload to a third-party without a direct AWS access (AWS credentials). The app does only a simple PUT to the url. This works with S3 or anything else without the need to adapt the existing code. So, using AWS SDK is not the right solution here. — ttulka, Apr 20 '18 at 06:28

score 0 · Answer 1 · answered Apr 20 '18 at 06:34

This should actually work, I am using Apache HTTP Client for this:

import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.util.EntityUtils;
import org.apache.http.entity.FileEntity;

...

String presignedUrl = ...
File file = ...

HttpPut httpPut = new HttpPut(presignedUrl);
httpPut.setEntity(new FileEntity(file));
HttpResponse httpResponsePut = httpClient.execute(httpPut);
if (httpResponsePut.getStatusLine().getStatusCode() >= HttpStatus.SC_BAD_REQUEST) {
    log.error("Error uploading file: " + file.getName() + " / " + httpResponsePut);
}
EntityUtils.consume(httpResponsePut.getEntity());

How to upload large files to S3 using presigned URL?

1 Answers1