File Download from Amazon S3 via REST (Jax-RS)

Question

I am trying to download a file from Amazon S3.

I want the user to visit my app via a GET api.

The app in turn gets the content from S3 and give it back to the user as a downloadable file.

Note:

I dont want to store the file locally in my server, i want it to be streamed form amazon s3 directly to the end user

I tried with a file of around 300 MB, if I run it locally like below the memory footprint is low, i.e. when the same file is present locally

   @GET
    @Path("/pdfdownload")
    @Produces("application/pdf")
    public Response getFile() {
    File file = new File('/pathToFile'); // in local

    ResponseBuilder response = Response.ok((Object) file);
    response.header("Content-Disposition", "attachment; filename=file.pdf");
    return response.build();
}

But when I download the same from Amazon s3, my tomcat server's memory quickly raises to around 600 MB, I think I am streaming the content, but when i look at the memory used i doubt it Am i missing something ?

@GET
@Path("/pdfdownload")
@Produces("application/pdf")
public Response getFile2() {
    final S3Object s3Object = getAmazonS3Object();// AWS S3
    final S3ObjectInputStream s3is = s3Object.getObjectContent();

final StreamingOutput stream = new StreamingOutput() {
    @Override
    public void write(OutputStream os) throws IOException, WebApplicationException {
        byte[] read_buf = new byte[1024];
        int read_len = 0;
        while ((read_len = s3is.read(read_buf)) > 0) {
            os.write(read_buf, 0, read_len);
        }
        os.close();
        s3is.close();
    }
};

ResponseBuilder response = Response.ok(stream);
response.header("Content-Disposition", "attachment; filename=file.pdf");
return response.build();
}

private S3Object getAmazonS3Object() {
    AWSCredentials credentials = new BasicAWSCredentials("accesskey",
            "secretkey");
    try {
        AmazonS3 s3 = new AmazonS3Client(credentials);
        S3Object s3object = s3.getObject(new GetObjectRequest("bucketName", "filename_WithExtension"));
        return s3object;
    } catch (AmazonServiceException e) {
        System.err.println(e.getErrorMessage());
        System.exit(1);
    }
    System.out.println("Done!");
    return null;
}

Pom :

<dependency>
            <groupId>com.sun.jersey</groupId>
            <artifactId>jersey-server</artifactId>
            <version>1.8</version>
        </dependency>


<dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
            <version>1.11.542</version>
        </dependency>

Similar to this S3 download pdf - REST API

I dont want to use PreSignedURl: https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURLJavaSDK.html

Please see this article on streaming: https://memorynotfound.com/low-level-streaming-with-jax-rs-streamingoutput/

Could some please help as why the memory spikes up?

score 0 · Answer 1 · answered May 15 '19 at 21:52

Thanks to all the post on stackoverflow and one of my colleague. My colleague found the answer, actually the above code doesnt have a memory issue, when I was monitoring the jvm i saw a spike, but didnt realize garbage collection didnt kick in.

I tried downloading 6 files each fo 300 MB +, the server holds its ground

File Download from Amazon S3 via REST (Jax-RS)

1 Answers1