Download file from Amazon S3 using REST API

Question

I have my own REST API to call in order to download a file. (At the end, the file could be store in different kind of server... Amazon s3, locally etc...)

To get a file from s3, I should use this method:

var url = s3.getSignedUrl('getObject', params);

This will give me a downloadable link to call.

Now, my question is, how can I use my own rest API to download a file when it comes from that link? Is there a way to redirect the call?

I'm using Hapi for my REST server.

{
    method: "GET", path: "/downloadFile",
    config: {auth: false},
    handler: function (request, reply) {
        // TODO
        reply({})
    }
},

Why do you need the link to then call that link to download the file, why not just download the file and return the stream? — peteb, Feb 01 '17 at 16:34
@peteb Wouldn't that make the file be downloaded 2 times for one request? Client will have to wait until the file is downloaded from the server, then stream to him? — Jaythaking, Feb 01 '17 at 16:40
not if you return the stream back to the requester. Think about it like this, you can get a stream from S3 with the contents of that file, you can then use `pipe()` to pipe the S3 stream directly back to the requester. It is never downloaded to the mediator, instead the mediator is just a go between. Especially if you use the AWS-SDK unbufferedStream. — peteb, Feb 01 '17 at 16:42
How do I setup the Hapi request for that? Do you have an example for me of a pipe request? Thx — Jaythaking, Feb 01 '17 at 16:47
I don't find any method call for an unbuffered stream in s3 javascript — Jaythaking, Feb 01 '17 at 16:52

peteb · Accepted Answer · 2017-02-01T19:09:06.170

Instead of using a redirect to download the desired file, just return back an unbufferedStream instead from S3. An unbufferedStream can be returned from the HttpResponse within the AWS-SDK. This means there is no need to download the file from S3, then read it in, and then have the requester download the file.

FYI I use this getObject() approach with Express and have never used Hapi, however I think that I'm pretty close with the route definition but hopefully it will capture the essence of what I'm trying to achieve.

Hapi.js route

const getObject = require('./getObject');

{
    method: "GET", path: "/downloadFile",
    config: {auth: false},
    handler: function (request, reply) {
        let key = ''; // get key from request
        let bucket = ''; // get bucket from request

        return getObject(bucket, key)
          .then((response) => {
            reply.statusCode(response.statusCode);

            response.headers.forEach((header) => {
              reply.header(header, response.headers[header]);
            });

            return reply(response.readStream);
          })
          .catch((err) => {
            // handle err
            reply.statusCode(500);
            return reply('error');
          });
    }
},

getObject.js

const AWS = require('aws-sdk');
const S3 = new AWS.S3(<your-S3-config>);

module.exports = function getObject(bucket, key) {
  return new Promise((resolve, reject) => {
    // Get the file from the bucket
    S3.getObject({
      Bucket: bucket,
      Key: key
    })
      .on('error', (err) => {            
        return reject(err);
      })
      .on('httpHeaders', (statusCode, headers, response) => {
        // If the Key was found inside Bucket, prepare a response object
        if (statusCode === 200) {
          let responseObject = {
            statusCode: statusCode,
            headers: {
              'Content-Disposition': 'attachment; filename=' + key
            }
          };

          if (headers['content-type'])
            responseObject.headers['Content-Type'] = headers['content-type'];
          if (headers['content-length'])
            responseObject.headers['Content-Length'] = headers['content-length'];

          responseObject.readStream = response.httpResponse.createUnbufferedStream();
          return resolve(responseObject);
        }
      })
      .send();
  });
}

Thanks a lot, I'm trying it right now and I'll let you know :) — Jaythaking, Feb 01 '17 at 17:06
Does this result in the S3 bucket content being streamed "through" his server, or does the requestor end up streaming it direct from S3? — Mike Goodwin, Feb 06 '23 at 15:01
@MikeGoodwin The content being requested is streamed through the local express server before going to the client, due to the direct `getObject()` usage. If that approach isn't ideal, using a signed URL to sign `getObject()` and returning the URL for the action would be the way to go. In that case, the client would request the Express instance to create an authenticated `getObject()` call but not execute it. That would then be represented as a URL that can be returned to the client for a direct to AWS request. This approach has security and cost effects that you should asses before proceeding. — peteb, Feb 06 '23 at 18:08

score 2 · Answer 2 · answered Feb 01 '17 at 16:32

2

Return a HTTP 303 Redirect with the Location header set to the blob's public URL in the S3 bucket.

If your bucket is private then you need to proxy the request instead of performing a redirect, unless your clients also have access to the bucket.

answered Feb 01 '17 at 16:32

Dai

141,631
28
261
374

How do I proxy a request? – Jaythaking Feb 01 '17 at 16:41
There should be no need to proxy the request. `HTTP/1.1 302 Found` `Location: https://${signed_url}` – Michael - sqlbot Feb 01 '17 at 21:52
@Michael-sqlbot If the S3 bucket is private and/or the OP doesn't want to disclose blob URIs to end-users/consumers. – Dai Feb 01 '17 at 22:13
This still works for that. If the bucket is private then a pre-signed URL generated right then and used for the redirect, with a short expiration time, can allow a single user to download the file right now and not later. Mine are set to 5 seconds. They can know the blob URIs all they like, but 5 seconds from now, it won't do them any good. :) The expiration time of the pre-signed URL is only checked when the request arrives at S3, so you don't have to finish downloading before it expires, you only have to start downloading before it expires. – Michael - sqlbot Feb 01 '17 at 22:29

Download file from Amazon S3 using REST API

2 Answers2

Hapi.js route

getObject.js

Linked