Download AWS S3 file directly to a directory

Question

I want to automatically save the file im downloading from AWS S3 to my application folder. Right now this can be done manually. Code below:

    router.get("/download", (req, res) => {
    //File S3 URL
    var fileKey =
        "key";

    AWS.config.update({
        accessKeyId: IAM_USER_KEY,
        secretAccessKey: IAM_USER_SECRET,
        Bucket: BUCKET_NAME
    });

    var s3 = new AWS.S3();
    var file = fs.createWriteStream("test.csv");

    var options = {
        Bucket: "name",
        Key: fileKey
    };

    res.attachment(fileKey);

    var fileStream = s3
        .getObject(options)
        .createReadStream()
        .on("error", error => {
        console.log(error);
        res.json({ error: "An error has occured, check console." });
        })
        .on("httpData", function(data) {

        file.write(data);
        })
        .on("httpDone", function() {
        file.end();
        });

    fileStream.pipe(res);
    //   fse.writeFileSync("text.csv");
    });

As mentioned before, the file can be download and saved manually. But how can write the file and save it automatically in an specific folder?

Thank you

What's wrong with fs.createWriteStream("/somefolder/test.csv")? — jarmod, Aug 13 '18 at 15:48
the output data is: [object Object], and the original file is a csv with over 70k rows. (98mb) — Blinhawk, Aug 13 '18 at 15:49
Your httpData handler is wrong. You've declared 'file' in two scopes. — jarmod, Aug 13 '18 at 16:01
Fixed that. Sadly, its not working. It's creating a 0KB File. — Blinhawk, Aug 13 '18 at 16:29
Please review the streaming example at https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/requests-using-stream-objects.html — jarmod, Aug 13 '18 at 16:37
I wrote up an answer so please accept if it’s good so people know your question was resolved. — jarmod, Aug 15 '18 at 12:29

score 15 · Accepted Answer · answered Aug 13 '18 at 16:57

15

Here's an example of downloading an S3 object to a specific local file:

const AWS = require('aws-sdk');
var s3 = new AWS.S3({apiVersion: '2006-03-01'});
var params = {Bucket: 'mybucket', Key: 'test.csv'};
var file = require('fs').createWriteStream('/tmp/test.csv');
s3.getObject(params).createReadStream().pipe(file);

answered Aug 13 '18 at 16:57

jarmod

71,565
16
115
122

I wonder how do you do this in async? I need to download 3 files and wait until it is done to proceed – RicardoE Jul 24 '19 at 05:58
@RicardoE only just saw your follow-up questions, apologies. One option might be to promisify the end of each stream and then await Promise.all(...) See https://stackoverflow.com/questions/33599688/how-to-use-es8-async-await-with-streams – jarmod Aug 19 '19 at 16:47

score 6 · Answer 2 · answered Mar 04 '21 at 23:50

The code for jarmod's answer, promisified:

const AWS = require('aws-sdk');
const s3 = new AWS.S3({apiVersion: '2006-03-01'});
const params = {Bucket: 'mybucket', Key: 'test.csv'};
const file = require('fs').createWriteStream('/tmp/test.csv');

new Promise((resolve, reject) => {
  const pipe = s3.getObject(params).createReadStream().pipe(file);
  pipe.on('error', reject);
  pipe.on('close', resolve);
});

Download AWS S3 file directly to a directory

2 Answers2