Export an array from NodeJS directly to amazon S3 without first saving it locally?

Question

This is the working flow of my application:

Get the data from sources.
Perform some operations.
Save the data on a csv file created locally.
After the completion, upload the csv to S3 and delete the local file.

However, when I tried to deploy this application in heroku, I get an error in writeFileSync and cannot create files. To tackle that, I want to store the csv values in a nested array, and then upload directly to S3 as a csv file without local interference.

How do I achieve this? If there are better approaches, please do mention.

How are you currently creating your CSV file locally? Please show us the code. — ChrisGPT was on strike, Feb 27 '22 at 13:21
@Chris I am just using fs.createWriteStream(loc) for an stream and writing data to the stream. — crosie, Feb 27 '22 at 14:12

score 1 · Accepted Answer · answered Feb 27 '22 at 15:49

1

Yes, you can write strings directly to S3 using the AWS SDK's putObject method.

The Body parameter can be one of:

Buffer
string value
stream

answered Feb 27 '22 at 15:49

jarmod

71,565
16
115
122

Thanks @jarmod. Can you also help me to for setting up writeStreams directly to S3? Any tutorials or stackoverflow answers? – crosie Feb 28 '22 at 03:39
1

There are a couple of streaming example [here](https://stackoverflow.com/questions/37336050/pipe-a-stream-to-s3-upload). – jarmod Feb 28 '22 at 16:10

Export an array from NodeJS directly to amazon S3 without first saving it locally?

1 Answers1