1

I am trying to write chunks of CSV formatted data to a file in Amazon S3 instead of writing to a temporary file through the WriteStream then creating a ReadStream on that file and sending it to S3. My program pulls rows of data from a database, processed it then formats each row in a CSV like so using S3's upload() api

let recordsCSVFormatted;
let offset = 0;
const batchSize = 500;
const writer = fs.createWriteStream('./someFile.csv')

do {
  recordsCSVFormatted = await getRecords(limit, offset); // gets records from DB, formats it in CSV string
  writer.write(recordsCSVFormatted);
  offset += batchSize;
} while (typeof recordsCSVFormatted === 'undefined' || (recordsCSVFormatted && recordsCSVFormatted.length))

const reader = fs.createReadStream('./someFile.csv');

// just assume here that Key and Bucket are provided in upload, they are in actual code
await new AWS.S3({...s3Opts}).upload({Body: reader}).promise() // pass the readable in here for AWS

How can I skip the step of creating a temporary file and then passing the file to AWS as a stream? I want to be able to stream the chunks of CSV information directly.

jman93
  • 367
  • 3
  • 9
  • 1
    Create a readable stream and push your CSV records there. Use the S3 client upload() option where the Body is of type ReadableStream. – jarmod Sep 26 '19 at 21:01
  • Each iteration is a chunk of just 1 Csv file though. After all the iterations through the loop are done the write stream closes resulting in 1 CSV file. I tried doing what you said already by creating a readable and pushing chunks to it but maybe I didn't do it right. Can you provide some example of this? – jman93 Sep 26 '19 at 21:29
  • Possible duplicate of [Can you upload to S3 using a stream rather than a local file?](https://stackoverflow.com/questions/31031463/can-you-upload-to-s3-using-a-stream-rather-than-a-local-file) – qkhanhpro Sep 27 '19 at 03:56

1 Answers1

0

Solved this by implementing the Readable class and implementing a custom read() function for S3 upload to consume

jman93
  • 367
  • 3
  • 9