How to upload parquet file to s3 bucket in nodejs?

Question

I am using parquetjs library. It is allowing me to create the parquet file, and save it locally, but I want to send it directly to the s3 bucket after creating it. The online examples did not help really. I am putting the code I am using below.

const parquet = require('parquetjs');

const parquetSchema = new parquet.ParquetSchema({
  name: { type: 'UTF8' },
  age: { type: 'INT64' }
});

var writer = await parquet.ParquetWriter.openFile(parquetSchema, 'fruits.parquet');

  rows.forEach(async (entry, i) => {
    // append a few rows to the file
    await writer.appendRow({
      name: entry.name,
      age: entry.age
    });
  });

  await writer.close();

After the close code, it is saving in the folder. I am trying to use the regular s3.putObject function of AWS SDK, but for parquet files it doesn't work. It is uploading an empty parquet file with the name. I tried pulling the local file via fs module, and attaching as body as well, but that doesn't work too.

score 0 · Answer 1 · answered Aug 07 '21 at 22:27

var fileStream = fs.createReadStream("F:/directory/fileName.ext");
var putParams = {
    Bucket: s3bucket,
    Key: s3key,
    Body: fileStream
};
s3.putObject(putParams, function(putErr, putData){
    if(putErr){
        console.error(putErr);
    } else {
        console.log(putData);
    }
});

This line of code finally helped me. Didn't have to convert it into any format.

How to upload parquet file to s3 bucket in nodejs?

1 Answers1