How to parse CSVs from s3 bucket to use in a javascript AWS Lambda function

Question

I'm trying to read in a CSV from an s3 bucket using the csvtojson library in AWS Lambda, but it's not working properly. Locally, my code works. But when I upload it to Lambda, it doesn't return anything. There are no errors in the Lambda console, so I'm having a hard time debugging. My code is below.

const AWS = require('aws-sdk');
const csvtojson = require('csvtojson');

const s3 = new AWS.S3();

const params = {
   Bucket: bucketName,
   Key: pathToFile 
};

const stream = s3.getObject(params).createReadStream();

csvtojson()
   .fromStream(stream)
   .then((json) => {
     console.log('Locally, this returns the CSV as JSON. On Lambda, it does not.');
   });

Does csvtojson not work on Lambda for some reason? Should I be using a different method to parse the CSV? Thanks!

score 2 · Answer 1 · answered Sep 23 '19 at 22:52

2

Your lambda is finishing before the Promise has completed. Replace the last section with this:

const json = await csvtojson().fromStream(stream);
console.log('Locally, this returns the CSV as JSON. On Lambda, it does not.');

answered Sep 23 '19 at 22:52

Joey Kilpatrick

1,394
8
20

score 0 · Answer 2 · answered Nov 30 '20 at 10:00

Use promise to solve this problem.

import * as AWS from 'aws-sdk';
const s3 = new AWS.S3();
const csv = require('@fast-csv/parse');


const params = {
    Bucket: 'bucket name',
    Key: 'uploads/img-links.csv'
};

const csvFile = s3.getObject(params).createReadStream();


let csvParsePromise = new Promise((resolve, reject) => {

    const parser = csv.parseStream(csvFile, { headers: true }).on("data", function (data) {
        parser.pause();  // can pause reading using this at a particular row
        console.log('One line from .csv >> ', data);
        parser.resume(); // to continue reading
    }).on("end", function () {
        resolve('csv parse process finished')
    }).on("error", function () {
        reject('csv parse process failed')
    });
});

try { await csvParsePromise; }
catch(err) {
    console.log('an error has occurred');
}```

score 0 · Answer 3 · answered Sep 28 '22 at 09:30

Get CSV content is simple
Try this

export async function getS3Object(
  bucket: string,
  key: string,
  s3: Pick<S3, "getObject">
): Promise<string> {
  const { Body } = await s3
    .getObject({
      Bucket: bucket,
      Key: key,
    })
    .promise();
  if (!Body) {
    throw new Error(`S3: Empty body for ${bucket} - ${key}`);
  }

  return Body.toString("utf-8");
}

How to parse CSVs from s3 bucket to use in a javascript AWS Lambda function

3 Answers3