Read a file line by line using Lambda / S3

Question

I want to read a file line by line located on S3. I tried the following code which I found searching online, but the Lambda function is exiting without invoking any of the readline callbacks. What am I doing wrong?

const aws = require('aws-sdk');
const s3 = new aws.S3({ apiVersion: '2006-03-01' });
const readline = require('readline');

exports.handler = async (event, context, callback) => {
    const bucket = event.Records[0].s3.bucket.name;
    const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
    const params = {
        Bucket: bucket,
        Key: key,
    };

    const s3ReadStream = s3.getObject(params).createReadStream();

    const rl = readline.createInterface({
      input: s3ReadStream,
      terminal: false
    });

    rl.on('line', (line) => {
      console.log(`Line from file: ${line}`);
    });
    rl.on('error', () => {
        console.log('error');
    });
    rl.on('close', function () {
        console.log('closed');
        context.succeed();
    });
    console.log('done');
};

score 21 · Accepted Answer · edited May 23 '19 at 11:38

I've found the problem. It's being awhile that I haven't coded on Lambda and I thought it would only exit when context was called. I'm now waiting for the promise to be resolved (or rejected which I'll implement later).

const aws = require('aws-sdk');
const s3 = new aws.S3({ apiVersion: '2006-03-01' });
const readline = require('readline');

exports.handler = async (event, context, callback) => {
    const bucket = event.Records[0].s3.bucket.name;
    const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
    const params = {
        Bucket: bucket,
        Key: key,
    };
    const s3ReadStream = s3.getObject(params).createReadStream();

    const rl = readline.createInterface({
      input: s3ReadStream,
      terminal: false
    });

    let myReadPromise = new Promise((resolve, reject) => {

        rl.on('line', (line) => {
          console.log(`Line from file: ${line}`);
        });
        rl.on('error', () => {
            console.log('error');
        });
        rl.on('close', function () {
            console.log('closed');
            resolve();
        });
    });

    try { await myReadPromise; }
    catch(err) {
        console.log('an error has occurred');
    }

    console.log('done reading!');
};

Jesus man.... you saved my life... I spent 8 hrs trying to figure out why it didn't work. It was your exact solution.... — Anjan Biswas, Nov 11 '18 at 01:53
Thanks for sharing your solution! I had the same issue but couldn't get it to run. My code works great on my local machine, but times out on lambda. I tried running your code and also timed out. It seems the 'line' callback is never being called. Are there any additional steps you took to make this work? — davidrac, Jan 28 '19 at 14:25

score 0 · Answer 2 · answered Oct 12 '18 at 19:43

0

getObject doesn't just return the object that was stored S3. It return a JSON object whose Body field holds the blob of the object stored to S3. See also in the Response part of the documentation here.

answered Oct 12 '18 at 19:43

Kalev

1,158
7
17

Thanks. I was based myself on this code: https://gist.github.com/maxrabin/e3e51abc365cd3f54d78 Do you know of how I can accomplish what I want? – Raisen Oct 12 '18 at 23:41
@Raisen Interesting. If the code you linked to ever worked, then my answer is probably wrong. What does sticks out is the fact that you're using a mix of `async` and `context.succeed` (legacy style Lambda programming). Why not stick to either the code as it appears in the link, or rewrite it using promises exclusively? – Kalev Oct 13 '18 at 00:20
I've figured out the problem. You were right, promises helped to resolve this. Thanks for your help! – Raisen Oct 14 '18 at 00:38

Read a file line by line using Lambda / S3

2 Answers2

Linked