Putting s3 object data into an array - node

Question

I'm trying to fetch data from an S3 object and put it into an array. I plan to map through this array and display the data on a React front end in grid/list whatever. I'm struggling with nested functions though, so I'd appreciate some help.

const dataFromS3 = async (bucket, file) => {
let lines = [];

const options = {
    Bucket: bucket,
    Key: file
  };

s3.getObject(options, (err, data) => {
if (err) {
  console.log(err);
} else {
  let objectData = data.Body.toString('utf-8');
  lines.push(objectData);
  console.log(lines);
  return lines;
}
  });
};

Formatting is a bit weird but this is my function to get data from s3. I want to take the output of this function in the form of an array and pass it to my '/' route which I'm testing:

app.get('/', async (req, res, next) => {


try {
    let apolloKey = await dataFromS3(s3Bucket, apolloKeywords);
    res.send(apolloKey);
  } catch (err) {
    console.log('Error: ', err);
  }
});

It seems that the return value of lines in the s3.getObject function needs to be returned within the first function so that I can access it in app.get but I can't seem to do it after some attempts. The value in lines turns into an empty array if I return it at the end of datafromS3() and I can't find a way to return it. I've tried using promises also using a method found here - How to get response from S3 getObject in Node.js? but I get a TypeError: Converting Circular Structure to JSON...

Thank you

score 1 · Accepted Answer · answered Mar 10 '20 at 10:54

1

You need to make your dataFromS3 func like htis. You were not returning anything from that. AWS also provided promise based function.

const dataFromS3 = async (bucket, file) => {
  const lines = [];

  const options = {
    "Bucket": bucket,
    "Key": file
  };

  const data = await s3.getObject(options).promise();
  const objectData = data.Body.toString("utf-8");
  lines.push(objectData); // You might need to conversion here using JSON.parse(objectData);
  console.log(lines);
  return lines;
};

answered Mar 10 '20 at 10:54

Ashish Modi

7,529
2
20
35

Hi Ashish, thanks for the quick response! Your suggestion worked but rather than bringing back an array with the individual items it was all one big array in the format ['x,x2,x3,x4']. When trying to JSON.parse it there's an error saying (Syntax Error: Unexpected token 'x' in JSON at position 0, with 'x' in this case being the first letter of the first word in the list. I tried looping too const data = await s3.getObject(options).promise(); objectData = data.Body.toString('utf-8'); for (let eachWord in objectData) { eachWord = lines.push(objectData); return eachWord; } – devtsi Mar 10 '20 at 11:10
what is in that file? and what is the expected output? I might be able to help? – Ashish Modi Mar 10 '20 at 11:16
So the file is a .txt file with just a list of words. Because I wanted to display these words on a front end to users, I thought the best way to do this was with an array. What are your thoughts? Do you think there is a better way? – devtsi Mar 10 '20 at 11:17
if they are comma separated, You could simply do objectData.split(","). This will return you array of words. – Ashish Modi Mar 10 '20 at 11:19
I think I'll try this. They aren't separated by commas but I can add them in and then run this. I'll let you know how it goes! – devtsi Mar 10 '20 at 11:21
Ashish, the file we had in S3 was actually using newlines to split the text so I just split with objectData.split("\n") and I got my array. Thanks for the suggestion! – devtsi Mar 10 '20 at 17:36
Perfect. Glad it helped – Ashish Modi Mar 10 '20 at 17:36

Putting s3 object data into an array - node

1 Answers1