Read JSON file stored in aws S3 with Nodejs

Question

I managed to read a JSON file stored in my S3 bucket but I'm having to do a lot of transformation that I dont fully understand.

If we log the data as it comes we get an output of Buffer data

s3.getObject(objParam, (err, data) => {
  if (err) throw err;
  
  console.log(" ~ file: reader.js ~ line 31 ~ rs ~ data", data.Body)

});

// outputs  data <Buffer 7b 0a 20 20 22 74 79 70 65 22 3a 20 22 42 75 66 66 ...

If we convert the above buffer to string with toString('utf-8') I get this:

data.Body.toString('utf8')

// outputs:
{
  "type": "Buffer",
  "data": [
    123,
    34,
    99,
    115,
    112,
    45,
    114,
    101,
    112,
    111,
    114,
....

The only way I managed to see my actual original JSON was converting the above to JSON, accessing the data property, creating a new buffer, then converting back again to toString('utf-8').

s3.getObject(objParam, (err, data) => {
  if (err) throw err;

  const jsonBytes = JSON.parse(data.Body.toString('utf-8'));
  const buffer = Buffer.from(jsonBytes.data);
  const bufferBackToString = buffer.toString('utf-8');
  console.log(" ~ file: reader.js ~ line 21 ~ s3.getObject ~ bufferBackToString", bufferBackToString);

});

// output: Logs my original JSON!
{
"myProperty": "myData"
} // or whatever...

If it was already a buffer, why did I have to convert to one and convert back again to string? Could the buffer have different encoding?

What happens if you just do `data.toString()` to start instead of `data.Body.toString('utf-8')`? — Nisala, Apr 20 '21 at 15:43
No, I meant without the `.Body`, just data.toString() or data.toString('utf-8') (although utf8 is default so you shouldn't need it afaik) — Nisala, Apr 20 '21 at 17:32
it prints [object Object]... the stuff I need is inside data.Body, there are some other unrelated properties that I dont need in data — JBird, Apr 21 '21 at 08:22
Ah, okay. Try the solutions here: https://stackoverflow.com/questions/36942442/how-to-get-response-from-s3-getobject-in-node-js — Nisala, Apr 21 '21 at 15:07

SemyenP · Answer 1 · 2023-01-23T07:24:24.157

2

const getJsonFromS3 = async (filename, bucketName) => {
  const s3Output = await s3.getObject({
    Key: filename,
    Bucket: bucketName
  })

  try {
    const jsonString = await s3Output.Body?.transformToString()
    const json = JSON.parse(jsonString ?? '')
    return json
  } catch (error) {
    console.error('error parsing json', error)
    return null
  }
}

all suggestions with data.toString('utf-8') not working for me. but using transformToString() worked for me

edited Jan 23 '23 at 07:24

answered Jan 21 '23 at 19:23

SemyenP

43
7

And `data.toString('binary')` will preserve binary data. This should be the accepted answer. – timkay Apr 06 '23 at 15:17

score 1 · Answer 2 · edited Nov 04 '22 at 19:10

1

const jsonBytes = JSON.parse(data.Body.toString('utf-8'));
console.log(jsonBytes);

worked for me , I did see in forums , that s3.getObject returns json directly not string

edited Nov 04 '22 at 19:10

devzom

676
1
9
19

answered Oct 31 '22 at 14:49

Abderrahmen BRINIS

36
1
3

Read JSON file stored in aws S3 with Nodejs

2 Answers2