0

I am trying to get data from a file from s3 bucket and write to a temp file, then read the data from that temp file. The file was successfully created and data which is some html shown in the temp file, but when I try to console.log the next step, it return empty.

 try {   
    ...
    const params = {Bucket: "somebucket", Key: "file.html"}
    let tempFile = fs.createWriteStream("./temp/file.html", 'utf8')
    s3.getObject(params).createReadStream().pipe(tempFile)

    fs.readFile('./temp/file.html', 'utf8', (err, data) => {
        if (err) console.log(err)
        else {
           console.log("cannot get data?") //this shows up
           console.log(data) // this one does not shows up
        }
    })
    ...
} catch {
  ...
}

Anyone have experience in this?

Update:

I realize the data is inside the file and can be output fine without the getObject() part, but the data cannot be display out. It might be due to the file being read before the data is fully inserted to the file. How should i change the code so I can delay the readFile to execute after the data inserted in the file is complete. I tried setTimeout but failed.

andromad
  • 143
  • 1
  • 2
  • 8
  • [How to get response from S3 getObject in Node.js?](https://stackoverflow.com/a/36944450/174777) – John Rotenstein Mar 20 '19 at 15:21
  • nope, the data is inside the file, so the getObject part is success. Maybe I need to delay the readFile execute after the getObject part completed. – andromad Mar 20 '19 at 15:47
  • i tried to use getObject without using pipe() and it did displayed the data but it didn't create the file since the createReadStream and createWriteStream have been removed. And what I want is get the data and save to local file first, then use the file as data source to further proceed. – andromad Mar 20 '19 at 16:42

1 Answers1

0

This is most likely an issue caused by the asynchronous nature of nodejs. Your function calls (s3.getObject and fs.readFile) are running async to each other, and therefore fs.readFile has nothing it can read when it executes.

You can await s3.getObject with a callback/promise and then fs.readFile once the callback/promise completes (or just access the data object directly instead of writing it to a file, depending on your use case, this option might be easier)

Deiv
  • 3,000
  • 2
  • 18
  • 30
  • but why readFileSync() is not working? and i tried await s3.getObject before but no luck. Maybe I did not do it with a promise. – andromad Mar 20 '19 at 17:00
  • 1
    [Seems like this is an issue with using createReadStream and then trying to readFile](https://stackoverflow.com/questions/49656822/using-readfilesync-after-creating-a-file-with-createwritestream-shows-empty-buff). I would suggest doing s3.getObject and handling the data without a createReadStream, instead, write the data returned directly to the file using writeFile – Deiv Mar 20 '19 at 17:05