Firebase Google Cloud Function: createReadStream results in empty file

Question

I try to process a Video file (stored in Google Firebase storage) through a Google Cloud Function. I have working code that download the entire video files into the NodeJS Google cloud function: await bucket.file(filePath).download({ destination: tempFile }). But the goal is only to read the framerate, therefore the headers of the videofile would suffice. But createReadStream gives me an empty tempFile. Any advise much appreciated!

exports.checkFramerate = functions.region('europe-west1').storage.object().onFinalize(async (object, context) => {
    const bucket =  admin.storage().bucket(object.bucket); // Bucket class
    const filePath = object.name;   //  videos/xbEXdMNFb1Blbd9r2E8m/comp_test.mp4
    const fileName = filePath.split('/').pop(); // comp_test.mp4
    const bucketDir = path.dirname(filePath); // videos/xbEXdMNFb1Blbd9r2E8m

    const tempFile = path.join(os.tmpdir(), 'temp.mp4')
    fs.closeSync(fs.openSync(tempFile, 'w'))
    console.log("tempFile size1", fs.statSync(tempFile).size)

    // await bucket.file(filePath).download({ destination: tempFile }); // this works: tempFile size2 = 3180152
    await bucket.file(filePath).createReadStream({                      // this does not work: tempFile size2 = 0
        start: 10000,
        end: 20000
      })
      .on('error', function(err) {console.log(err)})
      .pipe(fs.createWriteStream(tempFile));


    console.log("tempFile size2", fs.statSync(tempFile).size)

    mi(tempFile).then(data => {
        console.log("frameRate", data[0].general.frame_rate[0])
        return data[0].general.frame_rate[0];
    }).catch(e => {console.error(e)});
});

I tried implementing even the example of https://googleapis.dev/nodejs/storage/latest/File.html#createReadStream but to no avail. remoteFile.download works beautifully but remoteFile.createReadStream gives me empty files...

const remoteFile = bucket.file(filePath);
const localFilename = tempFile;

remoteFile.createReadStream()
    .on('error', function(err) {})
    .on('response', function(response) {})
    .on('end', function() {})
    .pipe(fs.createWriteStream(localFilename));

fs.stat(localFilename, (err, stats) => {
    if (err) {console.log(err)}
    return console.log("stats async",stats.size)
})

mrCod3r · Answer 1 · 2021-01-22T16:37:20.517

as mentioned, promise should be used

reading json file example

 let  buf = '';

 const loadData = async () => {
  return await new Promise((resolve, reject) => {
    storage.bucket('bucket-name').file('test-config.json')
      .createReadStream()
      .on('error', reject)
      .on('data', function(d) {
        buf += d;
      }).on('end', function() {
        resolve(buf)
      });
  })
}

const data = await loadData()

robsiemb · Accepted Answer · 2019-12-07T05:06:46.787

Your problem is that the stream API isn't promisifed. So, the await does nothing, and your function continues before the stream is piped, and the file is still zero-length when you stat it the second time.

The download method works just fine because it returns a Promise.

This answer outlines the general approach you need to take. In summary though, you basically want the section of your code that does the piping to read like this:

const stream = bucket.file(filePath).createReadStream({
    start: 10000,
    end: 20000
  })
  .pipe(fs.createWriteStream(tempFile));

await new Promise((resolve, reject) => {
    stream.on('finish', resolve);
    stream.on('error', reject);
  });

console.log("tempFile size2", fs.statSync(tempFile).size)

Your function will then wait until the finish event occurs when the piping is complete and the stream is closed. Obviously you probably want to do something more clever with the error handler too, but this is the general form of what you need.

Firebase Google Cloud Function: createReadStream results in empty file

2 Answers2

Linked