What might cause the firebase console to show the 'waiting' icon (spinning blue loop icon) after adding a file via cloud functions to cloud storage?

Question

So, I can now use a google cloud function to store a file in cloud storage, and after uploading a file I can see it's name in the firebase console, i.e. within the Storage area as you drill down via the various folders.

I can also query and retrieve+view the file, so it seems like it's stored ok.

However, I don't understand why I cannot click on the file in the firebase console and see a preview of it.

My only wild guess is that the cloud function to upload the file didn't release a file handle or something, and so firebase console thinks it's locked.

The file attributes shown in firebase console seem correct.

The cloud function code to upload the file is as follows:

const imageBuffer = Buffer.from(arr[1], 'base64');
const bufferStream = new stream.PassThrough();
bufferStream.end(imageBuffer);

const fullPath = `${filePath}/${data.filename}`

console.log('saving to fullPath', fullPath)

const myFile = bucket.file(fullPath)

bufferStream.pipe(myFile.createWriteStream({
       metadata: {
         contentType: mime
       },
       public: true,
       validation: "md5"
    }))
     .on('error', function (err) {
      console.log('error from image upload', err);
    })
    .on('finish', function () {
       console.log('!!!!!! finished')
     })

where arr[1] is the base64 portion of a string, i.e. remove the mime type stuff from the beginning so arr[1] is the pure file data as base64.

So, basically everything seems to be working perfectly except the firebase console can't view the file, unless I do an app redeploy, i.e. npm run build && firebase deploy && npm start, and then it seems (?) to free up the lock and I can view the image preview in the firebase console for firebase storage.

Have you tried the approach in [this answer](https://stackoverflow.com/a/48073631) where you push your data followed by `null` through `bufferStream` instead of using `.end()`? — samthecodingman, Apr 26 '21 at 09:05
nailed it. sending data followed by null fixed it. thanks. seems like a bug -- would have thought .end() is enough to signal that's all the data you're going to provide. also, mime is defined above, I just didn't show that bit of the code, but good spot. — Greg, Apr 28 '21 at 02:48
You are right in that it is a bug. The particular versions it affected aren't really known but it is gone in the latest LTS builds. This is just a version-agnostic way of circumventing the problem. — samthecodingman, Apr 28 '21 at 14:14

score 0 · Answer 1 · answered Apr 28 '21 at 02:50

0

Sending null at the end of the data as per the comments fixes this.

answered Apr 28 '21 at 02:50

Greg

31
2

score 0 · Answer 2 · answered May 01 '21 at 19:37

Revisiting this, rather than use the PassThrough stream which is adding unnecessary overhead, you could write your buffer to the stream itself directly.

const imageBuffer = Buffer.from(arr[1], 'base64');

const fullPath = `${filePath}/${data.filename}`

console.log('saving to fullPath', fullPath)

const myFile = bucket.file(fullPath)

const fileWriteStream = myFile
  .createWriteStream({
    metadata: {
      contentType: mime
    },
    public: true,
    validation: "md5"
  })
  .on('error', function (err) {
    console.log('error from image upload', err);
  })
  .on('finish', function () {
    console.log('!!!!!! finished')
  });
  
fileWriteStream.end(imageBuffer);

Rewritten as a promise-based utility function:

import { File } from "@google-cloud/storage";
import { storage } from "firebase-admin";

async function uploadDataURI(dataURI: string, options: { bucket?: string, filename: string, filepath: string, public?: boolean, contentType?: string }): Promise<File> {
  const { bucket, filename, filepath, public, contentType } = options;

  const [, mediaType, data] = dataURI.split(/[;,]/);

  let contentTypeFromURI: string, buffer: Buffer;
  if (mediaType.endsWith(";base64")) {
    contentTypeFromURI = mediaType.slice(0,-7);
    buffer = Buffer.from(data, "base64");
  } else {
    contentTypeFromURI = mediaType;
    buffer = Buffer.from(decodeURIComponent(data));
  }

  const storageRef = storage()
    .bucket(bucket)
    .file(`${filepath}/${filename}`);
    
  return new Promise((resolve, reject) => {
    try {
      const fileWriteStream = storageRef
        .createWriteStream({
          metadata: {
            contentType: contentType || contentTypeFromURI || undefined // coerce falsy values to undefined
          },
          public,
          validation: "md5"
        })
        .on('error', reject)
        .on('finish', () => resolve(storageRef));
      
      fileWriteStream.end(buffer);
    } catch (err) {
      reject(err);
    }
  }); 
}

What might cause the firebase console to show the 'waiting' icon (spinning blue loop icon) after adding a file via cloud functions to cloud storage?

2 Answers2

Linked