How to get a list of files in a Google Cloud Storage folder using Node.js?

Question

Using bucket.getFiles() it is possible to get all the files in a bucket.

My bucket has thousands of files and I really only want to get metadata on the files in a particular folder.

The documentation is not clear on how to just get the files from a folder. Apparently it is possible to limit the results with a GetFilesRequest but none of the options include a path or a folder, at least not explicitly.

Answer 1

There is an ability to specify a prefix of the required path in options, e.g.

async function readFiles () {
  const [files] = await bucket.getFiles({ prefix: 'users/user42'});
  console.log('Files:');
  files.forEach(file => {
    console.log(file.name);
  });
};

Now it's finally available on documentation (thanks @Wajahath for the update): https://googleapis.dev/nodejs/storage/latest/Bucket.html#getFiles

Answer 2

Google cloud storage doesn't have folders/sub directories. It is an illusion on top of the flat namespace. i.e what you see as sub directories are actually objects which has a "/" character in its name.

You can read more about how Google cloud storage subdirectories work at the following link https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork

So by setting the prefix parameter of GetFilesRequest to the sub directory name you are interested in will return the object you are looking for.

Answer 3

If you have lots of files in your bucket, you might want to look into listing them as a stream, so data aren't held up in the memory during the query.

GetFiles, lists everything in one shot:

  admin.storage().bucket()
    .getFiles({ prefix: 'your-folder-name/', autoPaginate: false })
    .then((files) => {
      console.log(files);
    });

getFilesStream, lists everything as a stream:

  admin.storage().bucket()
    .getFilesStream({ prefix: 'your-folder-name/' })
    .on('error', console.error)
    .on('data', function (file) {
      console.log("received 'data' event");
      console.log(file.name);
    })
    .on('end', function () {
      console.log("received 'end' event");
    });

Full docs and examples: link