1

Im new to Async / Await and trying to make these 3 callouts to get Google Drive metadata in parallel using Node.js. The 3 callouts are: rootFolder, folders, files.

Since these call upon the getGdriveList function which itself is an async function and includes an await. The await is needed as the API returns a page of records and I need to loop through all the pages to get the records. I use the await to await the API response and add the data to an array. This process renders the code to run in series.

I am looking for help to refactor to make this parallel. Thanks in advance

const {google} = require('googleapis');
const gOAuth = require('./googleOAuth')
const aws = require('aws-sdk');

// initialize google oauth credentenatials 
let readCredentials = gOAuth.readOauthDetails('credentials.json')
let authorized = gOAuth.authorize(readCredentials, getGfiles)

// get Google meta data on files and folders
function getGfiles(auth) {
  let rootFolder = getGdriveList(auth, {corpora: 'user', 
                                        fields: 'files(name, parents)', 
                                        q: "'root' in parents and trashed = false and mimeType = 'application/vnd.google-apps.folder'"})

  let folders = getGdriveList(auth, {corpora: 'user', 
                                    fields: 'files(id,name,parents), nextPageToken', 
                                    q: "trashed = false and mimeType = 'application/vnd.google-apps.folder'"})

  let files = getGdriveList(auth, {corpora: 'user', 
                                    fields: 'files(id,name,parents, mimeType), nextPageToken', 
                                    q: "trashed = false and mimeType != 'application/vnd.google-apps.folder'"})

  files.then(result => {console.log(result)})
}

const getGdriveList = async (auth, params) => {
  let list = []
  let nextPgToken
  const drive = google.drive({version: 'v3', auth})
  do {
    let res = await drive.files.list(params)
    list.push(...res.data.files)
    nextPgToken = res.data.nextPageToken
    params.pageToken = nextPgToken
  }
  while (nextPgToken)
  return list
}
Richard Lewis
  • 125
  • 8
  • 2
    There's no `await` in `getGfiles` so these already *do* run concurrently. (You are however logging only the result from one of them, ignoring - and not waiting for - the other two. You should add a `Promise.all` for that.) – Bergi Jun 24 '20 at 21:56
  • Thanks @Bergi this is great info! so to confirm, even though there is an `await` in `getGdriveList` which is utilized in each of the 3 callouts. it still is parallel? – Richard Lewis Jun 24 '20 at 21:59
  • I've just added the following to my code: `files.then(result => {console.log(result)})` `folders.then(result => {console.log(result)})` `rootFolder.then(result => {console.log(result)})` the first to return is `rootFolder` then `folders` then `files` – Richard Lewis Jun 24 '20 at 22:01

2 Answers2

2

Expanding on @CameronTacklind 's answer, array destructuring is the more idiomatic way of doing this.

const [rootFolder, folders, files] = await Promise.all([
  getGdriveList(...),
  getGdriveList(...),
  getGdriveList(...),
]);

// more code ...
ATOMP
  • 1,311
  • 10
  • 29
0

You essentially currently have:

const promise1 = parallelBackendWork(...);
const promise2 = parallelBackendWork(...);
const promise3 = parallelBackendWork(...);

promise3.then(result3 => {console.log(result3)});

You're creating 3 promises but only .then() on one of them. If that one (promise3) finishes first, I suspect your execution won't work as expected.

I suspect you'll want to do something to combine all 3 of those promises. This is what Promise.all([]) is for. You might like to checkout Promise.race([]) as well.

const promise1 = parallelBackendWork(...);
const promise2 = parallelBackendWork(...);
const promise3 = parallelBackendWork(...);

const promises = Promise.all([ promise1, promise2, promise3 ]);

promises.then(results => {console.log(results)});

Note, results will be an Array. You probably want to use destructuring as @ATOMP suggests.

Cameron Tacklind
  • 5,764
  • 1
  • 36
  • 45