Folder Iterator Continuation Token Recursive Loop

Question

Thank you guys for all your help thusfar with setting up this recursive loop in Team drive. Below is what I have so far. My only problem at this point is that there are tens of thousands of folders in my specified Team Drive. So I think I need to get a continuation token to keep the script running longer than 6 minutes. However I have absolutely no clue where in the script to get or store the tokens. I think the easiest would be to just run the token every 1-10 iterations to be safe. Below is my current code:

function generateFolderTree() {

  try {

    var rootFolder = DriveApp.getFolderById('xxxxxxxxxxxxxxx');
    var rootFolderName = rootFolder.getName();
    var rootsFolders = rootFolder.getFolders();

    //var ss = SpreadsheetApp.create("Emptyfolderslist"+ Math.floor(Date.now() / 1000));
    var ss = SpreadsheetApp.openById('xxxxxxxxxxxxxxxxxxxx');

    var sheet = ss.getSheets()[0];
    sheet.appendRow(["Parent Folder", "Folder URL", "Folder Name", "", "folder count", "file count"]);

    while( rootsFolders.hasNext()) {

      var currentFolder = rootsFolders.next()          
      var currentFolderFiles = currentFolder.getFiles();
      var currentFolderFolders = currentFolder.getFolders();
      var folderscount = 0;
      var filescount = 0;

      while( currentFolderFiles.hasNext()){
        filescount ++;
        currentFolderFiles.next();
      }

      while( currentFolderFolders.hasNext()){
        folderscount++;
        currentFolderFolders.next();
      }

      var path = '';
      var parentfldrs = currentFolder.getParents();
      while( parentfldrs.hasNext()) {
        path = path + " - " + parentfldrs.next().getName();
      }

      sheet.appendRow(["'============================="]);
      sheet.appendRow([path, currentFolder.getUrl(), currentFolder.getName(), "", folderscount, filescount]);

      getChildFolders(currentFolder, ss); 
    }
  } catch (e) {
    Logger.log(e.toString());
  }
}


function getChildFolders(aFolder, ss) {

  var childFolders = aFolder.getFolders();
  var filesInFolder = aFolder.getFiles();
  var sheet = ss.getSheets()[0];

  var foldercount = 0;
  var filecount = 0;
  var childFolder = '';
  var parentFolders = '';
  var loop = 0;

  while( childFolders.hasNext()){
    foldercount = 0;
    filecount = 0;
    childFolder = childFolders.next();

    var subFolders = childFolder.getFolders();
    var subFiles = childFolder.getFiles();

    while( subFolders.hasNext()) {
      foldercount++;
      subFolders.next();
    }

    while( subFiles.hasNext()) {
      filecount++;
      subFiles.next()
    }

    var parentfldrs = childFolder.getParents();

    while( parentfldrs.hasNext()) {
      parentFolders = parentfldrs.next().getName();
    }

    sheet.appendRow([parentFolders, childFolder.getUrl(), "'----", childFolder.getName(), foldercount, filecount]);

    getChildFolders(childFolder, ss);
  }
}

I have read a couple other links here with similar questions using search, shown below:

Google Apps Script: How to use ContinuationToken with recursive folder iterator

Google Apps Script To Copy Entire Google Drive File Structure; How To Avoid Timeouts?

Answer 1

The only way to store dynamic items between execution calls to Google Apps Script is to use either the PropertiesService or CacheService. Static items which generally do not change between executions (such as a spreadsheet id or a reference to a particular CacheService) can safely be stored as global variables.

If you plan to use the continuation token within a few minutes / hours, I would recommend CacheService. If you need to keep the continuation token for use after more than 6 hours, PropertiesService is required (as 6 hours is the max time something will stay in CacheService after being put in).

You will need to add program logic that enables resuming from an arbitrary point in your loops. Resuming from an arbitrary point will be easier to do if you ensure you only enter loops that will complete in the time remaining - i.e. store the start time in a variable (var start = new Date().getTime()), and check how much time is remaining before you start a new loop (if (new Date().getTime() - start < someMaxDuration) {...}). If there is insufficient time (say 1m, 30s, etc), then you should save state and exit. If you do not save state and exit cleanly, you will duplicate data, as some inner loops will get repeated.

The simplest place to start is storing the continuationToken for the root level iterator, as done in the answer to your linked question: Google Apps Script To Copy Entire Google Drive File Structure; How To Avoid Timeouts? :

...
while( rootFolders.hasNext() {
  // If more than 2/3 the execution time has been used, bail out.
  if( new Date().getTime() - start > 200 * 1000)
    return;

  ...

  getChildFolders(currentFolder, ss);

  // Store the continuation token for the maximum time (6h).
  if( rootFolders.hasNext())
    cache.put("ROOT_CONTINUE_TOKEN", rootFolders.getContinuationToken(), 21600);
  else
    cache.remove("ROOT_CONTINUE_TOKEN");
}
...

To resume, you'd check cache/properties for that key first, and if not present, start from the beginning. You might need to store additional values in Cache/Properties (and similarly use the stored values):

var cache = CacheService.getScriptCache();
function generateFolderTree() {
  var cached = cache.get("someKey");
  var folderIterator;
  if(cached) {
    folderIterator = DriveApp.continueFolderIterator(cached);
    /* code for initializing a resumed iteration */
  }
  else {
    /* code for initializing an iteration from scratch */
  }

  /* code for iterating */
}

Answer 2

If you want an out-of-the-box template you can use for recursive folders and pause/resume feature. Check out my answer to a similar question.

Essentially this is done by storing an array of objects so that we know where to resume in each folder iterator and file iterator.