Workaround to 6 minutes app script limits on google drive file copying

Question

/**
 * Copies all the files in a folder to another one.
 */
function copyFolderContents_(source, target) {
  // Iterate files in source folder
  const filesIterator = source.getFiles()
  while (filesIterator.hasNext()) {
    const file = filesIterator.next()

    // Make a copy of the file keeping the same name
    file.makeCopy(file.getName(), target)
  }
}

/**
 * Recursivelly copies a folder and all its subfolders with all the files
 */
function copyFolder_(toCopy, copyInto) {
  // Makes the new folder (with the same name) into the `copyInto`
  const newFolder = copyInto.createFolder(toCopy.getName())

  // Copy the contents
  copyFolderContents_(toCopy, newFolder)

  // Iterate any subfolder
  const foldersIterator = toCopy.getFolders()
  while (foldersIterator.hasNext()) {
    const folder = foldersIterator.next()

    // Copy the folder and it's contents (recursive call)
    copyFolder_(folder, newFolder)
  }
}

/**
 * Entry point to execute with the Google Apps Script UI
 */
function copyFolder() {
  // Get the folders (by ID in this case)
  const toCopy = DriveApp.getFolderById('')
  const copyInto = DriveApp.getFolderById('')

  // Call the function that copies the folder
  copyFolder_(toCopy, copyInto)
}

this script suffer from the 6 minutes app script limit when copying large folders.

How to solve this problem? recursion makes it difficult to divide the files into several parts and let app-script to run several times using this

You could gather files IDs and folders structure first. Probably it takes less than 6 minutes. Then you could make all folders by the list (if it takes less 6 minutes). Then you could divide the list ID's into several small lists, etc. But probably the most efficient way is just Desktop version of G-Drive. — Yuri Khristich, Aug 11 '21 at 09:50
There isn't a simple answer. The idea would be to stop your execution after 5 minutes and create a trigger to start again in one minute. And you also want to save your progress somewhere, like a JSON file so that the script doesn't start from the beginning. It's a fairly complex procedure, but I've implemented it in the past so it works. I will be publishing an article about this approach on Medium in the coming weeks, so you might want to keep an eye on that if you are interested. — Dmitry Kostyuk, Aug 11 '21 at 09:53
@LukAron take a look at [this answer](https://stackoverflow.com/a/8608327/11551468) it effectively uses this approach. You would need to make sure to get a list of all the file IDs first and save them to the properties store then loop through them copying each one and removing it from the queue. `DriveApp.getFiles()` seems to return files in the order that they were last modified, so on successive runs you won't get the same order and so storing the full list in the first run is important. — Rafa Guillermo, Aug 11 '21 at 11:26
Does this answer your question? [Google app script timeout ~ 5 minutes?](https://stackoverflow.com/questions/14450819/google-app-script-timeout-5-minutes) — Linda Lawton - DaImTo, Oct 20 '21 at 14:32

score 2 · Accepted Answer · answered Aug 16 '21 at 17:30

2

As promised, here's a fairly long and detailed answer on how to bypass the maximum script runtime.

The basics are: you need to exit gracefully, save your progress, set up a trigger, re-run as many times as necessary, then clean up the trigger and any files you saved your progress in.

answered Aug 16 '21 at 17:30

Dmitry Kostyuk

1,354
1
5
21

This article has been an immense help, thank you - I'm working on the questions at the end! Also got a lot out of [your article](https://medium.com/geekculture/using-design-patterns-in-google-apps-script-ceaa7606d2d6) on design patterns in GAS, but one question is tormenting me. In the Forex facade you have a `WeakMap` named `fetch`. As far as I can tell it caches the `_fetch` function, not its return value, for each set of parameter values. Why? Or have I misunderstood? JS isn't my first language. – Chris Aug 23 '21 at 10:23
1

Hi @Chris, thanks for your feedback, I appreciate it. Regarding the Design Patterns article, to avoid spamming everyone here, I invite you to ask you question directly on the article in question and I will be happy to respond promptly. – Dmitry Kostyuk Aug 23 '21 at 12:54
Thanks @Dmitry, I have posted there. Tried slightly cheekily here as medium deducts an article from my three per month allowance when I log in, so that's your two articles..will have to investigate private browsing, etc. – Chris Aug 23 '21 at 14:01
1

Here are the friend links that don't affect your quota: [Design Paterns](https://medium.com/geekculture/using-design-patterns-in-google-apps-script-ceaa7606d2d6?sk=fc97b1ecb43c8a8aaef739aafe1e30c5), [Script Runtime](https://medium.com/geekculture/bypassing-the-maximum-script-runtime-in-google-apps-script-e510aa9ae6da?sk=ca2b3c4d2fa429e02ba5f81a94d0c780) – Dmitry Kostyuk Aug 23 '21 at 14:08

score 0 · Answer 2 · answered Aug 11 '21 at 12:05

I think due to recursion, separate folders and file to be copy in a simple for loop and use this to run is no longer feasible, since the task is dependent, child couldnt be copied if parents aren't there yet.

the only workaround is to build a tree in some database to record the heiarachy in account A, and each node in the tree is marked with a label "copied" if have been gone thought by the script.

after each time the script terminates, we navigate the tree in a DFS way to find the undone node(file or folder)

mongodb support 100 depth in max. Should be able to capture the tree. ref

score 0 · Answer 3 · answered Aug 24 '21 at 19:21

You could use my answer here plus something like the facade below for the quickest approach, although might be more complex than you want.

class DriveFacade {
  // drive functions facade

  static createFolder(parent, path) {
    // creates nested folders if they don't already exist
    let newFolder = parent
    let existsSoFar = true
    path.forEach(folderName => {
      if (existsSoFar) {
        const existingFolder = newFolder.getFoldersByName(folderName)
        if (existingFolder.hasNext()) {
          newFolder = existingFolder.next()
        } else {
          existsSoFar = false
        }
      }
      if (!existsSoFar) {
        newFolder = newFolder.createFolder(folderName)
      }
    })
    return newFolder
  }

  static createOrReplaceFile(parent, filename, content, mimeType) {
    const files = parent.getFilesByName(filename)
    if (files.hasNext()) {
      const file = files.next()
      if (file.getMimeType() === mimeType) {
        file.setContent(content)
        return file
      } else {
        file.setTrashed(true)
      }
    }
    return parent.createFile(filename, content, mimeType)
  }
}

Workaround to 6 minutes app script limits on google drive file copying

3 Answers3

Linked