deploy nodejs app to gae/se: gcloud app deploy : This deployment has too many files

Question

I want to run gitlab-ci pipeline to deploy my express.js web app to GAE/SE environment.

When my gitlab-ci pipeline begins and runs gcloud app deploy, gcloud command fails as below:

ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: This deployment has too many files. New versions are limited to 10000
      files for this app.
    field: version.deployment.files[...]

I tried to count my files using command find . -type f | wc -l, but it says there are only 3886 files in my repository.

My questions:

Why GAE/SE server says there're too many files to upload?
If I overlook something, is there any means to resolve this? (Or, I have to use flex env?)

.gitlab-ci.yml

image: node:8

cache:
  paths:
  - node_modules/

stages:
  - deploy

deploy:production:
  stage: deploy
  environment:
    name: production
    url: `${GAE_URL}`
  before_script:
    - wget https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz
    - tar zxvf google-cloud-sdk.tar.gz && ./google-cloud-sdk/install.sh --usage-reporting=false --path-update=true
    - PATH="google-cloud-sdk/bin:${PATH}"
  script:
    - echo $GAE_KEY_FILE > gae_auth.json
    - ./google-cloud-sdk/bin/gcloud auth activate-service-account --key-file gae_auth.json
    - ./google-cloud-sdk/bin/gcloud app deploy --project=$GAE_PROJECT_ID
  only:
  - dev

.gcloudignore

.git/
node_modules/
*~

score 2 · Answer 1 · answered Dec 17 '18 at 21:37

There is a limit of 1000 files per directory and 10000 files per app on App Engine. I believe that a library being deployed with your app is causing this error. You can use the app.yaml file skip_files directive to ignore file(s)/folder(s) that are not needed from being deployed to App Engine.

To identify which file(s) are being deployed and potentially causing this error you can set the verbosity flag to debug in your deploy command as follows.

gcloud app deploy --verbosity=debug

I have also found a similar StackOverflow answer that describes all of this detail here.

Hope this helps.

Thank you for your advice. Though ```skip_files``` settings inside app.yaml file does not resolve this issue, ```gcloud app deploy --verbosity=debug``` options help me understand its behavior more clearly. — tm1o2, Dec 18 '18 at 13:24

score 2 · Answer 2 · answered Dec 18 '18 at 03:39

If I read your .gitlab-ci.yml correctly you seem to be installing the cloud SDK inside the same directory from which you're deploying your app - which is thus your app directory.

Since the SDK dir is not included in your .gcloudignore file it is uploaded as part of your application. And I counted the files in the SDK dir after your wget and tar zxvf commands:

$> wget https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz
$> tar zxvf google-cloud-sdk.tar.gz
$> find google-cloud-sdk/ | wc -l
6442

Adding that to your app's 3.8k files - it's over the 10k limit.

You could try to install the SDK somewhere outside you app dir or you could add its dir pattern to the .gcloudignore file.

Thank you so much for your help. Your assumption is absolutely correct. After I added `google-cloud-sdk/` dir to `.gcloudignore` file, `gcloud app deploy` commands successfully finished. I think I have to know more about how docker and gitlab-ci runner works. — tm1o2, Dec 18 '18 at 13:30

deploy nodejs app to gae/se: gcloud app deploy : This deployment has too many files

2 Answers2