Error deploying TensorFlow App with AppEngine GoogleCloud

Question

I have deployed correctly the python3 App building-an-app:

https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard_python37/building-an-app/building-an-app-1

But now I have tried to deploy my App that use a neuronal network, and it works correctly in local with virtualenv and using requeriments.txt file:

Flask==1.1.1
numpy==1.17.4
scipy==1.3.2
gensim==3.8.1
tensorflow==2.0.0

But when I use "gcloud app deploy", after 5 minutes with this message "Updating service [default]...", I am getting this error:

ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build e281a017-86ee-450f-8291-bac8a7a0df30 status: FAILURE.
Build error details: {"error":{"errorType":"BuildError","canonicalCode":"INVALID_ARGUMENT","errorId":"3883CE8A","errorMessage":"02 Dec 2019 22:12:03 INFO     Arguments: ['--parser_script=/usr/local/bin/ftl.par', '--src=', '--entrypoint=', '--name=eu.gcr.io/lullaai-chatbot/app-engine-tmp/app/ttl-2h:8f040a55-3198-4998-beb0-75817f57192c', '--directory=/workspace', '--destination=/srv', '--cache-repository=eu.gcr.io/lullaai-chatbot/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00']\n02 Dec 2019 22:12:03 INFO     Unparsed arguments: ['--name=eu.gcr.io/lullaai-chatbot/app-engine-tmp/app/ttl-2h:8f040a55-3198-4998-beb0-75817f57192c', '--directory=/workspace', '--destination=/srv', '--cache-repository=eu.gcr.io/lullaai-chatbot/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00']\n02 Dec 2019 22:12:03 INFO     Executing ['/usr/local/bin/ftl.par', '--name=eu.gcr.io/lullaai-chatbot/app-engine-tmp/app/ttl-2h:8f040a55-3198-4998-beb0-75817f57192c', '--directory=/workspace', '--destination=/srv', '--cache-repository=eu.gcr.io/lullaai-chatbot/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00']\nINFO     FTL version python-v0.17.0\nINFO     Beginning FTL build for python\nINFO     FTL arg passed: virtualenv_dir /env\nINFO     FTL arg passed: ttl 168\nINFO     FTL arg passed: python_cmd /opt/python3.7/bin/python3.7\nINFO     FTL arg passed: cache True\nINFO     FTL arg passed: virtualenv_cmd virtualenv\nINFO     FTL arg passed: entrypoint None\nINFO     FTL arg passed: exposed_ports None\nINFO     FTL arg passed: pip_cmd /env/bin/python3.7 -m pip\nINFO     FTL arg passed: tar_base_image_path None\nINFO     FTL arg passed: export_cache_stats False\nINFO     FTL arg passed: builder_output_path \"\"\nINFO     FTL arg passed: destination_path /srv\nINFO     FTL arg passed: sh_c_prefix False\nINFO     FTL arg passed: base eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00\nINFO     FTL arg passed: cache_key_version v0.17.0\nINFO     FTL arg passed: cache_salt \nINFO     FTL arg passed: cache_repository eu.gcr.io/lullaai-chatbot/app-engine-tmp/build-cache/ttl-7d\nINFO     FTL arg passed: venv_cmd /opt/python3.7/bin/python3.7 -m venv /env\nINFO     FTL arg passed: name eu.gcr.io/lullaai-chatbot/app-engine-tmp/app/ttl-2h:8f040a55-3198-4998-beb0-75817f57192c\nINFO     FTL arg passed: global_cache False\nINFO     FTL arg passed: upload True\nINFO     FTL arg passed: fail_on_error True\nINFO     FTL arg passed: output_path None\nINFO     FTL arg passed: directory /workspace\nINFO     FTL arg passed: additional_directory /.googleconfig\nINFO     FTL arg passed: verbosity DEBUG\nINFO     starting: full build\nINFO     starting: builder initialization\nINFO     Loading Docker credentials for repository 'eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00'\nINFO     Loading Docker credentials for repository 'eu.gcr.io/lullaai-chatbot/app-engine-tmp/app/ttl-2h:8f040a55-3198-4998-beb0-75817f57192c'\nINFO     builder initialization took 0 seconds\nINFO     starting: build process for FTL image\nINFO     starting: checking_cached_interpreter_layer\nINFO     starting: check python version\nINFO     python version full cmd:\n/opt/python3.7/bin/python3.7 --version\nINFO     python version stderr:\n\nINFO

Anybody know what could be the problem?

Checking the build log for errors, I have discovered one line that says:

Step #1 - "builder": exited with error [Errno 12] Cannot allocate memory

So I have changed app.yaml to:

runtime: python37

handlers:
  # This configures Google App Engine to serve the files in the app's static
  # directory.
- url: /static
  static_dir: static

  # This handler routes all requests not caught above to your main app. It is
  # required when static routes are defined, but can be omitted (along with
  # the entire handlers section) when there are no static files defined.
- url: /.*
  script: auto

resources:
  cpu: 2
  memory_gb: 8
  disk_size_gb: 10

But I am still getting the same error :(

It is the build log:

ERROR: build step 1 "gcr.io/gae-runtimes/python37_app_builder:python37_20191019_3_7_4_RC00" failed: exit status 1
ERROR
Finished Step #1 - "builder"
Step #1 - "builder": IOError: [Errno 2] No such file or directory: '""/output'
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__/ftl/common/ftl_error.py", line 77, in InternalErrorHandler
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__.py", line 60, in main
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
Step #1 - "builder":     exec code in run_globals
Step #1 - "builder":   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Step #1 - "builder":     "__main__", fname, loader, pkg_name)
Step #1 - "builder":   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": Traceback (most recent call last):
Step #1 - "builder": tar_runtime_package is likely not on the path
Step #1 - "builder": exited with error [Errno 12] Cannot allocate memory
Step #1 - "builder": ERROR    tar_runtime_package tar -pcf /tmp/tmplWd4v8.tar --hard-dereference --transform flags=r;s,^,/.googleconfig/, --exclude *.pyc .
Step #1 - "builder": INFO     full build took 79 seconds
Step #1 - "builder": INFO     build process for FTL image took 79 seconds
Step #1 - "builder": INFO     Building app layer took 0 seconds
Step #1 - "builder": INFO     tar_runtime_package took 0 seconds
Step #1 - "builder": INFO     tar_runtime_package tar -pcf /tmp/tmplWd4v8.tar --hard-dereference --transform flags=r;s,^,/.googleconfig/, --exclude *.pyc .
Step #1 - "builder": INFO     starting: tar_runtime_package
Step #1 - "builder": INFO     starting: Building app layer
Step #1 - "builder": INFO     Building app layer took 76 seconds
Step #1 - "builder": INFO     Finished gzipping tarfile.
Step #1 - "builder": INFO     gzip_tar_runtime_package took 58 seconds
Step #1 - "builder": 
Step #1 - "builder": INFO     `gzip_tar_runtime_package` stdout:
Step #1 - "builder": INFO     gzip_tar_runtime_package gzip /tmp/tmpStZGpR.tar -1
Step #1 - "builder": INFO     starting: gzip_tar_runtime_package
Step #1 - "builder": INFO     tar_runtime_package took 8 seconds
Step #1 - "builder": 
Step #1 - "builder": INFO     `tar_runtime_package` stdout:
Step #1 - "builder": INFO     tar_runtime_package tar -pcf /tmp/tmpStZGpR.tar --hard-dereference --transform flags=r;s,^,/srv/, --exclude *.pyc .
Step #1 - "builder": INFO     starting: tar_runtime_package
Step #1 - "builder": INFO     starting: Building app layer
Step #1 - "builder": INFO     checking_cached_requirements.txt_layer took 1 seconds
Step #1 - "builder": INFO     [CACHE][HIT] v0.17.0:PYTHON (requirements)->62b99263456c1af3bff1307a585449789858bb17c695a2accfa55a3e5ee8f950
Step #1 - "builder": INFO     check python version took 0 seconds
Step #1 - "builder": 
Step #1 - "builder": INFO     `python version` stderr:
Step #1 - "builder": /opt/python3.7/bin/python3.7 --version
Step #1 - "builder": INFO     `python version` full cmd:
Step #1 - "builder": INFO     starting: check python version
Step #1 - "builder": 
Step #1 - "builder": gunicorn
Step #1 - "builder": tensorflow==2.0.0
Step #1 - "builder": gensim==3.8.1
Step #1 - "builder": scipy==1.3.2
Step #1 - "builder": numpy==1.17.4
Step #1 - "builder": Flask==1.1.1
Step #1 - "builder": INFO     new_descriptor_contents: 
Step #1 - "builder": 
Step #1 - "builder": gunicorn
Step #1 - "builder": tensorflow==2.0.0
Step #1 - "builder": gensim==3.8.1
Step #1 - "builder": scipy==1.3.2
Step #1 - "builder": numpy==1.17.4
Step #1 - "builder": Flask==1.1.1
Step #1 - "builder": INFO     descriptor_contents:
Step #1 - "builder": INFO     {"architecture": "amd64", "author": "Unknown", "config": {}, "created": "2019-12-02T22:00:00Z", "history": [{"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Unknown", "created": "2019-12-02T22:00:00Z", "created_by": "//containerregistry/client:ftl.par"}, {"author": "Bazel", "created": "1970-01-01T00:00:00Z", "created_by": "bazel build ..."}], "os": "linux", "rootfs": {"diff_ids":

Thanks!

Answer 1

Thanks for the feedback, I finally got it working. I had to change the app.yaml to manual scaling and env flex:

runtime: python37
env: flex

manual_scaling:
  instances: 1

resources:
  cpu: 2
  memory_gb: 8
  disk_size_gb: 10

and I had to create a docker image, and push it to repository, then use this image for deploy. It is the Dockerfile:

FROM gcr.io/google-appengine/python

COPY ["main.py", "/"]
COPY ["model.h5", "/"]
COPY ["run_server.sh", "/"]

RUN chmod +x /run_server.sh

RUN pip3 install --upgrade pip

RUN pip3 install tensorflow && \
    pip3 install h5py && \
    pip3 install flask && \
    pip3 install gensim


EXPOSE 8080

ENTRYPOINT ["/run_server.sh"]
CMD ["master"]

And it is the run_server.sh that I use:

#!/bin/bash
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
python3 -u /main.py

Then you have to follow the next steps:

Generation Docker Image: docker build . -t [IMAGE]

Tag Local Docker Image: docker tag [IMAGE] eu.gcr.io/[PROJECT-ID]/[IMAGE]

Push Docker Image: docker push eu.gcr.io/[PROJECT-ID]/[IMAGE]

Deploy App: gcloud app deploy --image-url=eu.gcr.io/[PROJECT-ID]/[IMAGE]

And it is working nice now :)

Answer 2

The error you’re experiencing seems to be pointing out that the GAE instance you’re using is running out of memory.

As an option, you might want to try changing the class of the instance to the one that has a higher memory limit. However, keep in mind that it will also affect the pricing.