Deploying an app on Google App Engine - Error from requirements.txt

Question

I can't deploy an app on Google App Engine.

The problem seems to come from my requirements.txt file:

• When I deploy the example building-an-app-1, it works well.
• When I replace the original requirements.txt file with mine, it doesn't work*.

Here is my requirements.txt:

Flask==1.1.1
flask-wtf==0.14.2
unidecode
numpy
openfoodfacts
os

When I remove the last 2 packages, it works.

What is the problem?

*gcloud deploy app returns:

File upload done.
Updating service [default]...failed.                                           
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 813ab290-60a1-427c-830f-3a7e2d35f39b status: FAILURE.
Build error details: {"error":{"errorType":"BuildError","canonicalCode":"INVALID_ARGUMENT","errorId":"288DA415","errorMessage":"01 Feb 2020 17:23:53 INFO     Arguments: ['--parser_script=/usr/local/bin/ftl.par', '--src=', '--entrypoint=', '--name=eu.gcr.io/skilful-alpha-259713/app-engine-tmp/app/ttl-2h:34df8d6c-74e5-4583-bd94-165295531608', '--directory=/workspace', '--destination=/srv', '--cache-repository=eu.gcr.io/skilful-alpha-259713/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00']
01 Feb 2020 17:23:53 INFO     Unparsed arguments: ['--name=eu.gcr.io/skilful-alpha-259713/app-engine-tmp/app/ttl-2h:34df8d6c-74e5-4583-bd94-165295531608', '--directory=/workspace', '--destination=/srv', '--cache-repository=eu.gcr.io/skilful-alpha-259713/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00']
01 Feb 2020 17:23:53 INFO     Executing ['/usr/local/bin/ftl.par', '--name=eu.gcr.io/skilful-alpha-259713/app-engine-tmp/app/ttl-2h:34df8d6c-74e5-4583-bd94-165295531608', '--directory=/workspace', '--destination=/srv', '--cache-repository=eu.gcr.io/skilful-alpha-259713/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00']
INFO     FTL version python-v0.17.0
INFO     Beginning FTL build for python
INFO     FTL arg passed: virtualenv_dir /env
INFO     FTL arg passed: ttl 168
INFO     FTL arg passed: python_cmd /opt/python3.7/bin/python3.7
INFO     FTL arg passed: cache True
INFO     FTL arg passed: virtualenv_cmd virtualenv
INFO     FTL arg passed: entrypoint None
INFO     FTL arg passed: exposed_ports None
INFO     FTL arg passed: pip_cmd /env/bin/python3.7 -m pip
INFO     FTL arg passed: tar_base_image_path None
INFO     FTL arg passed: export_cache_stats False
INFO     FTL arg passed: builder_output_path \"\"
INFO     FTL arg passed: destination_path /srv
INFO     FTL arg passed: sh_c_prefix False
INFO     FTL arg passed: base eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00
INFO     FTL arg passed: cache_key_version v0.17.0
INFO     FTL arg passed: cache_salt 
INFO     FTL arg passed: cache_repository eu.gcr.io/skilful-alpha-259713/app-engine-tmp/build-cache/ttl-7d
INFO     FTL arg passed: venv_cmd /opt/python3.7/bin/python3.7 -m venv /env
INFO     FTL arg passed: name eu.gcr.io/skilful-alpha-259713/app-engine-tmp/app/ttl-2h:34df8d6c-74e5-4583-bd94-165295531608
INFO     FTL arg passed: global_cache False
INFO     FTL arg passed: upload True
INFO     FTL arg passed: fail_on_error True
INFO     FTL arg passed: output_path None
INFO     FTL arg passed: directory /workspace
INFO     FTL arg passed: additional_directory /.googleconfig
INFO     FTL arg passed: verbosity DEBUG
INFO     starting: full build
INFO     starting: builder initialization
INFO     Loading Docker credentials for repository 'eu.gcr.io/gae-runtimes/python37:python37_20191019_3_7_4_RC00'
INFO     Loading Docker credentials for repository 'eu.gcr.io/skilful-alpha-259713/app-engine-tmp/app/ttl-2h:34df8d6c-74e5-4583-bd94-165295531608'
INFO     builder initialization took 0 seconds
INFO     starting: build process for FTL image
INFO     starting: checking_cached_interpreter_layer
INFO     starting: check python version
INFO     `python version` full cmd:
/opt/python3.7/bin/python3.7 --version
I.

Answer 1

According to the error you are receiving Cloud Build is failing to install the last two dependencies that you are specifiying on your requirements.txt file. If you go to the Cloud Build History page and check your failed deployments you'll notice that the deployments are failing with a similar error message to this one:

Step #1 - "builder": INFO     `pip_download_wheels` had stderr output:
Step #1 - "builder":   Could not find a version that satisfies the requirement openfoodfacts==0.1.0 (from -r requirements.txt (line 10)) (from versions: )
Step #1 - "builder": No matching distribution found for openfoodfacts==0.1.0 (from -r requirements.txt (line 10))

There is no need to include the os module on the requirements.txt since it is part of Python's 3.7 Standard Library. Notice for example that the main.py you are basing your application from makes an import of the datetime module and this is not specified on the requirements.txt file.

The Open Food Facts package has not actually been released on PyPI (i.e., https://pypi.org/project/openfoodfacts/ is 404) so you'll need to refer to the relevant section of the App Engine's documentation and add it manually, in a similar fashion to this:

1. Create a lib directory on your application's Project Directory (e.g. my using mkdir lib).
2. Copy the dependencies you need to the new created lib directory by issuing the following command: pip3 install git+git://github.com/openfoodfacts/openfoodfacts-python.git -t lib.
3. Go to the lib directory and verify that all dependencies have been installed correctly.
4. Modify your requirements.txt file to use the dependencies required by the Open Food Facts package. It should look like this:

Flask==1.1.1
flask-wtf==0.14.2
unidecode==1.1.1
numpy==1.18.1
certifi==2019.11.28
chardet==3.0.4
idna==2.8
requests==2.22.0
urllib3==1.25.8

5. Import the module in your app in a similar fashion to import lib.openfoodfacts as openff.
6. Use the gcloud app deploy command to deploy your application.

At the end you should have a project directory with a similar file structure than this one:

building-an-app-1/
    app.yaml
    lib/
        ...
        openfoodfacts/
            autosuggest.py
            facets.py
            __init__.py
            openbeautyfacts.py
            openpetfoodfacts.py
            products.py
            __pycache__/
            utils.py
        ...
    main.py
    requirements.txt
    static/
        script.js
        style.css
    templates/
        index.html

where you can import the os and openfoodfacts modules on the main.py file by using the following:

...
import datetime
import os
import lib.openfoodfacts as openff
from flask import Flask, render_template
app = Flask(__name__)
@app.route('/')
....

Answer 2

Did you try using this requirements.txt file locally like this?:

pip install -r requirements.txt

It looks like the last two lines of your requirements.txt file do not correspond to Python packages that can be installed with pip.

Answer 3

Indeed, the presence of both os and openfoodfacts in the requirements.txt file (which needs to contain only pip-installable packages, see Declaring and managing dependencies) are causing pip install errors:

[venv] openfoodfacts> pip install os
Collecting os
  Could not find a version that satisfies the requirement os (from versions: )
No matching distribution found for os

[venv] openfoodfacts> pip install openfoodfacts
Collecting openfoodfacts
  Could not find a version that satisfies the requirement openfoodfacts (from versions: )
No matching distribution found for openfoodfacts

You shouldn't need to add os to your requirements, it is part of the standard python installation, you can just import it.

And openfoodfacts doesn't appear to be a regular Pypi package. If it's coming from a pip repository which doesn't require SSH keys you may be able to specify it in the requirements.txt file, see How to add a package-specific index-url to requirements.txt?. Otherwise you need to install it in your app so that it can be uploaded together with your app code, check the Private dependencies section.

You can/should always try to install your dependencies locally as a sanity check for your requirements.txt file. Be mindful with pip repos requiring SSH keys as you may have those by default accessible via your account configs.