How to use a Python library with no pip install on AWS Elastic Beanstalk or Google App Engine?

Question

I have a Flask web service using a Python library PyDDE, which is distributed on PyPI but somehow can't be pip installed. I can install this library on my local environment using pip install git+https://github.com/hensing/PyDDE.git or clone and install setup.py.

But when I tried to upload and build the project on either AWS Elastic Beanstalk or Google App Engine Flexible Environment, this becomes an issue that they can't find a matching distribution of this library.

Is there any workaround to solve this issue?

Here is the error message from GAE Flexible:

Step #1:   Could not find a version that satisfies the requirement PyDDE (from -r requirements.txt (li
ne 7)) (from versions: )
Step #1: No matching distribution found for PyDDE (from -r requirements.txt (line 7))
Step #1: The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
Finished Step #1
ERROR
ERROR: build step "gcr.io/cloud-builders/docker@sha256:331786b295647a4560ed004a46761ee91409e754df7ffa7
54a67000bd020729a" failed: exit status 1
Step #1:
------------------------------------------------------------------------------------------------------

ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at https://console.cloud.google.com/gcr/buil
ds/6ace1019-8bf5-434b-b916-5c6cae84de9f?project=project-190120 Failure status: UNKNOWN: Err
or Response: [2] Build failed; check build logs for details

And here is my requirements.txt:

Flask==0.12.2
Werkzeug==0.12.2
gunicorn==19.7.1
pandas>0.20.0
numpy>1.13.0
scipy>0.19.0
PyDDE
google-api-python-client

UPDATE:

I added this line in requirements.txt:

-e git+git://github.com/hensing/PyDDE.git#egg=PyDDE

And it works for GAE.

Thank you @hansaplast !!

However, AWS Elastic Beanstalk doesn't accept this requirements.txt. It throws this error:

INFO: Created CloudWatch alarm named: awseb-e-3tu2ajdxug-stack-AWSEBCloudwatchAlarmLow-8NOPISSRAAEH
ERROR: Your requirements.txt is invalid. Snapshot your logs for details.
ERROR: [Instance: i-03e92fa3c58b6e010] Command failed on instance. Return code: 1 Output: (TRUNCATED).
..)
  File "/usr/lib64/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 2.
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
ERROR: Create environment operation is complete, but with errors. For more information, see troubleshooting documentation.

can you also put the content of `requirements.txt` into your question? — hansaplast, Dec 25 '17 at 23:15
*if* you use requirements.txt, can you try adding `-e git://https://github.com/hensing/PyDDE.git` into `requirements.txt` as [proposed by this answer](https://stackoverflow.com/a/16584935/119861)? ` — hansaplast, Dec 25 '17 at 23:16

score 2 · Accepted Answer · answered Apr 15 '18 at 19:12

I was able to fix a similar issue in the following way:

Create a config file to install git, because that doesn't come pre-installed on EB:

Path: root/.ebextensions
File: conf.config (name doesn't matter, extension needs to be config)

Add the following to the conf.config file:

packages:
  yum:
    git: []

Add the following to your requirements.txt:

-e git://github.com/hensing/PyDDE.git#egg=PyDDE

Re-run eb deploy

Links: - https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environment-configuration-methods-during.html#configuration-options-during-ebcli-ebextensions

score 1 · Answer 2 · edited Dec 25 '17 at 23:26

1

Create a folder in your project folder, e.g lib
Install your wanted library into that folder using pip install -t <dir>
Create a file named appengine_config.py in the same folder as your app.yaml file and edit it as follows:
```
from google.appengine.ext import vendor 
vendor.add('lib') # Folder name
```
Deploy

edited Dec 25 '17 at 23:26

hansaplast

11,007
2
61
75

answered Dec 25 '17 at 23:19

Evya

2,325
3
11
22

Yes, I did follow this procedure. But still it builds all libraries in the Docker container based on `requirements.txt` while deploying. – Chris Yao Dec 25 '17 at 23:42
@EvyatarMeged This applies to the standard environment only, not to the flexible one. – Dan Cornilescu Dec 26 '17 at 06:58

hansaplast · Answer 3 · 2017-12-30T16:14:18.013

1

Instead of PyDDE, add this to your requirements.txt:

-e git://https://github.com/hensing/PyDDE.git#egg=PyDDE

AWS Elastic Beanstalk doesn't accept this requirements.txt

It seems you have put git+git://github.com/hensing/PyDDE.git#egg=PyDDE into your requirements.txt. Remove the git+ and the #egg=PyDDE from the file (as put above) and then it should work.

edited Dec 30 '17 at 16:14

answered Dec 26 '17 at 11:38

hansaplast

11,007
2
61
75

It seems like GAE requires this part `#egg=PyDDE` otherwise it is invalid. – Chris Yao Dec 26 '17 at 15:41

How to use a Python library with no pip install on AWS Elastic Beanstalk or Google App Engine?

3 Answers3