34

How to add third party python libraries in Google App Engine, which are not provided by Google? I am trying to use BeautifulSoup in Google App Engine and unable to do so. But my question is for any library I want to use in Google App Engine.

Lipis
  • 21,388
  • 20
  • 94
  • 121
Moazzam Khan
  • 3,130
  • 2
  • 20
  • 35
  • possible duplicate of [How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)](http://stackoverflow.com/questions/4863557/how-do-i-manage-third-party-python-libraries-with-google-app-engine-virtualenv) – Wernight Sep 17 '15 at 15:10
  • I just posted an updated answer to the OP question here: https://stackoverflow.com/a/67863189/305689 – wescpy Jun 06 '21 at 20:01

6 Answers6

58

Google has provided a documented way for included third-party libraries in your GAE project.

See the "Adding Third-party Packages to the Application" section of the Libraries in Python 2.7 docs.

If you want to include additional pure-python third-party packages, you can do so by setting up vendoring. Vendoring allows you to install packages to a subdirectory of your project and include them in your code. To use vendoring, create (or modify) appengine_config.py in the root of your project. 

from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')

And then just put all your libs' source code in your lib dir

> pip install beautifulsoup4 -t lib

So your project directory structure looks like this:

project
- lib
  - bs4
- your_code.py

This will allow your project's source files to import libs' packages/modules as if they were added to your PYTHON_PATH. For example:

# file: your_code.py
import bs4  # no need for 'from lib import bs4'
# do stuff with bs4...

You can also easily install everything from a requirements.txt file by doing the following command

> pip install -t lib -r requirements.txt
Jesse Webb
  • 43,135
  • 27
  • 106
  • 143
  • 2
    This answer seems to be better then the one that is accepted. Thanks Jesse! – Alexander Trakhimenok Oct 06 '15 at 14:17
  • 2
    this worked for me for a long time, but just began failing inexplicably. I am now getting 'virtualenv: cannot access lib: No such virtualenv or site directory' – bgenchel Jan 25 '16 at 23:17
  • 1
    @bgenchel @PrestonCrawford Were either of you able to resolve your problems? I have never seen that `virtualenv` error and this approach is still currently working for me. If you still haven't solved it, please post a new question and link it here and I will try to help. Make sure to add many details. – Jesse Webb Mar 30 '16 at 15:09
  • What happened for me, I think, was that I was trying to import from lib as a package, not understanding that vendor.add("lib") means you can access libraries inside as though they were on your python path. The error was thrown because after running the pip install on my requirements file, I had forgotten to put an __init__.py inside the folder. Thanks for offering!! – bgenchel Mar 30 '16 at 16:51
  • I was and I wasn't. I'm only getting the error through PyCharm right now. Trying to sort that out. – Preston Crawford Apr 28 '16 at 06:32
  • this is so frustating. not working for me, followed all steps but nope – Jasmeet Nov 22 '16 at 09:24
47

Actually I think this answer fits better here.

If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually.

In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:

hello
├── libs
│   └── bs4 
├── hello.py 
└── app.yaml

then in your hello.py you have to put these two lines in the beginning of the file:

import sys
sys.path.insert(0, 'libs')

After doing that you'll be able to use any 3rd party library that you're going to put in that libs directory.

For example:

from bs4 import BeautifulSoup
Community
  • 1
  • 1
Lipis
  • 21,388
  • 20
  • 94
  • 121
3

You simply copy the folder containing the library you want to use into your app engine project.

Then when you deploy it's uploaded with your application and is available for use.

EDIT: Jesse's answer is how I now do this. So do that!

Community
  • 1
  • 1
Paul Collingwood
  • 9,053
  • 3
  • 23
  • 36
1

The way it worked here is:

import sys
# sys.path.insert(0, 'libs') #"Old" way, not working for me.
sys.path.append(os.path.join(os.path.dirname(__file__), "libs")) # This works!

Then import normally:

from bs4 import BeautifulSoup
Rafael Vogel
  • 21
  • 1
0

Just put Beautifulsoup in the root of your project and upload it all

Fábio Diniz
  • 10,077
  • 3
  • 38
  • 45
0

pip install -t lib package_name

lib: the location for third party libraries

Then you are good to use this package like a normal library you use from ipython or terminal.

Vishvajit Pathak
  • 3,351
  • 1
  • 21
  • 16