Google App Engine - Big Query - Python can't find libraries google.cloud

Question

I'm building an App Engine project using Python on Ubuntu Linux 16.4 using Eclipse and PyDev.

I'm using the Flask Framework which works fine, what I want to do is connect to Big Query and that's where I am having trouble.

So I have used:

pip install --upgrade google-cloud-bigquery

To install the Big Query Library and I can see the packages in /usr/local/lib/python2.7/dist-packages

For some reason Eclipse doesn't seem to think the library is registered, I can live with that but when I run my project dev_appserver.py App Engine can't find the library:

I have this in my Python PyDev:

It is weird as if I add from lib.google.cloud import bigquery PyDev resolves, the dist-packages is in the PyDev path so I don't get why it doesn't pick it up.

You'll also notice I've commented out the apiclient.discovery and oauth2client.client, I was going to try another way but App Engine complained it couldn't find the SignedJwtAssertionCredentials library so I am sure this is all linked and caused by the same problem.

In my appengine_config.py I have:

Answer 1

(Jun 2021): There seems to be some confusion in the OP and various responses, so let's make it super clear for Python 2 users:

1. Add google-cloud-bigquery to requirements.txt
2. Create appengine_config.py to match that of Step 4 on this page (which you seem to have)
3. Delete the lines with apiclient or googleapiclient and oauth2client (wrong client libraries) from main.py
4. Run pip2 install -t lib -r requirements.txt (or just pip if you don't have Python 3 installed)
5. NOTE: Python 3 users don't have to play these games... step 1 above is all you need to do... no appengine_config.py file, no running pip install, and no lib folder.

For a broader audience, see my answer to this more general question.

Answer 2

If you want to use 3rd party libraries that are not included this list, then you'll have to add them manually.

In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:

hello
├── libs
│   └── bs4 
├── hello.py 
└── app.yaml

then in your hello.py you have to put these two lines in the beginning of the file:

import sys
sys.path.insert(0, 'libs')

After doing that you'll be able to use any 3rd party library that you're going to put in that libs directory. For example:

from bs4 import BeautifulSoup

Answer 3

While @Bravin is on the right path the recommended procedure for vendoring in 3rd party libraries is a bit different:

1. Create a directory to store your third-party libraries, such as lib/.

   mkdir lib
   

2. Use pip (version 6 or later) with the -t <directory> flag to copy the libraries into the folder you created in the previous step. For example:

    pip install -t lib/ <library_name>
   

   Using Homebrew Python on Mac OS X?

3. Create a file named appengine_config.py in the same folder as your app.yaml file.

4. Edit the appengine_config.py file and provide your library directory to the vendor.add() method.

   # appengine_config.py
   from google.appengine.ext import vendor
   
   # Add any libraries install in the "lib" folder.
   vendor.add('lib')