9

I want to use two Python libraries (Google's Cloud Library, and their Cloud SDK) in a single application, but they have conflicting names (they both use google in their base import names and do not use relative imports internally). How can I use them in a single app?

Changing the library's code to use proper relative imports is not practical. Also, I know I can use virtualenv to access these libraries from separate python applications, but how do I access them from within the same python app?

Details of the Naming Conflict

Here are some of the details on the import. When I import a module from the Cloud Library (I run import google.cloud.datastore), there is an exception about another import within that library:

>>> import libs.google.cloud.datastore
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\[ProjectDIR]\libs\google\cloud\datastore\__init__.py", line 52, in <module>
    from google.cloud.datastore.batch import Batch
ImportError: No module named cloud.datastore.batch

The library is trying to do an absolute import, rather than a relative one. The reason that the Google Cloud Library cannot import google.cloud.datastore.batch is because google is already defined in the SDK, there is a naming conflict:

>>> print google.__path__
['C:\\Program Files (x86)\\Google\\Cloud SDK\\google-cloud-sdk\\platform\\google_appengine\\google']

Because the Cloud Library uses absolute imports, and the name google is already defined in the SDK, then the import fails.

Community
  • 1
  • 1
speedplane
  • 15,673
  • 16
  • 86
  • 138
  • The second link goes to the Google app engine SDK (providing code under the `google.appengine` package), the first is `google.cloud`. I don't see how the two have conflicting names. What issues are you experiencing? You can always import modules under a different name at any rate; `from google.cloud import datastore as gc_datastore` would bind the name `gc_datastore` in your current module; it doesn't matter what the original name was. – Martijn Pieters Dec 30 '16 at 16:21
  • Hi @MartijnPieters, the problem has to do with the internals of the library. The library at the first link does not use relative imports, so when you try to import it, you get naming conflicts with the second. See line 57 here: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/datastore/google/cloud/datastore/__init__.py#L57 – speedplane Dec 30 '16 at 17:29
  • So does the other actually include a `google.cloud.datastore` package? You haven't clearly linked to anything there. And I suspect that the Google App Engine SDK already *includes* the Cloud SDK as part of the installation, so you wouldn't need to install the other one too. – Martijn Pieters Dec 30 '16 at 17:31
  • @MartijnPieters, I added more detail. The problem is that both libraries define `google` and the cloud library uses absolute imports. And no, the SDK does not include the Cloud libraries. The Cloud Libraries are newer ways of accessing Cloud resources that use a REST interface. – speedplane Dec 30 '16 at 17:38
  • Why are you using `libs` in this path: `import libs.google.cloud.datastore`? Why isn't that a top-level installation instead? That's your *real* problem. – Martijn Pieters Dec 30 '16 at 17:58
  • And when I install the [SDK for app engine](https://cloud.google.com/appengine/docs/python/download) *no `google` namespace is included*. – Martijn Pieters Dec 30 '16 at 18:00
  • (1) I don't believe using `libs` in the path is the problem. Installing it top level wouldn't work because `import google` was already defined in the top level as part of the SDK. (2) After installing the app engine SDK, run the following command: `import google; print google.__path__[0]`. You should get a path to the Google SDK. – speedplane Dec 30 '16 at 18:52
  • I'd like to contradict Martijn Pieters here, the issue is with namespace packages. The GAE SDK google namespace isn't quite right, so some hacks are needed in appengine_config.py – bossylobster Dec 30 '16 at 19:16

1 Answers1

9

The google packages take care to register themselves as a namespace package. With a properly set up sys.path there is no conflict here.

You need to set up your library environment correctly. Add a appengine_config.py file in the root of your project with:

from google.appengine.ext import vendor

# Add any libraries installed in the "lib" folder.
vendor.add('lib')

This adds the lib subdirectory in the right location of sys.path. See the Installing a third-party library section in the Developing Python Apps on App Engine How-To.

From here on out imports of google.cloud just work:

$ ls -1d lib *.py *.yaml
app.yaml
appengine_config.py
lib
main.py
$ pip install -t lib google-cloud
# installing into the lib subdirectory
$ cat main.py
import google
from google.cloud import datastore
from google.appengine.api import memcache
import os.path

here = os.path.dirname(os.path.abspath(__file__))

def app(*args, **kwargs):
    return '''
google: {}<br />
google.cloud.datastore: {}<br />
google.appengine.api.memcache: {}'''.format(
        os.path.relpath(google.__file__, here),
        os.path.relpath(datastore.__file__, here),
        os.path.relpath(memcache.__file__, here))

and in the browser I am served:

google: ../google-cloud-sdk/platform/google_appengine/google/__init__.py
google.cloud.datastore: lib/google/cloud/datastore/__init__.pyc
google.appengine.api.memcache: ../google-cloud-sdk/platform/google_appengine/google/appengine/api/memcache/__init__.pyc
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Okay, this looks good, I'll give it a try. But can you also import the SDK libraries as well? In one file, could I do `import google.cloud as gcloud` and also `import google.appengine as appengine`? – speedplane Dec 30 '16 at 20:55
  • @speedplane: updated to add an SDK sample too. Yes you can. – Martijn Pieters Dec 30 '16 at 21:02
  • This is a bit of a nit, but the command `pip install -t lib google-cloud` adds a bunch of unnecessary stuff into the lib folder (e.g., egg-info, dist-info files). Any clean way to only install the libraries that will be used? Or should I just delete the unnecessary stuff after installing? – speedplane Dec 30 '16 at 21:14
  • @speedplane: I'd not worry about those, and keep things as simple as possible. Don't complicate the bootstrap procedure for a new developer. – Martijn Pieters Dec 30 '16 at 21:14
  • @speedplane: IIRC (it's been a year or 2 since I worked on GAE projects) you can add ignores to your yaml file instead so those egg-info directories are not uploaded, using the [`skip_files` option](https://cloud.google.com/appengine/docs/python/config/appref#skip_files). – Martijn Pieters Dec 30 '16 at 21:16