0

I'm trying to implement a cron job on GAE which will look into a csv file in a cloud storage bucket and load that into a BigQuery table. The code works fine on my mac (macOS Sierra, 10.12.6).

However, when I am deploying the app on GAE through gcloud app deploy app.yaml and configuring the cron job using gcloud app deploy cron.yaml, the cron job fails at: from google.cloud import bigquery . I have tried this solution to no avail. Any idea what I am doing wrong?

Below is the full error message from cron log. 

Traceback (most recent call last): (/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/cgi.py:122)
  File "/base/data/home/apps/f~gcp-101-181605/20171002t142145.404509292894601117/uploadBQ.py", line 17, in <module>
    from  google.cloud import bigquery
  File "/base/data/home/apps/f~gcp-101-181605/20171002t142145.404509292894601117/lib/google/cloud/bigquery/__init__.py", line 32, in <module>
    from google.cloud.bigquery.client import Client
  File "/base/data/home/apps/f~gcp-101-181605/20171002t142145.404509292894601117/lib/google/cloud/bigquery/client.py", line 21, in <module>
    from google.cloud.bigquery.job import CopyJob
  File "/base/data/home/apps/f~gcp-101-181605/20171002t142145.404509292894601117/lib/google/cloud/bigquery/job.py", line 34, in <module>
    import google.cloud.future.polling
  File "/base/data/home/apps/f~gcp-101-181605/20171002t142145.404509292894601117/lib/google/cloud/future/polling.py", line 23, in <module>
    import tenacity
  File "/base/data/home/apps/f~gcp-101-181605/20171002t142145.404509292894601117/lib/tenacity/__init__.py", line 28, in <module>
    from monotonic import monotonic as now
  File "/base/data/home/apps/f~gcp-101-181605/20171002t142145.404509292894601117/lib/monotonic.py", line 41, in <module>
    import ctypes
  File "/base/data/home/runtimes/python27_experiment/python27_dist/lib/python2.7/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes

For reference, I am working in a virtual environment with python version 2.7.10. My app.yaml is:

runtime: python27
api_version: 1
threadsafe: false

handlers:
- url: /cronjobs
  script: uploadBQ.py
  login: admin

- url: /.*
  script: main.app

Content of cron.yaml:

cron:
- description: cron to BQ
  url: /cronjobs
  schedule: every 15 minutes from 10:20 to 11:20

Contents of requirement.txt (installed on lib folder through pip install -r requirements.txt -t lib):

google-api-python-client
google-cloud

Content of appengine_config.py

# appengine_config.py
from google.appengine.ext import vendor

# Add any libraries install in the "lib" folder.
vendor.add('lib')
Sal
  • 335
  • 2
  • 8
  • 1
    I can't reproduce this, but the stacktrace shows that `_ctypes` is being imported through a module named `tenacity`; this seems to be a recent change (and seems to have been reverted even more recently per github), so you could try try installing an older version say 0.26 or even 0.25 (current is 0.27; we use 0.22), `pip install -t lib google-cloud-bigquery==0.26` (I'd recommend completely reinstalling everything in `lib` first to ensure you get a clean install). – snakecharmerb Oct 02 '17 at 18:37
  • That did it. It was really a version issue as you said. I downgraded to version 0.19 and it is working like a charm. See https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2464 for detailed discussion. Thanks. – Sal Oct 02 '17 at 22:52

1 Answers1

0

From BigQuery Client Libraries (on the PYTHON tab):

Installing the client library

pip install --upgrade google-cloud-bigquery

Note that the instructions are for generic python apps, I just included it for the name of the library.

So you need to add google-cloud-bigquery to your requirements.txt file, re-run the requirements installation and the app deployment on GAE.

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • Not working. I am getting this error `ImportError: No module named cloud` coming from `from google.cloud import bigquery`. My new `requirements.txt` include only `google-cloud-bigquery`. On the GAE source, I can see the `/google/cloud` is present in the `lib` folder, but it still does not work. Is my path import fine: `sys.path.append(os.path.join(os.path.dirname(__file__), "lib"))`? – Sal Oct 02 '17 at 04:33