3

I have been trying to deploy a Cloud Function with some private dependencies (pyodbc) as I couldn't get it working thru requirements.txt. Please note, I don't want to use Docker here. So all I have built here is below files,

1. main.py
2. process.py ( this one use pyodbc to connect to teradata)
3. libs (folder)
   3.1 pyodbc-4.0.30.dist-info (package)
   3.2 pyodbc (python extension module)
   3.3 __init.py__ ( this is to make this folder as module)
4.requirements.txt

enter image description here I also updated process.py file to import pyodbc module as below,

import libs.pyodbc

Please note: I used GCP docs to install the pyodbc package and put in libs using https://cloud.google.com/functions/docs/writing/specifying-dependencies-python On top this, I am also requirements.txt to import as default.

But I am still getting module error as below. 

Error message: Code in file main.py can't be loaded.
Did you list all required modules in requirements.txt?
Detailed stack trace: Traceback (most recent call last):
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 305, in check_or_load_user_function
    _function_handler.load_user_function()
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 184, in load_user_function
    spec.loader.exec_module(main)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/user_code/main.py", line 9, in <module>
    from process import process
  File "/user_code/process.py", line 6, in <module>
    import libs.pyodbc
ModuleNotFoundError: No module named 'libs.pyodbc'

Any leads or help from here is really appreciated. All I am trying to achieve here is, Read CSV files from GCP bucket and process it thru dataframe which loads into teradata and generate output file back into another GCP bucket. I am trying to achieve all with Cloud Functions only. Thank you

Dustin Ingram
  • 20,502
  • 7
  • 59
  • 82
Sid
  • 153
  • 1
  • 12
  • Does [this](https://stackoverflow.com/questions/55278707/gcp-cannt-deploy-with-install-pyodbc) question help? It may avoid having to package locally, where I suspect you'll run into the same problem regardless (requiring ODBC header files). – djnz Feb 17 '20 at 02:12
  • No. I had it checked already and it was related to running app engine. Well I am trying to make it simple by just using cloud functions only. All I need here is to execute pyodbc in cloud functions environment without adding further complexity like docker. – Sid Feb 17 '20 at 03:05
  • "On top this, I am also `requirements.txt` to import as default." what does this mean? – Dustin Ingram Feb 17 '20 at 04:34
  • I meant, I also use requirements.txt here to specify additional dependencies that I haven't packaged alongside here. pyodbc is the only dependency that I wanted to import from the packaged code. – Sid Feb 17 '20 at 04:44
  • I would avoid pyodbc as it requires the odbc binaries - if this is to interact with teradata, just use native https://pypi.org/project/teradatasql/ – yan-hic Apr 12 '20 at 22:30

1 Answers1

3

The pyodbc project might be a bit of a special case here, because:

  1. The project requires some platform-specific code;
  2. They haven't published source distributions for their latest release (only built distributions).

Here's what I did to get this to work. Starting with an empty libs directory, first download the latest available source distribution:

$ pip download pyodbc --no-binary :all:

Make a directory for the module:

$ mkdir libs/pyodbc

Untar the source distribution into the module:

$ tar xf pyodbc-4.0.28.tar.gz -C libs/pyodbc

Then, in the function you can do:

import libs.pyodbc
Dustin Ingram
  • 20,502
  • 7
  • 59
  • 82
  • Shouldn't this directory require an empty _ _init.py_ _ file to turn this directory into a module here? – Sid Feb 17 '20 at 04:46