2

I want to create a multi-serivce app engine app, using the first diagram shown on this page. https://cloud.google.com/appengine/docs/standard/python/configuration-files#an_example

I want to use third party libraries so I used have a lib folder under the root directory, one yaml under the root directory. Then I want one microservice called predict. So I created a folder called predict under the root directory as well, then under this folder, I want to write py files using the packages in the lib as well.

What I'm doing is like this:

import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
vendor.add(os.path.join(parentdir, 'lib'))

Which didn't work for me as the error says:

ValueError: virtualenv: cannot access /base/data/home/apps/f~project-name/lib: No such virtualenv or site directory

Obviously it didn't work for me, what's the right way to do this?

Update: Dan's solution worked for me! So when I'm deploying my microservice predict, I need to go inside predict directory and deploy it. I think that's why it can't find lib. By symlink the lib library using bash. 

ln -s ../lib/ ./lib

I solved this problem.

Elona Mishmika
  • 480
  • 2
  • 5
  • 21

2 Answers2

4

The problem you're facing is that each service can only access files in its own service directory, which is the directory where its app.yaml file is located, they can't access the app's root directory.

My approach (all my services use standard environment):

  • created a lib dir in the top app dir and installed in it each package that I want shared across multiple services
  • each service has its own subdir in the app's dir and a lib directory in it where:
    • I installed libraries needed only by that service (if any)
    • I symlinked the needed shared libraries from the top lib dir

This is my app dir structure:

app_dir/
app_dir/dispatch.yaml
app_dir/cron.yaml
app_dir/index.yaml
app_dir/queue.yaml
app_dir/lib/
app_dir/lib/shared_lib_1/
app_dir/lib/shared_lib_2/
app_dir/lib/shared_lib_3/
app_dir/service1/
app_dir/service1/app.yaml
app_dir/service1/lib/shared_lib_1 -> ../../lib/shared_lib_1
app_dir/service1/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service1/lib/service1_lib_1
app_dir/service2/
app_dir/service2/app.yaml
app_dir/service2/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service2/lib/shared_lib_3 -> ../../lib/shared_lib_3
app_dir/service2/lib/service2_lib_1

No fumbling with the lib path is required, all I have in each service is

vendor.add('lib')

See related:

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • 1
    Could you just symlink the whole lib dir? Not sure if that would make much of a difference in resource usage. – new name Jul 19 '18 at 13:01
  • Symlinking the entire `lib` dir is possible (I actually started that way) but only if if all services have the same needs. It does matter in the app file quota - at deployment time the symlinks are replaced by their content, so the number of files in a symlinked shared lib will be multiplied by the number of services sharing it. If pulling heavy libs the impact could be significant. – Dan Cornilescu Jul 19 '18 at 13:08
0

Check all the directories in the path and the final file to make sure they all exist first before code runs. If they don't, prepend the code you showed here with code to create them...

Seyi Shoboyejo
  • 489
  • 4
  • 11