2

I have a groovy script that fetches data about all of my app sales with various companies and generates reports for me. This has been working well, but I've migrated to a new server and now I'm running into a problem.

To download Android app sales from Google Play, you have to use the gsutil command to copy the sales data from Google Storage. So, in my groovy script I execute that command. It works just fine on my old server and when I issue the same command in my shell on the new server it works.

It just doesn't work on the new server from within the groovy script. Here is the error I get:

$ ./fetch_googleplay_report.groovy 2017-01-29
Fetching 2017-01-29 for Android Google Play
Running command: /usr/bin/gsutil cp -r gs://pubsite_prod_rev_XXXXXXXXXXXXX/sales/salesreport_201701.zip .
gsutil command output: 
gsutil command error output: which: no python2 in ((null))
which: no python2.7 in ((null))
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 75, in <module>
    main()
  File "/usr/lib64/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 59, in main
    'platform/gsutil', 'gsutil', *args)
  File "/usr/lib64/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 45, in ExecutePythonTool
    execution_utils.ArgsForPythonTool(_FullPath(tool_dir, exec_name), *args))
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/core/execution_utils.py", line 170, in ArgsForPythonTool
    python_executable = kwargs.get('python') or GetPythonExecutable()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/core/execution_utils.py", line 59, in GetPythonExecutable
    raise ValueError('Could not find Python executable.')
ValueError: Could not find Python executable.

$ which python
/usr/bin/python
$ which python2
/usr/bin/python2

Python is definitely installed and in my path, but when gsutil is running when called by my groovy script, it can't seem to find it. I'm guessing from the stacktrace that when it says no python2 in ((null)) it means that it's path is null. I'm not sure though.

Here is the groovy code snippet that is executing the command:

def cmd = "/usr/bin/gsutil cp -r gs://pubsite_prod_rev_XXXXXXXXXXX/sales/$reportfilename .";

println("Running command: $cmd")

def out = new StringBuffer(), err = new StringBuffer()
def proc = cmd.execute([], incomingdir)
proc.consumeProcessOutput(out, err)
proc.waitForOrKill(5000)

println("gsutil command output: $out")
println("gsutil command error output: $err")

EDIT: If I move the gsutil command into a shell script wherein I specifically source my .bash_profile to set up my PATH and have the groovy script call that shell script instead, then the file gets downloaded just fine.

So, this does appear to be a problem with gsutil getting called from a groovy script. Any ideas?

Kenny Wyland
  • 20,844
  • 26
  • 117
  • 229
  • Is the CLOUDSDK_PYTHON variable set differently between your shell but not in your groovy script, by chance? If not, I think you are correct that the system path in your groovy script is different. This seems independent of gsutil/gcloud - you need to make sure python is on the path. – Travis Hobrla Jan 31 '17 at 23:53
  • python is definitely in my path from my shell (as shown by the `which` commands), but how do I control the path environment within the groovy script? I've set `CLOUDSDK_PYTHON` in my shell to point to `/usr/bin/python2`, but that environment variable I set in my shell is also not available, apparently, when executing the groovy script... but this all worked on my old server. I'm so confused. – Kenny Wyland Feb 01 '17 at 23:56
  • This is really a question about groovy environment variables (since "which" from within your script doesn't work). Perhaps http://stackoverflow.com/questions/21236268/access-to-build-environment-variables-from-a-groovy-script-in-a-jenkins-build-st can help? – Travis Hobrla Feb 02 '17 at 19:09
  • The answer on that other SO post just suggests passing in the missing environment variables as parameters, but my problem is that `gsutil` is the script that is trying to access the environment and it doesn't take those things as params. :( – Kenny Wyland Feb 02 '17 at 20:40
  • Ah, I see now. It seems the issue then is just how to pass the environment from groovy to Python. cmd.execute's first argument is the environment, so instead of passing an empty list, can you just include it as described at https://coderwall.com/p/nswp1q/calling-other-processes-from-groovy ? – Travis Hobrla Feb 03 '17 at 17:25
  • I'll give that a try! – Kenny Wyland Feb 03 '17 at 20:44

1 Answers1

1

In this case you should prefix the execution of your Google Cloud SDK tools with CLOUDSDK_PYTHON=/usr/bin/python, eg. 

CLOUDSDK_PYTHON=/usr/bin/python gsutil cp -r ...

This workaround is from hidekuro in github.com/GoogleCloudPlatform/gsutil/issues/402.

YaguraStation
  • 558
  • 2
  • 12