1

I have a python script that does calculations on google compute engine instances. The code works fine in terms of doing the calculations, but at certain points in the code it needs to add/delete files from a cloud storage bucket and I do this using gsutil. This works well when run from my local computer, but isn't working when the same code is run from a google cloud instance. By "not working" an error message is reported at the offending line, but my code carries on running and just ignores the steps that involve gsutil.

My understanding from Google's documentation is that gcloud instances boot with the "gsutil" utility already installed. My instances boot running a script like this (where is my actual google username):

#! /bin/bash

sudo apt-get update
sudo apt-get -yq install python-pip
sudo pip install --upgrade google-cloud
sudo pip install --upgrade google-cloud-storage
sudo pip install --upgrade google-api-python-client
sudo pip install --upgrade google-auth-httplib2

mkdir -p /home/<xxxx>/code
mkdir -p /home/<xxxx>/rawdata
mkdir -p /home/<xxxx>/processeddata
sudo chown -R <xxxx> /home/<xxxx>

gsutil cp gs://<codestorebucket>/worker-python-code/* /home/<xxxx>/code/
gsutil -m cp gs://<rawdatabucket>/* /home/<xxxx>/rawdata/

I dont run my code from the boot script yet as I want to "SSH" into the instance and run it myself from the command line while I am still developing. When I SHH into the instance the directories have all been created and all of the code and raw datafiles have been copied. I can run my ".py" file and it runs, but there are lines which use the python command:

subprocess.call('gsutil -q rm gs://<mybuckname>/<myfilename>', shell=True)

This generates an error which reads:

ERROR: (gsutil) Failed to create the default configuration. Ensure your have the correct permissions on: [/home/<xxxx>/.config/gc
loud/configurations].
  Could not create directory [/home/<xxxx>/.config/gcloud/configurations]: Permission denied.

If it provides any clues, in the "daemon.log" file there an error line which reads:

chown: invalid user: ‘<xxxxx>’

which is reported when the sudo chown... command line runs.

The instances have full access to all APIs. If I run

whoami

The response is "xxxxx". If I run 

echo $UID

The response is 1000.

I am a Linux novice, as I have only "learnt" about it through needing to do stuff on google instances. There is a link here where a user appears to have a similar problem. He fixes it using a sudo chown type command line, but when I run an equivalent command I am told that it "cannot access '/home/paulgarlick07/.config/': No such file or directory"

I'm really confused, and any help would be very much appreciated. If any additional info is required to help resolve this please let me know!

Maxim
  • 4,075
  • 1
  • 14
  • 23
Paul
  • 528
  • 5
  • 17
  • Could you share the minimal Python code needed to reproduce the issue? – Maxim Dec 02 '18 at 20:50
  • Hi maxim. The minimum steps to reproduce this issue are: 1) create a bucket called tester123 and add a blank test.csv file to it 2) create a demo.py file that contains only the lines "import subprocess" and "subprocess.call('gsutil -q rm gs://tester/test.csv', shell=True)". 3) Create a new instance without any start-up script. 4) SSH into the instance, copy demo.py to somewhere. 5) Run the python file using "python demo.py". Error message is now "AccessDeniedException: 403 Insufficient OAuth2 Scope....", even though copying data from the bucket just worked fine! – Paul Dec 03 '18 at 03:55
  • I recommend that [you report](https://cloud.google.com/support/docs/issue-trackers) this behavior to our product team to help investigate this issue further. Google will make sure to assist you with it. – Mahmoud Sharif Dec 04 '18 at 14:38

1 Answers1

3

gsutil is not a program. It is a script. Therefore you need to execute a shell with gsutil as a command line argument. You will need to pass the full pathname for gsutil which might be different on your system.

subprocess.call('/bin/sh /usr/bin/gsutil -q rm gs://<mybuckname>/<myfilename>', shell=True)

If you are running gsutil from a service, then you will need to ensure that the user that the service is running under has gsutil setup. gsutil stores its configuration files based from the home directory of the user that it is executing under.

John Hanley
  • 74,467
  • 6
  • 95
  • 159