How to authorize root to run gsutil?

Question

backup_to_gcs.sh is a backup script that uses gsutil rsync to copy files to Google Cloud Storage.

The backup script runs when called by user wolfv (who installed the google-cloud-sdk):

$ /home/wolfv/scripts/backup_to_gcs/backup_to_gcs.sh
backup_to_gcs.sh in progress ...
backup_to_gcs.sh pass

So far so good. But root needs to run gsutil so that the backup script can be called from Anacron. So lets have root run the script.

It's safe for root and wolfv to share credentials because it's the same person.

How can root run gsutil?

I am running gsutil 4.27 on Linux.

UPDATE_1

Thanks to mhouglum, root can now call the backup script. Just needed to set some parameters in root's .bashrc. Details follow.

Two lines from "gsutil version -l" are different for user wolfv and root:

using cloud sdk
pass cloud sdk credentials to gsutil

The backup script runs when called by user wolfv. Here is wolfv data (BOTO_CONFIG and BOTO_PATH are empty):

$ gsutil version -l
gsutil version: 4.27
checksum: 522455e2d24593ff3a2d3d237eefde57 (OK)
boto version: 2.47.0
python version: 2.7.13 (default, Jun 26 2017, 10:20:05) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]
OS: Linux 4.11.11-300.fc26.x86_64
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /home/wolfv/.boto, /home/wolfv/.config/gcloud/legacy_credentials/REDACTED_EMAIL@gmail.com/.boto
gsutil path: /home/wolfv/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
$ echo $BOTO_CONFIG

$ echo $BOTO_PATH

$ echo $PATH
/home/wolfv/google-cloud-sdk/bin:/home/wolfv/google-cloud-sdk/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin

This is user wolfv .bashrc:

$ cat .bashrc
# .bashrc

# after changing .bashrc, use source command to reload your .bashrc file
#   $ source "$HOME/.bashrc"
#   $ source ~/.bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# User specific aliases and functions
export PATH=${PATH}:$HOME/scripts

# The next line updates PATH for the Google Cloud SDK.
if [ -f '/home/wolfv/google-cloud-sdk/path.bash.inc' ]; then source '/home/wolfv/google-cloud-sdk/path.bash.inc'; fi

# The next line enables shell command completion for gcloud.
if [ -f '/home/wolfv/google-cloud-sdk/completion.bash.inc' ]; then source '/home/wolfv/google-cloud-sdk/completion.bash.inc'; fi

The backup script also runs when called by root. Here is root data:

$ sudo -s
# gsutil version -l
gsutil version: 4.27
checksum: 522455e2d24593ff3a2d3d237eefde57 (OK)
boto version: 2.47.0
python version: 2.7.13 (default, Jun 26 2017, 10:20:05) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]
OS: Linux 4.11.11-300.fc26.x86_64
multiprocessing available: True
using cloud sdk: False
pass cloud sdk credentials to gsutil: False
config path(s): /home/wolfv/.boto, /home/wolfv/.config/gcloud/legacy_credentials/REDACTED_EMAIL@gmail.com/.boto
gsutil path: /home/wolfv/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
[0 root@localhost ~]
# echo $BOTO_CONFIG

[0 root@localhost ~]
# echo $BOTO_PATH
/home/wolfv/.boto:/home/wolfv/.config/gcloud/legacy_credentials/REDACTED_EMAIL@gmail.com/.boto
[0 root@localhost ~]
# echo $PATH
/usr/lib64/ccache:/sbin:/bin:/usr/sbin:/usr/bin:/home/wolfv/google-cloud-sdk/platform/gsutil:/home/wolfv/google-cloud-sdk/bin:/home/wolfv/google-cloud-sdk/platform/gsutil:/home/wolfv/google-cloud-sdk/bin

This is the root .bashrc:

# cat .bashrc
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# added by wolf so anacron can call backup script
export PATH=${PATH}:/home/wolfv/google-cloud-sdk/platform/gsutil:/home/wolfv/google-cloud-sdk/bin
export BOTO_PATH=/home/wolfv/.boto:/home/wolfv/.config/gcloud/legacy_credentials/REDACTED_EMAIL@gmail.com/.boto

# The next line updates PATH for the Google Cloud SDK.
#if [ -f '/home/wolfv/google-cloud-sdk/path.bash.inc' ]; then source '/home/wolfv/google-cloud-sdk/path.bash.inc'; fi

# The next line enables shell command completion for gcloud.
#if [ -f '/home/wolfv/google-cloud-sdk/completion.bash.inc' ]; then source '/home/wolfv/google-cloud-sdk/completion.bash.inc'; fi

UPDATE_2

Both root on terminal and anacron can run the backup script.
When root runs the script, files are updated on GCS; good.
When anacron runs the script, it returns error code 127; bad.

The behavior is repoduced with this simple script called ls.sh:

#!/bin/bash

LOG_FILE="/home/wolfv/scripts/test/ls.log"

echo "I am $(whoami). date=$(date)" >> "$LOG_FILE"

gsutil ls >> "$LOG_FILE"
errorCode=$?

echo "errorCode=$errorCode" >> "$LOG_FILE"

root calls script from terminal:

# /home/wolfv/scripts/test/ls.sh

Result in ls.log:

I am root. date=Mon Aug 14 05:21:23 MDT 2017
gs://wolfv/
gs://wolfv2/
gs://wolfvtest/
errorCode=0

anacron calls script:

# anacron -n -f ls_id

Result in ls.log:

I am root. date=Mon Aug 14 05:21:38 MDT 2017
errorCode=127

What is causing error code 127?

UPDATE_3

I repeated tests from UPDATE_2, but using cron instead of anacron. Results are similar with cron getting the same errorCode=127.

wolfv calls script from terminal:

$ /home/wolfv/scripts/test/ls.sh

Result in ls.log:

I am wolfv. date=Fri Aug 18 20:58:57 MDT 2017
gs://wolfv/
gs://wolfv2/
gs://wolfvtest/
errorCode=0

cron calls script:

$ crontab -l
#test gsutil on cron
*/10 * * * * /home/wolfv/scripts/test/ls.sh

Result in ls.log:

I am wolfv. date=Fri Aug 18 21:20:01 MDT 2017
errorCode=127

The gsutil script runs fine from terminal, but not from cron/anacron.
What could be causing cron and anacron to return error code 127?

That gsutil in your PATH looks like the executable and not the directory. Try removing the second gsutil? — Brandon Yarbrough, Aug 06 '17 at 04:36
@ Brandon Yarbrough, Thanks for catching the second gsutil in PATH. I updated the question. Still need to figure out how to authenticate root. — wolfv, Aug 06 '17 at 05:19

mhouglum · Accepted Answer · 2017-08-13T00:46:59.880

2

Once you've run sudo -s, check that everything is set to the values you expect:

Does echo $BOTO_CONFIG show the correct boto file? If not, maybe root's .bashrc isn't being sourced?
Does echo $PATH include the gsutil directory you added?
When you run gsutil version -l, you should see:
- The config path(s) label, followed by /home/wolfv/.boto
- The gsutil path label, followed by /home/wolfv/google-cloud-sdk/platform/gsutil/gsutil. If this isn't the case, there's probably a gsutil executable that shows up in your PATH before the gsutil directory you appended, and you might want to try prepending the gsutil directory to your PATH so that it's found first, e.g. export PATH=/home/wolfv/google-cloud-sdk/platform/gsutil:${PATH}.

If everything there looks fine as root, you might run gsutil version -l both as yourself and as root and compare the output, letting you see what's configured differently between those environments.

EDIT:

The TL;DR of the comments below is: if you're trying to load more than 1 boto config file, you should unset BOTO_CONFIG (which only expects 1 file path), and instead use the BOTO_PATH environment variable, which allows specifying multiple file paths delimited by whatever os.pathsep evaluates to on your system (: on Linux, ; on Windows).

edited Aug 13 '17 at 00:46

answered Aug 07 '17 at 09:21

mhouglum

2,468
13
21

Thanks for the detailed trouble shooting guidelines. But I still couldn't get the root to run the backup script. I added **More information** to the question. – wolfv Aug 08 '17 at 15:51
Boto expects BOTO_CONFIG to contain a string containing exactly 1 file path; your BOTO_CONFIG variable includes multiple files. If you want multiple boto configs to be loaded, you should unset BOTO_CONFIG and instead set those colon-delimited filepaths as the value for BOTO_PATH. – mhouglum Aug 10 '17 at 18:26
I changed BOTO_CONFIG to contain one path. Still get the same error though. I updated and added .bashrc files to the question's **More information** section. – wolfv Aug 10 '17 at 23:12
OK, I did that, now root config path(s): no config found. And still get the same error. I updated the question's **More information** section to reflect the changes. – wolfv Aug 12 '17 at 02:41
root can now call the backup script! Your last comment was correct. Please ignore by previous comment. Thanks for all your help mhouglum. – wolfv Aug 12 '17 at 19:05
1

Glad to help! Boto's implicit behaviors are convenient... until you want to venture outside their intended basic use cases, and then they're just confusing :) – mhouglum Aug 13 '17 at 00:38
Deleted a comment above that had your email address in it, and I also redacted your email address from the question's text. For those confused, here's what my deleted comment said: In your first example (not as root), it shows config path(s) loading two boto files. Your credentials are probably coming from ~/.boto. To duplicate the exact boto/auth setup you have as non-root, for root you should set BOTO_PATH to /home/wolfv/.boto:/home/wolfv/.config/gcloud/legacy_credenti‌als/REDACTED_EMAIL@gmail.‌com/.boto and clear BOTO_CONFIG – mhouglum Aug 13 '17 at 00:51
I added **UPDATE_2** to the question. And thanks for the redacting :) – wolfv Aug 14 '17 at 11:37

How to authorize root to run gsutil?

1 Answers1

Linked