Botocore fails to read credentials when run as daemon.service

Question

I had my script running smoothly from command line, however, when I start it as a systemd.service, I get the following error:

iot_local.service - My iot_local Service
   Loaded: loaded (/lib/systemd/system/iot_local.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2018-04-01 23:06:45 UTC; 5s ago
  Process: 2436 ExecStart=/usr/bin/python /home/ubuntu/myTemp/iot_local.py (code=exited, status=1/FAILURE)
 Main PID: 2436 (code=exited, status=1/FAILURE)

Apr 01 23:06:45 ip-172-31-29-45 python[2436]:   File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 358, in resolve
Apr 01 23:06:45 ip-172-31-29-45 python[2436]:     service_name, region_name)
Apr 01 23:06:45 ip-172-31-29-45 python[2436]:   File "/usr/local/lib/python2.7/dist-packages/botocore/regions.py", line 122, in construct_endpoint
Apr 01 23:06:45 ip-172-31-29-45 python[2436]:     partition, service_name, region_name)
Apr 01 23:06:45 ip-172-31-29-45 python[2436]:   File "/usr/local/lib/python2.7/dist-packages/botocore/regions.py", line 135, in _endpoint_for_partition
Apr 01 23:06:45 ip-172-31-29-45 python[2436]:     raise NoRegionError()
Apr 01 23:06:45 ip-172-31-29-45 python[2436]: botocore.exceptions.NoRegionError: You must specify a region.
Apr 01 23:06:45 ip-172-31-29-45 systemd[1]: iot_local.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 23:06:45 ip-172-31-29-45 systemd[1]: iot_local.service: Unit entered failed state.
Apr 01 23:06:45 ip-172-31-29-45 systemd[1]: iot_local.service: Failed with result 'exit-code'.

it seems to fail on this line:

DB=boto3.resourse('dynamodb')

If I add the region as an argument, the script still fails later because can not find credentials. So, when I provide region, id and a key as argument, everything works:

boto3.resource('dynamodb', region_name='us-west-2', aws_access_key_id=ACCESS_ID, aws_secret_access_key=ACCESS_KEY)

The obvious problem is that when this script is run as service, it fails to obtain the info from the ~/.aws/config and ~/.aws/credentials, which I made sure to contain all the necessary information by running aws configure as mentioned here.

[default]
aws_access_key_id=XXXXXXXXXXXXXX
aws_secret_access_key=YYYYYYYYYYYYYYYYYYYYYYYYYYY

I also tried this:

export AWS_CONFIG_FILE="/home/ubuntu/.aws/config"

and this

sudo chown root:root ~/.aws

but it did not help. Any ideas why .service does not "see" the credentials files?

When running it as a service, have you explicitly set it up to run as the ubuntu user? Otherwise the 'home' directory is no longer /home/ubuntu so ~/.aws/credentials won't be loaded from the right place. — jarmod, Apr 01 '18 at 23:55
You could research the systemd documentation. Here’s an example: https://askubuntu.com/questions/676007/how-do-i-make-my-systemd-service-run-via-specific-user-and-start-on-boot/676022 Also, you should revert your chown command - you don’t want the ubuntu user’s ~/.aws folder to be owned by root:root. — jarmod, Apr 02 '18 at 01:12
@jarmod Nice, I set `User=ubuntu` in `[Service]`, and this thing works now. You should post it as an answer. — Nazar, Apr 06 '18 at 22:12

score 1 · Answer 1 · answered Nov 22 '18 at 19:05

The answer is much simpler - the environment variables arent getting loaded, in your case the aws credentials.

Create a File sudo vi /lib/systemd/system/iot_local.service With the below given content.

[Unit]
Description=Spideren MQ RPC service

[Service]
WorkingDirectory=/opt/cocoon_predev/
Environment="AWS_DEFAULT_REGION=us-xxxx"
Environment="AWS_ACCESS_KEY_ID=xxxxxxxxx"
Environment="AWS_SECRET_ACCESS_KEY=xxxxxxxxxx"
Environment="LANG=en_US.UTF-8"
Environment="PYTHONIOENCODING=utf-8"
ExecStart=/usr/bin/python2 /home/myproject/iotservice.py
User=myusername
Restart=on-failure
RestartSec=90s

And then finally run the below commands to activate the service.

sudo systemctl daemon-reload
sudo systemctl start iot_local.service 
sudo systemctl status iot_local.service

To check Errors in daemon startup - Run the following command

journalctl -u spiderenrpc.service

I have done this on Ubuntu - so check on your systemd formats specific to your os.

score 1 · Answer 2 · answered Dec 11 '20 at 07:31

I had a similar issue with Fluent Bit AWS Firehose plugin failed to read the AWS credentials, when its run as systemd service.

When manually invoking the Fluent Bit from command-line, the credentials were loaded.

The credentials was indeed stored under the user root, and the service was running as root, by default.

To fix the issue, I had to explicitly specify the user as root in the systemd unit:

[Service]
User=root

and run systemctl daemon-reload

score 0 · Accepted Answer · answered Apr 06 '18 at 23:34

When systemd runs your script as a service, the script is no longer being run by the ubuntu user so the home directory is no longer /home/ubuntu. That means that ~/.aws/credentials no longer refers to /home/ubuntu/.aws/credentials and your script is therefore trying to load credentials from the wrong place (probably /root/.aws/credentials).

You can configure systemd to run your script as a specific user. Add User=ubuntu in the [Service] section.

Botocore fails to read credentials when run as daemon.service

3 Answers3