How to make S3FS use IMDS v2 when mounting s3 buckets from EC2 instance

Question

We are unable to mount a S3 bucket on an AWS EC2 instance that requires IMDS v2 (IMDS v1 works fine, though!). We're using s3fs-fuse v1.90 which is supposed to support IMDS v2 (on Ubuntu 18.04).

This is how we've been trying to mount our S3 bucket:

sudo s3fs <bucket> <mount-point> -o iam_role=auto,url=https://s3.eu-central-1.amazonaws.com,endpoint=eu-central-1,allow_other,uid=1000,gid=1000,mp_umask=007,use_cache=/tmp/s3foldercache,dbglevel=debug -f

However, we receive this error message:

2021-09-02T12:40:53.157Z [INF]       curl.cpp:LoadIAMRoleFromMetaData(2919): Get IAM Role name
2021-09-02T12:40:53.157Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-09-02T12:40:53.158Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/
2021-09-02T12:40:53.159Z [ERR] curl.cpp:RequestPerform(2394): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
 </head>
 <body>
  <h1>401 - Unauthorized</h1>
 </body>
</html>

2021-09-02T12:40:53.160Z [CRT] s3fs.cpp:s3fs_init(3395): could not load IAM role name from meta data.
2021-09-02T12:40:53.160Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors

Again, when we allow IMDS v1 on our EC2 instance, the mount command works fine.

How do we make sure s3fs will use IMDS v2?

EDIT: Many thanks to @Noah for contributing a fix for this to s3fs. I've pulled the latest commit but am seeing a new error message now. Apparently IMDSv2 token retrieval fails, regardless of whether I apply iam_role=auto or iam_role=some_profile (removing -f wouldn't change anything either):

2021-09-06T05:43:58.857Z [INF] s3fs.cpp:s3fs_init(3382): init v1.90(commit:9d1552a) with OpenSSL
2021-09-06T05:43:58.857Z [INF]       curl.cpp:LoadIAMRoleFromMetaData(3181): Get IAM Role name
2021-09-06T05:43:58.857Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-09-06T05:43:58.857Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/api/token
2021-09-06T05:43:58.889Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>417 - Expectation Failed</title>
 </head>
 <body>
  <h1>417 - Expectation Failed</h1>
 </body>
</html>
2021-09-06T05:43:58.890Z [ERR] curl.cpp:LoadIAMRoleFromMetaData(3212): AWS IMDSv2 token retrieval failed: -5
2021-09-06T05:43:58.890Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/
2021-09-06T05:43:58.892Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
 </head>
 <body>
  <h1>401 - Unauthorized</h1>
 </body>
</html>
2021-09-06T05:43:58.892Z [CRT] s3fs.cpp:s3fs_init(3395): could not load IAM role name from meta data.
2021-09-06T05:43:58.892Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors

score 1 · Accepted Answer · answered Sep 02 '21 at 16:18

Does it work if you specify the IAM role explicitly?

It looks like there may be an issue with retrieval of the IAM role information (needed with iam_role=auto) when tokens are required.

Additionally, when testing this, I had issues with when the -f option is used, so also try removing that.

Below are logs that show the proper token retrieval when mounting a bucket with the following command. Bucket access via the mount point works as expected.

$ sudo s3fs <REDACTED> /mnt -o iam_role=s3fs-testing,allow_other,uid=1000,gid=1000,mp_umask=007,use_cache=/tmp/s3foldercache,dbglevel=debug

Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: init v1.89(commit:unknown) with GnuTLS(gcrypt)
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: check services.
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: IAM Access Token refreshing...
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]:       [IAM role=s3fs-testing]
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: curl.cpp:RequestPerform(2254): connecting to URL http://169.254.169.254/latest/api/token
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]:       HTTP response code 200
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]:       Setting AWS IMDSv2 API token to <REDACTED>
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: curl.cpp:RequestPerform(2254): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/s3fs-testing
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]:       HTTP response code 200
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]:       IAM credential response = "{
                                          "Code" : "Success",
                                          "LastUpdated" : "2021-09-02T15:36:20Z",
                                          "Type" : "AWS-HMAC",
                                          "AccessKeyId" : "REDACTED",
                                          "SecretAccessKey" : "REDACTED",
                                          "Token" : "REDACTED",
                                          "Expiration" : "2021-09-02T22:10:48Z"
                                        }"
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: IAM Access Token refreshed

This should be fixed permanently upstream as of https://github.com/s3fs-fuse/s3fs-fuse/pull/1760 — Noah, Sep 03 '21 at 16:04
Thank you @Noah for even committing a fix for this to s3fs. I've pulled your changes but am seeing a new error message indicating that token retrieval fails. I updated my original post with the latest error message. — dovregubben, Sep 06 '21 at 06:29
I have no idea what could be triggering the "417 - Expectation Failed" response from IMDS. What instance type are you using? If you use curl, do you get a token? `$ curl -v -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" http://169.254.169.254/latest/api/token ; echo` — Noah, Sep 07 '21 at 16:38
I do receive a valid token when running that curl command. I believe this is another issue apart from the one I originally posted here, so I'll close this one assuming that your submission to s3fs has fixed the ```iam_role=auto``` problem (even though I haven't been able to validate it, yet). I opened a new topic for the 417 error msg: https://stackoverflow.com/questions/69104247/s3fs-how-to-mount-s3-buckets-on-aws-ec2-instances-behind-proxy-using-imds-v2 — dovregubben, Sep 08 '21 at 13:50
I could finally validate that ```iam_role=auto``` now also works for IMDS v2 ! — dovregubben, Sep 09 '21 at 08:28

How to make S3FS use IMDS v2 when mounting s3 buckets from EC2 instance

1 Answers1