In a Go application run as a k8s Job, I am attempting to assume a serviceaccount <-> IAM Role when trying to GetObject from an S3 bucket. I attach a serviceAccountName field to the Job spec in the yaml file, and I attach the AWS IAM to the service account. I have verified that the role has access to the S3 bucket in question.
The serviceaccount is created and seems to have the correct annotations:
Name: identity
Namespace: identity
Labels: app=identity
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<acct-id>:role/identity-s3-role-dev
Image pull secrets: <none>
Mountable secrets: identity-token-bf2wz
Tokens: identity-token-bf2wz
Events: <none>
In my kubectl describe pod <name> , I see in the environment section:
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::<id>:role/identity-s3-role-dev
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7rx85 (ro)
which implies that the pod has assumed the correct role (identity-s3-role-dev), and that the token file is mounted and all that.
But, that IAM role is not being used to talk to S3. When I execute my code, I get an error message:
panic: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 4TFT6G0T08547RBQ, HostID: r2+1NduQf+v4F1gL/AlERLrNtJR1jeI9mIL7oKoWUFLvuY7lyq+Ug9ElK3YHvl5h8rsbgq/uRAw=, api error AuthorizationHeaderMalformed: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.
No authentication at all appears to be attached to the PutObject request.
As for the Go code, I'm using the AWS SDK v2 with these imports:
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
to load the config here:
localconfig := load_config() // this loads my configuration with viper
cfg, err := awsconfig.LoadDefaultConfig(context.TODO())
if err != nil {
return nil, err
}
client = s3.NewFromConfig(cfg)
input := &s3.PutObjectInput{
Body: manager.ReadSeekCloser(bytes.NewReader(b)), // b is the payload
Bucket: &localconfig.s3bucket,
Key: &localconfig.s3key,
}
_, err = client.PutObject(context.TODO(), input)
The one thing I can think of that I am not sure is that I do not specify to use any sort of profile when loading configuration. But the AWS SDK docs seem to imply that if my pod has access to AWS creds through a service account (which I think I demonstrated is true above) they will be auto-loaded by the LoadDefaultConfig functionality.
So, what steps am I missing to get my credentials for the indicated service account and IAM role attached to my S3 put request?
Edit, I have been working on this for about a day now, and this is what I have discovered.
In order to get header information attached so that we no longer get that 400 error, you add in the configuration of the go code:
cfg, err := awsconfig.LoadDefaultConfig(
context.TODO(),
awsconfig.WithSharedConfigProfile("anything"),
)
if err != nil {
return nil, err
}
This changes the error message to:
operation error S3: GetObject, https response error StatusCode: 403, RequestID: 3H40VPYKAF64E0ZY, HostID: <long-id>, api error AccessDenied: Access Denied
The long as the short of it is, I am now attempting to access using the role of the associated nodes: arn:aws:iam::<acct-id>:role/node_group01-eks-node-group-20220420182829219400000001. This role does not have the required access, I want arn:aws:iam::<acct-id>:role/identity-s3-role-dev. Since I don't have access, I am getting a 403 now, which is expected.
So it seems like the problem might be on the Go SDK side, where I am not picking up the correct credentials.
I do note that it does not appear to matter what I put in with "anything" string, I always appear to pull the same node role credentials (which must be the default or something) and that gives me the 403 error.
