Snowplow Collector on K8S does not use service account

Question

It seems that we cannot make the Snowplow container (snowplow/scala-stream-collector-kinesis) use the service account we provide. It always uses the shared-eks-node-role but not the provided service account. The config is set to default for both the accessKey as the secretKey.

This is the service account part we use:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: thijs-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123:role/thijs-eks-service-account-role-snowplow

And when I inspect the pod I can see the account:

AWS_ROLE_ARN:                 arn:aws:iam::123:role/thijs-eks-service-account-role-snowplow

The error then shows not the right account.

Exception in thread "main" com.amazonaws.services.kinesis.model.AmazonKinesisException: User: arn:aws:sts::123:assumed-role/shared-eks-node-role/i-123 is not authorized to perform: kinesis:DescribeStream on resource: arn:aws:kinesis:eu-west-1:123:stream/snowplow-good (Service: AmazonKinesis; Status Code: 400; Error Code: AccessDeniedException; Request ID: 123-123-123; Proxy: null)

Answer 1

The collector itself doesn't do any role swapping. It only cares to receive credentials via one of three methods:

• the default creds provider chain
• a specific IAM role
• environment variables.

The most popular deployment is on an EC2 instance, in which case the default EC2 role can be used to access other resources in the account.

It looks like when you are deploying it on EKS things are not as straightforward. The collector seems to work with this assumed role: arn:aws:sts::123:assumed-role/shared-eks-node-role/i-123 but it is not authorised with Kinesis permissions. Do you know what process creates that role? Perhaps you could add the missing Kinesis policies there?

Answer 2

I had the same issue. First make sure you have the IAM role setup correctly according to https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html. Make sure the names are consistent and it has the right permissions.

Once you've double-checked that, make sure you are on a recent version of snowplow. An old version might not have the right version of the AWS SDK. You need at least AWS SDK v1.12.128 or for AWS SDK v2, 2.10.11 [link].

Finally set the aws accessKey and secretKey in your snowplow configuration file to default. Redeploy and make sure the pod and service account has been recreated. You should be good at this point.

Reference:

https://github.com/snowplow/stream-collector/issues/186

Answer 3

I have the same issue.

It can't use env for the values, because those are not set. However, the collector is runnning as a container - it should use the default credential chain.

From the notes, it looks like without the the env variables being set, i should use iam - which when I do this, it uses the IAM Instance Profile, which loads the underlying nodes role - not the role specified by the SA.

The SDK supports IRSA (i have updated snowplow collector container image to 1 that has supported SDK of greater than 1.11.704 as per supported versions), and from what I can see from the collector docs, the streams config needs an aws block with either env or iam as values... but I want to use the default credential chain without specifying a method....

If i connect to the container, i can see that the creds are set up as per the SA:

$ env | grep -i aws
AWS_REGION=my-region
AWS_DEFAULT_REGION=my-region
AWS_ROLE_ARN=arn:aws:iam::<redacted>:role/sp-collector-role
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

But when I run the collector, it still uses the nodes IAM Instance profile, and I don't see any activity under sp-collector-role. Is there a way to use the default credential chain? e.g. with aws CLI in on a container in the same service account, I don't specify any credentials, but when I run aws sts get-caller-identity the SDK resolves the IRSA role correctly.