1

I am following the next example to connects to Amazon Kinesis Data Streams and outputs the joined results to Amazon S3 in parquet format: https://docs.aws.amazon.com/glue/latest/dg/glue-etl-scala-example.html

When I configure the connection to AWS and trying to run it, it throws the next error that can't find the credentials and I have assigned in "C:\Users\user.aws\credentials" also I have added environment variables :

AWS_ACCESS_KEY_ID=XXX
AWS_SECRET_ACCESS_KEY=XxXX

But still failing with:

 ERROR AWSInstanceProfileCredentialsProviderWithRetries: Got an exception while fetching credentials org.apache.s
park.sql.kinesis.shaded.amazonaws.SdkClientException: Unable to load credentials from service endpoint
Asier Gomez
  • 6,034
  • 18
  • 52
  • 105

1 Answers1

0

The error message is not about your own credentials that you have in "C:\Users\user.aws\credentials", but about missing credentials in an instance profile.

The class AWSInstanceProfileCredentialsProviderWithRetries extends InstanceProfileCredentialsProvider which is:

Credentials provider implementation that loads credentials from the Amazon EC2 Instance Metadata Service

The code you are executing should be running on EC2 instance, therefore it expects to get credentials form instance role. What's more the instance role should have all necessary permissions which your code requires, e.g., using Kinesis.

From the link provided in your question, it is not clear where the example code should be exactly executed. Whether it is a custom ec2 instance, or some glue-related managed instance.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • And it is not possible to run it locally? I use the next config: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-scala And the code of the example I add in the question. :S – Asier Gomez Jun 01 '20 at 09:42
  • @AsierGomez It should be possible, but you need to use different credentials provider in your code. There are many for java listed [here](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AWSCredentialsProvider.html). For example, `DefaultAWSCredentialsProviderChain` can use "Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI". – Marcin Jun 01 '20 at 09:44
  • 1
    Finally adding the credentials as options to the sparkSession.readStream it works: ```.option("awsAccessKeyId", "XXXX") .option("awsSecretKey", "xxxXXX") .option("aws_iam_role", "test-kinesis-full-access")``` – Asier Gomez Jun 01 '20 at 10:09
  • Glad to hear it worked out. Thanks for letting me know. – Marcin Jun 01 '20 at 10:13