1

I am newbie in aws. there is a week I try to stream data from Nifi to Redshift as test to learn how it work. what I do is that generating a simple record "aa,12,2020" with Nifi send it to kinesis firehose and kinesis send it to s3 bucket and then from there to a table in redshiftcluster. I have this error all the time

Redshift.ConnectionFailed

I had try it without nifi with test data from kinesis but the result is the same. what I had done til now to resolve the problem is what is writen in this link: AWS Kinesis Firehose not inserting data in Redshift it is a little more detailed in compare with aws help. in this list all things seems to be ok, until

At this point, you should be able to see the connection attempts in Redshift logs : `select * from stl_connection_log where remotehost like '52%' order by recordtime desc;

there is any record in my redshift DB.

how much I red, all saying that it is about security and rolls. I think I miss something and I can not find it. I had 3 times delete all process and make it again, but not helped. here is my settings for testing with kinesistestdata: Kinesisfirehose:

S3 Bucket: data file content in bucket:

{"ticker_symbol":"AAPL","sector":"TECHNOLOGY","change":0.26,"price":90.78}{"ticker_symbol":"WMT","sector":"RETAIL","change":1.84,"price":95.82}{"ticker_symbol":"CVB","sector":"TECHNOLOGY","change":-0.07,"price":57.51}{"ticker_symbol":"DFT","sector":"RETAIL","change":-0.06,"price":131.62}

Object URL-xml file in bucket:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>11111</RequestId>
<HostId>
1111111111111    </HostId>
</Error>

Redshift Cluster:

  • attached IAM Role AWSServiceRoleForRedshift firehose_delivery_role (same role in kinesis)
  • Network and Security: Virtual private cloud (VPC) info
  • both CIDR and inbound rules: 172.31.0.0/16 13.58.135.96/27
  • Availability Zone : us-east-2a
  • VPC security group:default
  • Publicly accessible: yes

I have read also the Jason pollicies file like: https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html#using-iam-s3 but it is not clear how should I do it. on the other side the policy are added via GUI, so it should work.

would you please help me how should I go step forward? Thanks alot!

--------some days later-------------

after finding error of copy command I tried to find problem with help of this page: s3 access denied 403 I could not find a problem specifically but there is no error in kinesis:redshift log and no error in Redshift database with these 3 query:

select * from STL_LOAD_ERRORS;
select * from  STL_FILE_SCAN;
select * from STL_S3CLIENT_ERROR order by recordtime desc;

but still no data is coping in redshift.(in kinesis monitoring tab the "DeliveryToRedshift Success " is still on zero) I will appreciate if anyone has an idea.

samsoft
  • 21
  • 7
  • I tried to test my redshift wtih copy command as below: COPY firehose_test_table FROM 's3://...file' CREDENTIALS 'aws_iam_role=arn:aws:iam::....' MANIFEST JSON'auto'; but it is wierd too. the copy command response successfully but select command shows noting inserted in the database. – samsoft Apr 19 '20 at 15:29

1 Answers1

1

first step: ok, I am a step forward. with this query in redshift, a problem is shows itself

select * from STL_S3CLIENT_ERROR order by recordtime desc;

the problem is the same: access denied! but it is more resources to search about. for those who have the same problem as me, these two queries can also be helpfull:

select * from STL_LOAD_ERRORS;
select * from  STL_FILE_SCAN;

here is the link for more information. System tables for troubleshooting data loads

second step: there was some problems with my copy syntax. there is different way to write arn in copy and it is depend on which type of security you like to use. I should write it like this:

copy category from 's3://mybucket/testdata/category_pipe.txt'  
iam_role 'arn:aws:iam::11111111111'
                    delimiter '|' region 'us-east-2';

here is more information about ARN

now my copy example work and I could make connection between redshift and s3 successfully

samsoft
  • 21
  • 7