1

guys!

I'm just starting to learn NiFi. Don't throw stones) just help or guide. I need to read parquet data from s3 bucket, I don’t understand how to set up lists3 and fetchs3object processors for reading data. full path looks like this: s3://inbox/prod/export/date=2022-01-07/user=100/ 2022-01-09 06:51:23 23322557 cro.parquet I"ll write data to sql database - I don"t have problems with it.

I tried to configure the lists3 processor myself and I think is not very good bucket inbox aws_access_key_id aws_secret_access_key region US EAST endpoint override URL http://s3.wi-fi.ru:8080

Archi
  • 25
  • 6

1 Answers1

1

What I would do is try to test the Access Key ID, and Secret Key outside of NiFi to make sure that they are working. If they are working fine, then it’s an issue with the NiFi configuration. If the keys/id isn’t working, then by getting new values that work and providing them to NiFi, it might have a better shot of working.

Mike R
  • 464
  • 5
  • 16
  • Hi! I tried to test the Access Key ID, and Secret Key they are working. Besides it works in Nifi, but I don't understand how to configure the processors correctly and what should I use after fetchs3object to write data to sql database. – Archi Jan 11 '22 at 14:29
  • https://medium.com/@evanescence1106/fetch-object-from-s3-using-apache-nifi-660c314e96b1 Here’s a good start for fetching an object. What I would recommend is looking at these: https://stackoverflow.com/questions/41489816/connect-sql-to-apache-nifi https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.PutSQL/ – Mike R Jan 11 '22 at 15:00