0

I run an export from an app that we have and it dumps CSV files into S3 by month. The same process drops all the files at once and nothing is different about the file drops. My IAM user has full S3 access. I get FileNotFoundError: /mybucket/funnel-ad/raw/data/funnel_data_2020_04 and Access Denied during the handling of above exception. The file absolutely does exist, and I've checked manually in s3, plus I am generating the list of files by pulling from s3. Here's my code

session = Session(aws_access_key_id=AK,
                  aws_secret_access_key=SAK)
s3 = session.resource('s3')
bucket = s3.Bucket('mybucket')

for file in bucket.objects.filter(Prefix='funnel-ad/raw/data'):
    if 'funnel_data' in file.key:
        readpath = 's3://mybucket/'+str(file.key)
        df = pd.read_csv(readpath,engine='python', error_bad_lines=False)
        df = df.reindex(sorted(df.columns), axis=1)
        dct = df.to_dict(orient = 'records')
RagePwn
  • 411
  • 2
  • 8
  • 22
  • Have you checked bucket policy? Maybe it denies access to it? Also the object could be not owned by the owner of the bucket. In this case its ACL should be changed. – Marcin Apr 01 '20 at 22:24
  • The same process drops all 39 source files from the app db. I've done a full dump where it redrops all 39 files at once, so they're all created at the same time with the same permissions/creds/etc. The first 38 files are processed without problem by the code I attached. – RagePwn Apr 01 '20 at 23:07
  • if you are getting `FileNotFoundError`, maybe the name of the last file/object has some strange name, or characters that are incorrectly parsed by python or written in the csv? – Marcin Apr 02 '20 at 01:09
  • Might be useful: [How to import a text file on AWS S3 into pandas without writing to disk](https://stackoverflow.com/a/51777553/174777) – John Rotenstein Apr 02 '20 at 07:35

0 Answers0