1

I have old code below that gzips a file and stores it as json into S3, using the IO library ( so a file does not save locally). I am having trouble converting this same approach (ie using IO library for a buffer) to create a .txt file and push into S3 and later retrieve. I know how to create txt files and push into s3 is as well, but not how to use IO in the process.

The value I would want to be stored in the text value would just be a variable with a string value of 'test'

Goal: Use IO library and save string variable as a text file into S3 and be able to pull it down again. 

x = 'test'
inmemory = io.BytesIO() 
    with gzip.GzipFile(fileobj=inmemory, mode='wb') as fh:

        with io.TextIOWrapper(fh, encoding='utf-8',errors='replace') as wrapper:
            wrapper.write(json.dumps(x, ensure_ascii=False,indent=2))
    inmemory.seek(0)
    s3_resource.Object(s3bucket, s3path + '.json.gz').upload_fileobj(inmemory)  
    inmemory.close()

Also any documentation with that anyone likes with specific respect to the IO library and writing to files, because the actual documentation ( f = io.StringIO("some initial text data") ect..https://docs.python.org/3/library/io.html ) It just did not give me enough at my current level.

0004
  • 1,156
  • 1
  • 14
  • 49

1 Answers1

1

Duplicate.

For sake of brevity, it turns out there's a way to override the putObject call so that it takes in a string of text instead of a file.

The original post is answered in Java, but this additional thread should be sufficient for a Python-specific answer.

Hari Amoor
  • 442
  • 2
  • 7
  • Thank you Hari Amoor, but still not quite sure how the IO library would be used for my specific need – 0004 Dec 31 '19 at 05:58
  • You shouldn't need the I/O library. Your current workflow is to create a fake file with the I/O – Hari Amoor Dec 31 '19 at 06:00
  • Apologies, that last comment was inconclusive. – Hari Amoor Dec 31 '19 at 06:00
  • You don't need the I/O library at all. You just need the S3 object, and you can use `put_object` to save the given string into a file on the bucket. – Hari Amoor Dec 31 '19 at 06:01
  • I guess for consistency of what the code was for other loads into s3 entailed is looming to mirror if possible – 0004 Dec 31 '19 at 06:04
  • Then you don't have to do anything differently. Currently, you have `s3path + "json.gz"` as your file name, but there's no reason you can't change that to `s3path +"txt.gz` and use the exact same code. You don't have to do anything special to create a text file as opposed to any other file other than changing the extension. – Hari Amoor Dec 31 '19 at 06:09
  • Hmm, I would like it not zipped , I am confused then what the json.dumps call is doing ? – 0004 Dec 31 '19 at 06:12
  • Omit it. Since you don't have a JSON-structured object, you don't need it. Code should be `wrapper.write(x, ...)` in that line and `s3_resource.Object(s3bucket, s3path + ".txt.gz").upload_fileobj(in memory)` in the line where you upload the file. – Hari Amoor Dec 31 '19 at 06:14
  • thank you so much for your help Hari and pointing that out. I have to sleep , but will accept your answer but may have another question in the comments here in the morning when u implement. But thank you again. ! – 0004 Dec 31 '19 at 06:16
  • Sure, LMK how it goes! – Hari Amoor Dec 31 '19 at 06:16