0

I currently try to understand the import io.

1) What I still didn't figure out what happens in the variable buffer_2. Why is that step necessary?

2) I couldn't figure out what the default delimiter is for csv.writer. Is it necessary to set this parameter?

buff = io.StringIO()
writer = csv.writer(buff, dialect='excel', delimiter=',')
writer.writerow(["a", "b", "c"])
buffer_2 = io.BytesIO(buff.getvalue().encode())

# Updated code sample here
BUCKET_NAME = 'fbprophet'
OBJECT_NAME = 'blah.csv'

s3.upload_fileobj(buffer_2, BUCKET_NAME, OBJECT_NAME)
Joey Coder
  • 3,199
  • 8
  • 28
  • 60
  • Sorry, you are right. I changed that before to buffer. It's "fixed" now in my sample. – Joey Coder Aug 14 '19 at 10:30
  • 2
    `import io` is needed to let you use the methods/types in the `io` module. Have you read the docs to [BytesIO](https://docs.python.org/3/library/io.html#io.BytesIO)? – Holloway Aug 14 '19 at 10:30
  • 1
    The default delimiter for the `csv` module is a comma. You do not need to specify it, but obviously nothing breaks if you are explicit – roganjosh Aug 14 '19 at 10:30
  • 1
    Take a look at this: https://www.journaldev.com/19178/python-io-bytesio-stringio – Kostas Charitidis Aug 14 '19 at 10:31

1 Answers1

3

I don't know where this code comes from, so I may only guess. I'd say that the author needed a binary buffer - a buffer of bytes which acts like a binary file. This is the io.BytesIO instance. But csv.writer() works with text and expects a text file. The io.StringIO instance acts like a text file. So the contents is written into the text file-like object (buff) first and then encoded into bytes and wrapped in a binary file-like object (buffer_2).

Tomáš Cerha
  • 309
  • 1
  • 7
  • The end goal of that code is to upload it directly on an AWS S3 bucket. So as far I understood now is that it probably must be a binary file that I upload (?). The rest you explained was clear to me, thank you! – Joey Coder Aug 15 '19 at 08:17
  • I updated the code snippet to show the next two lines where the "file" get's directly uploaded to S3. Is my understanding right that I need that binary file for that and can't "just" upload the text file from `io.StringIO` – Joey Coder Aug 15 '19 at 08:21
  • According to [the doc](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.upload_fileobj): "The file-like object must be in binary mode." – Tomáš Cerha Aug 15 '19 at 11:28
  • And according to [this question](https://stackoverflow.com/questions/5358322/csv-modules-writer-wont-let-me-write-binary-out), you should be able to write into a binary file directly with `csv.writer` which would save you the unnecessary step and some memory. – Tomáš Cerha Aug 15 '19 at 11:44