0

I scraped a website, created a PySpark DataFrame and now I would like to upload the DataFrame in a CSV format to my AWS S3 bucket without saving to disk.

I use PyCharm IDE (where I set my Access Key and Secret Access Key and selected the region. The program is written in Python.

I created the DataFrame like this:

df = spark.createDataFrame(pdf, schema=mySchema)

Thus, it is not yet a CSV file, because if I create a CSV that gets saved to disk.

Van Gran
  • 67
  • 9
  • https://stackoverflow.com/questions/31031463/can-you-upload-to-s3-using-a-stream-rather-than-a-local-file Take a look at this – qkhanhpro Aug 22 '19 at 02:59
  • Terrific, thanks it solved the issue! It's a possible duplicate, will make sure to delete the question. – Van Gran Aug 22 '19 at 16:19

0 Answers0