Add .csv extension in output file AWS Python Glue Job

Question

I am trying to save csv file from python glue dynmaic frame. Below is my code -

glueContext.write_dynamic_frame.from_options(
frame=splender_df,
connection_type="s3",
connection_options={"path": 's3://splender_df/'},
format="csv",

But it is creating 20 files and none of them has csv extension(.csv). How can I have only one file with extension .csv in the name of the file.

Try renaming the generated files afterwards. See this [answer](https://stackoverflow.com/a/53108702/10732434). — sanitizedUser, Jan 25 '22 at 19:46

score 1 · Accepted Answer · answered Jan 25 '22 at 19:50

Did you specified the extra options such as separators and quote_char? Like the following example:

glueContext.write_dynamic_frame.from_options(
frame = datasource1,
connection_type = "s3", 
connection_options = {
    "path": "s3://s3path"
    }, 
format = "csv", 
format_options={
    "quoteChar": -1, 
    "separator": "|"
    }, 
transformation_ctx = "datasink2")

You can find these options in this link.

Add .csv extension in output file AWS Python Glue Job

1 Answers1