I am trying to write dataframe to Redshift table with following code using jdbc connection.
It is running very slow(running more than 20hours to process). Dataframe has 100 partitions.
Can you suggest how do we improve the performance for writing df to redshift table using jdbc connection
df.write.format('jdbc') \
.option("url", url) \
.option("dbtable", dbtable) \
.option("user",user) \
.option("password", password) \
.mode('overwrite').save()
Please provide your thoughts on how we can improve the performance.
Thanks, Bab