This works as expected and creates a new table. But the data is stored in a format that only spark can read. How do I store the data in csv format?
from pyathena.pandas.util import to_sql
to_sql(
mrdf,
"mrdf_table3",
conn,
"s3://" + bucket + "/tutorial/s3dir3/",
schema="hunspell",
index=False,
if_exists="replace",
)
I tried flavor="csv" or flavor="textfile" but the file that is generated is still not readable.
Update: Connection string
from pyathena import connect
bucket = "hunspell"
conn = connect(
aws_access_key_id="XXX",
aws_secret_access_key="XXX",
s3_staging_dir="s3://" + bucket + "/tutorial/staging/",
region_name="us-east-1",
)
