I'm trying to read CSV using spark in databricks:
spark.read.format('csv').option('header', 'true')\
.option('inferschema', True)\
.option('quote', '\"')\
.option("escape", '\"')\
.load(path_to_csv)\
.createOrReplaceTempView('table_name')
But it doesn't read correcrtly following line:
""Sample Company",LLC"
Instead of getting:
+------------------------+
| col1 |
+------------------------+
| "Sample Company",LLC|
+------------------------+
I get following result:
+------------------+--------------------+
| col1 | col2|
+------------------+--------------------+
|""Sample Company" | LLC" |
+------------------+--------------------+
Tried different combinations of "quotes" and "escape" options, but nothing works so far.