1

I'm trying to read CSV using spark in databricks:

spark.read.format('csv').option('header', 'true')\
  .option('inferschema', True)\
  .option('quote', '\"')\
  .option("escape", '\"')\
  .load(path_to_csv)\
  .createOrReplaceTempView('table_name')

But it doesn't read correcrtly following line:

""Sample Company",LLC"

Instead of getting:

+------------------------+
|                   col1 |
+------------------------+
|    "Sample Company",LLC|
+------------------------+

I get following result:

+------------------+--------------------+
|             col1 |                col2|
+------------------+--------------------+
|""Sample Company" | LLC"               |
+------------------+--------------------+

Tried different combinations of "quotes" and "escape" options, but nothing works so far.

0 Answers0