0

After processing a big data on pyspark, I saved it on csv using the following command:

df.repartition(1).write.option("header", "true").option("delimeter", "\t").csv("csv_data", mode="overwrite")

Now, I want use pd.read_csv() to load it again.

info = pd.read_csv('part0000.csv', sep='\t', header='infer')

info is returned as 1 column where the data is separated by comma not '\t'.

col1name,col2name,col3name
val1,val2,val3

I tried to specify the sep=',' but I got an parsing error where some rows have more than 3 cols.

How to fix that without skipping any rows ? Is there anything to do with spark to resolve it such as specify a '|' as delimiter

LearnToGrow
  • 1,656
  • 6
  • 30
  • 53

2 Answers2

0

The csv format writer method DOESN'T have the delimeter option, guess what you need is the sep option.

Please refer to here

过过招
  • 3,722
  • 2
  • 4
  • 11
0

As mentioned here, pandas treats the char " for queting, and expects " after every " which is not always true in my case. To fix this problem, we have to specify quoting=3 to not use quote.

data = pd.read_csv('data.csv', header='infer', sep='\t', engine='python', quoting=3)
LearnToGrow
  • 1,656
  • 6
  • 30
  • 53