difference between spark read textFile and csv

Question

I am trying to read a text file delimited by |. I am trying this

spark.read.format("com.databricks.spark.csv").option("header","true").option("delimiter", "|").option("inferSchema","true").csv("/tmp/file.txt").show()

I am only reading/seeing only the header but no data.

When I try the same with textFile, I am getting data but all in one column

spark.read.format("com.databricks.spark.csv").option("header","true").option("delimiter", "|").option("inferSchema","true").textFile("/tmp/file.txt").show()

Is there a way to read data via csv? I am using spark 2.4.4

Are you running this on Windows? – ernest_k Mar 05 '20 at 16:28 — ernest_k, Mar 05 '20 at 16:28
I am running it in an emr cluster – Srinivas Mar 05 '20 at 16:32 — Srinivas, Mar 05 '20 at 16:32

score 0 · Answer 1 · answered Mar 06 '20 at 07:52

0

The reason for the issue was the file is in UTF16 so I had to convert it and do run dostounix on it. Thanks for your advice. Apologies I really did not know that

answered Mar 06 '20 at 07:52

Srinivas

2,010
7
26
51

difference between spark read textFile and csv

1 Answers1