How to import two csv files into the same dataframe ( the directory for files are different)

Question

I want to create dataframe from two csv files have same schema but the folder path is different from the another

score 3 · Answer 1 · edited May 17 '18 at 05:47

3

In Spark 2.x :

Single DataFrame from CSV files stored in different directories

val df = spark.read.option("header", "true").option("inferSchema", "true").csv(path1,path2)

Single Dataframe from CSV files stored in directory in a recursive way (using wildcard characters)

val df = spark.read.option("header", "true").option("inferSchema", "true").csv(parent-directory/\*/*)

edited May 17 '18 at 05:47

koiralo

answered Feb 28 '18 at 10:16

PoojanKothari

thanks for providing this. was not sure if you noted it, but the question in this case is tagged with pyspark and the OP's question uses Python syntax. Your scala code is pretty similar to what we might use with Python, but could still be confusing to readers seeking Python guidance. – E. Ducateme Mar 01 '18 at 00:22

Ramesh Maharjan · Accepted Answer · 2018-02-28T08:28:00.017

0

You can give list of string paths while you read the csv files using sqlContext

sqlContext.read.format("com.databricks.spark.csv").csv(["path1", "path2"]).show(truncate=False)

or using load

sqlContext.read.format("com.databricks.spark.csv").load(["path1", "path2"]).show(truncate=False)

You can play with other options as header inferSchema etc...

edited Feb 28 '18 at 08:28

answered Feb 28 '18 at 08:17

Ramesh Maharjan

2 Answers2