2

A dataframe in Spark can be created from an external file. Is there a way to use multiple files to create a single object?

EX: If I have say a csv files or parquet files that record the same data (say the counts of an item purchased on a day of the week). Is there a way to initialize a single dataframe from each day's file to get a single dataframe that represents the data of a week's purchases?

knowads
  • 705
  • 2
  • 7
  • 24

1 Answers1

2

The way spark reads input from a file depends on the underlying Hadoop API's. This means quite often that they extend the same usage, including being able to handle compressed files, or multiple files.

In your case, you would just provide a file input using a wildcard or separated individually by comma. See How to read multiple text files into a single RDD?

Community
  • 1
  • 1
YoYo
  • 9,157
  • 8
  • 57
  • 74
  • If I provided a folder name with one type of file, would Spark be able to take in all the data in said folder? – knowads Jul 18 '16 at 23:48
  • 1
    Yes, assuming you can all parse them the same way, using following path specifier: `"/my/own/path/*"` – YoYo Jul 18 '16 at 23:50