2

I need to read contiguous files in pySpark. The following works for me.

from pyspark.sql import SQLContext    
file = "events.parquet/exportDay=2015090[1-7]"
df = sqlContext.read.load(file)

How do I read files 8-14?

deltap
  • 4,176
  • 7
  • 26
  • 35

2 Answers2

3

Use curly braces.

file = "events.parquet/exportDay=201509{08,09,10,11,12,13,14}"

Here's a similar question on stack overflow: Pyspark select subset of files using regex glob. They suggest either using curly braces, OR performing multiple reads and then unioning the objects (whether they are RDDs or data frames or whatever, there should be some way).

Community
  • 1
  • 1
kathleen
  • 170
  • 1
  • 5
  • Whilst this may theoretically answer the question, [it would be preferable](//meta.stackoverflow.com/q/8259) to include the essential parts of the answer here, and provide the link for reference. – Enamul Hassan Jun 09 '16 at 17:55
2

It uses shell globbing, I believe.

The post: How to read multiple text files into a single RDD?

Seems to suggest the below should work.

"events.parquet/exportDay=2015090[89],events.parquet/exportDay=2015091[0-4]"

Community
  • 1
  • 1
Barry Loper
  • 21
  • 5