How can I read data from multiple directories in pyspark

Question

I have directories/files in S3 in the below structure.

root/
    20180101/files.txt
    20180102/files.txt
    20180103/files.txt

Now i want to pass a date range as start_date=20180101 and end_date=20180102 . I want the pyspark code to read files from these directories included in the range. How can i achieve this.

**The range is configurable, i.e it can be 1 week/30days/90days

What have you done so far? Please provide your attempts as well :) — Ala Tarighati, Oct 08 '18 at 11:33
@AllaTarighati I know that if I just pass the root directory `/root` , than all folders can be read, but i need to read folders based on dates provided. — darkmatter, Oct 08 '18 at 11:36

score 0 · Answer 1 · answered Oct 09 '18 at 06:00

I created a list of paths of the date range and passed to sc.text().

start = datetime.datetime.strptime(start_date, '%Y%m%d')
end = datetime.datetime.strptime(end_date, '%Y%m%d')
step = datetime.timedelta(days=1)
paths = []

while start <= end:
    paths.append(s3_input_path+str(start.date().strftime("%Y%m%d"))+"/")
    start += step
str1 = ','.join(paths)

How can I read data from multiple directories in pyspark

1 Answers1