PySpark: how to resolve path of a resource file present inside the dependency zip file

Question

I have a mapPartitions on an RDD and within each partition, a resource file has to be opened. This module that contains the method invoked by mapPartitions and the resource file is passed on to each executor using the --py-files argument as a zip file.

To make it clear:

rdd = rdd.mapPartitions(work_doing_method)

def work_doing_method(rows):
    for row in rows:
        resource_file_path = os.path.join(os.path.dirname(__file__), "resource.json")
        with open(resource_file_path) as f:
            resource = json.loads(f.read())
            ...

When I do this after passing the zip file which includes all of this using the --py-file parameter to the spark-submit command,

I get IOError: [Errno 20] Not a directory:/full/path/to/the/file/within/zip/file

I do not understand how Spark uses the zip file to read the dependencies. The os.path.dirname utility returns the full path including the zip file, for eg. /spark/dir/my_dependency_file.zip/path/to/the/resource/file. I believe this should be the problem. I tried many combinations to resolve the path of the file. Any help is appreciated.

Thanks!

score 1 · Answer 1 · answered Nov 13 '18 at 12:58

1

I think when you add a file to a Spark job, it will be copied to the working directory of each executor. I've used the SparkFiles API to get absolute paths to files on the executors.

You can also use the --archives flag to pass in arbitrary data archives such as zipfiles. What's the difference between --archives, --files, py-files in pyspark job arguments

answered Nov 13 '18 at 12:58

Graham Healy

51
6

The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though. – void Nov 14 '18 at 08:45

void · Accepted Answer · 2018-11-23T11:47:00.133

1

We get the path to a resource file within an egg/zip file (inside the executor working dir) when we look for the absolute path. I ended up using the zipfile module in Python and actually open it like here.

edited Nov 23 '18 at 11:47

answered Nov 23 '18 at 10:43

void

2,403
6
28
53

Yeah, this works, but I'd love to have a solution that works in both client and cluster deploy modes. – Aleksey Tsalolikhin Dec 09 '21 at 17:58

PySpark: how to resolve path of a resource file present inside the dependency zip file

2 Answers2

Linked