pyspark submit command on AWS EMR

Question

I have 4 python scripts and one configuration file of .txt . out of 4 python files , one file has entry point for spark application and also importing functions from other python files . But configuration file is imported in some other python file that is not entry point for spark application . I want to write spark submit command in pyspark , but I am not sure how to provide multiple files along configuration file with spark submit command when configuration file is not python file but text file or ini file.

for demonstration: 4 python files : file1.py , file2.py , file3.py . file4.py

1 configuration file : conf.txt

file1.py : this file has spark session and calling to all other python files . file3.py : this python file is reading conf.txt .

I want to provide all these files with spark submit but not sure about command . command I have tried is below :

'Args': ['spark-submit',
                         '--deploy-mode', 'cluster',
                         '--master', 'yarn',
                         '--executor-memory',
                         conf['emr_step_executor_memory'],
                         '--executor-cores',
                         conf['emr_step_executor_cores'],
                         
                         '--conf',
                         'spark.yarn.submit.waitAppCompletion=true',
                         '--conf',
                         'spark.rpc.message.maxSize=1024',
                        
                         f'{s3_path}/file1.py', 
                         '--py-files',
                         f'{s3_path}/file2.py',
                         f'{s3_path}/file3.py',
                         f'{s3_path}/file4.py',
                         '--files',
                         f'{s3_path}/config.txt'
                        
                         
                        ]

but above command is throwing an error : File "file1.py", line 3, in from file2 * ModuleNotFoundError: No module named 'file2'

Check it looks your import is going before session initiation — Nikita Rogatov, Sep 24 '20 at 12:26
I am not using sc.addFile() function instead passing python files with --py-file option with spark submit . when i run spark submit command and providing python files with --py-files does still import statement are required once application is initialized ( spark session) . Since I have run the same spark submit command in my local machine it was working but running on aws emr it is giving this issue . — Jay, Sep 24 '20 at 13:36

A.B · Answer 1 · 2020-09-25T08:41:21.603

option 1 Put py-files with comma separated syntax before the actual file as

'Args': ['spark-submit',
                '--py-files',
                'file2.py,file3.py,file4.py',
                'file1.py',
                '--files',
                 f'{s3_path}/config.txt]
        }

In your case it may be like (f'{s3_path}/file2.py,{s3_path}/file3.py,{s3_path}/file4.py') Now to include text file

   sc.textFile("config.txt")

Option 2: Zipping the files

moreover you can zip them and include like this

First put them in a directory for instance myfiles/ ( In addition make empty __init__.py file at root level in this directory like myfiles/__init__.py )

From outside this directory,make a zip of it (for example myfiles.zip)

For submission, you can add this zip as

'Args': ['spark-submit',
                '--py-files',
                'myfiles.zip',
                'file1.py'
        }

Now include this zip with sc.addPyFilefunction

sc.addPyFile("myfiles.zip")

Considering your have __init__.py , file2.py, file3.py, file4.py and config.txt in myfiles.zip

You can now use them as

from myfiles.File1 import *
from myfiles.File2 import *

Update: you asked that

in option2 : do i need to provide path for '--py-files', 'myfiles.zip', with spark submit or in sc.addPyFile() or with both ?

yes you need to provide path of the myfile.zip like, /home/hadoop/myfiles.zip, so this means that you need to have this file master node, you can either do it with bootstrap script to copy it from s3,

or have a step to copy these

{
        'Name': 'setup - copy files4',
        'ActionOnFailure': 'TERMINATE_CLUSTER',
        'HadoopJarStep': {
            'Jar': 'command-runner.jar',
            'Args': ['aws', 's3', 'cp',
                YOUR_S3_URI + 'myfiles.zip',
                '/home/hadoop/']
        }
    }

I have run this job in local and this hasworked in local. :spark-submit file1.py --master local --driver-memory 2g --executor-memory 2g --py-files file12.py,file3.py,file4.py --files conf.txt .....but running same command in aws emr is throwing an error . — Jay, Sep 24 '20 at 13:55
I think you have not included the files correctly, on local since you have them in correct place, it is working, but on emr they arent submitted correctly — A.B, Sep 24 '20 at 13:58
I have run below command but still I am getting same error . f'{s3_path}/file1.py', '--py-files', f'{s3_path}/file2.py,{s3_path}/file3.py,{s3_path}/file3.py', '--files', f'{s3_path}/configuration.txt', — Jay, Sep 25 '20 at 05:59
in option2 : do i need to provide path for '--py-files', 'myfiles.zip', with spark submit or in sc.addPyFile() or with both ? — Jay, Sep 25 '20 at 07:06
yes you need to provide path of the `myfile.zip` like, `/home/hadoop/myfiles.zip`. More details in the answer, i have updated it. BTW, Do upvote to appreciate the help, it keeps other motivated to help on this volunteer platform :) — A.B, Sep 25 '20 at 08:39

pyspark submit command on AWS EMR

1 Answers1