0

I'm trying to use some particle physics software and run it in a loop with slightly changed variables each time. It has specific functionality to take a txt file and scan it for relevant commands for this, so naturally I am trying to make use of this. I have written a python script (this was originally written in python3 to run on my laptop, but I need to move over to the university's cluster. I've converted to python2 as the particle physics software itself is python 2 and trying to run both versions of python on the cluster simultaneously is not something I know how to do. Specifically the cluster has python2.6.9 installed, I do not have any ability to change this, I also cannot update it or install new modules. The problematic part of my code is:

     def run_madgraph():
        # starts madgraph and tells it to run mg5_aMC using mg_run_iridis.txt as a file for it.
            print('starting....')
            subprocess.check_call(['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt'])

mg5_aMC starts up the physics software, mg_run_iridis.txt is the text file with the information about variables. I don't understand why this doesn't work as the following command works just fine in the bash terminal (on the cluster):

    python ./madgraph/bin/mg5_aMC mg_run_iridis.txt

As far as I can tell the issue is on the python-bash side rather than it being anything in the physics software. The error I get is:

(MGenv) [cb27g11@green0155 Development]$ python MainIridis_script.py 
about to run madgraph 1.0 0.9
starting....
python: can't open file '/home/cb27g11/Development/./madgraph/bin/mg5_aMC mg_run_iridis.txt': [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/home/cb27g11/Development/MainIridis_script.py", line 41, in <module>
    the_main_event()
  File "/home/cb27g11/Development/MainIridis_script.py", line 21, in the_main_event
    run_madgraph()
  File "/home/cb27g11/Development/MainIridis_script.py", line 38, in run_madgraph
    subprocess.check_call(['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt'])
  File "/home/cb27g11/.conda/envs/MGenv/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt']' returned non-zero exit status 2.

Just to be clear, /home/cb27g11/Development, definitely exists:

(MGenv) [cb27g11@green0155 Development]$ pwd
/home/cb27g11/Development

As do the various files involved:

(MGenv) [cb27g11@green0155 Development]$ ls
AD_noCharge_021020.tar.gz  NoChar_Allh_1.0_0.9  mg_run_basic.txt
MainIridis_script.py       NoChar_Allh_1_0.9    mg_run_iridis.txt
NoCh_h1Only.tar.gz         iridis_script.pbs    py.py
NoCh_h2Only.tar.gz         madgraph

Including mg5_aMC:

(MGenv) [cb27g11@green0155 Development]$ cd madgraph/
(MGenv) [cb27g11@green0155 madgraph]$ cd bin
(MGenv) [cb27g11@green0155 bin]$ ls
mg5  mg5_aMC  py.py

I don't know how else to go about this, and ultimately however I do it, I still need to call this script file inside another script (bash this time) in order to submit it to the cluster.

I'm unsure if it's possible to just avoid the python script all together and switch to bash, I suspect at least some of it needs to remain in python but I'm a physicist not a programmer so I could easily be wrong! Here's the full script in anycase:

from __future__ import absolute_import, division, print_function
    import numpy as np
    import subprocess
    import fileinput
    
    tan_beta = np.arange(1, 21.2, 0.2) #Array of tan(beta) values
    sin_bma = np.arange(0.9, 1.001, 0.001)#Array of sin(beta-alpha) values
    hcm =  1.500000e+05 # Mass of charged Higgses in GeV
    textfilepath = 'mg_run_iridis.txt' #path to txt file madgraph will use
    Process = 'NoChar_Allh'
    
    def the_main_event():
            with open('mg_run_basic.txt','r') as old_card:
                    text =old_card.read() #stores string of old_card ready for editing
            for i in range(0, len(tan_beta)):
                    tb = tan_beta[i] # loops over tan(beta) values
                    for j in range(0, len(sin_bma)):
                            sbma = sin_bma[j] # loops over sin(beta-alpha) values
                            make_input(tb, sbma, hcm, text)
                            print('about to run madgraph ' + str(tb) + ' ' + str(sbma))
                            run_madgraph()
    
    def make_input(Tb, Sbma, Hcm, Text):
    # inputs are the value of tan_beta, the value of sin(beta-alpha) values, the desired mass for the charged higgses and a string of text
    
            with open(textfilepath, 'w') as new_card:
            #simulation card, the .txt file that gets fed to madgraph
                    sim_card = Text.replace('TBV', str(Tb))
                    sim_card = sim_card.replace('SBMAV', str(Sbma))
                    sim_card = sim_card.replace('HCMV', str(Hcm))
                    sim_card = sim_card.replace('NAME', str(Process)+'_' + str(Tb) +'_' + str(Sbma))
                    new_card.write(sim_card) # saves new txt file for madgraph
    
    
    def run_madgraph():
    # starts madgraph and tells it to run mg5_aMC using mg_run_iridis.txt as a file for it.
            print('starting....')
            subprocess.check_call(['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt'])
    
    
    the_main_event()

It seems like from the errors it's treating the madgraph executable (I'm not sure if thats correct terminology, I mean the file that starts up the propgram) as a module?

Any help would be really appreciated!

Ciara
  • 65
  • 5
  • 1
    I guess what Mikhail wrote is right. Just an addition. If you ever get an error like blablabla "is not callable" then you should inspect behind what kind of object you placed some (), because this tells python to interpret the object as callable and python complains if the object doesn't implement `__call___`. In the case above you could just have executed `help(subprocess)` to see what kind of object it is. – jottbe Nov 08 '20 at 14:11
  • By the way, why are you using `shell=True` when your string has no shell directives? `subprocess.run(['python', './madgraph/bin/mg5_aMC', 'mg_run_iridis.txt'])` is slightly more efficient, and much easier to modify to parameterize values without introducing security bugs (look up "shell injection"). – Charles Duffy Nov 08 '20 at 14:24
  • But then even better would be to not run Python as a subprocess of Python. If you can `import` the library you want to use instead, do that. – tripleee Nov 08 '20 at 14:58
  • @CharlesDuffy Honestly I'm not entirely sure why but on my laptop the command would only work with shell=True added. Though I was running it in a docker to avoid having to do the finicky installation of madgraph (this was a time restraint based decision more than anything). I should probably have mentioned it originally. The original command, that worked on my laptop was: `subprocess.run('sudo docker run -t -i -v $HOME/outputs:/var/MG_outputs --mount type=bind,source=$HOME/models,target=/app hfukuda/madgraph /home/hep/MG5_aMC_v2_6_3_2/bin/mg5_aMC /app/mg_run.txt', shell=True)` – Ciara Nov 09 '20 at 11:20
  • I'm going to add the full script to my original post @triplee as I'm not really well versed enough to know if I can do it in bash or not. Hopefully you'll be able to look at it and point me in the right direction? – Ciara Nov 09 '20 at 11:21
  • Before you do, perhaps review https://stackoverflow.com/questions/4256107/running-bash-commands-in-python/51950538#51950538 – tripleee Nov 09 '20 at 11:35
  • @tripleee Thank-you, but I'm still having difficulty. .run does not work as it treats madgraph as a module, .check_call, .call and .Popen all return errors related to child_exception, which is not something I really understand. Do you think I need to use pipes? I just want it to run the process then go ahead and change the relevant file and run it again on a loop, I didn't think it should be particularly complicated. – Ciara Nov 09 '20 at 11:55
  • Without seeing the actual errors, hard to say what's wrong. – tripleee Nov 09 '20 at 11:57
  • The `child_exception` errors should be included in the question. Please [edit] them in. – Charles Duffy Nov 09 '20 at 13:41
  • (If you don't have a `subprocess.run`, it means you're not on the latest version of Python; have you told us which version you're using?) – Charles Duffy Nov 09 '20 at 13:42
  • ...and once you've edited the question to show the child_exception errors, please `@`-notify me -- if the question is then a new one that isn't already given in the knowledge base, I'll reopen it. (Normally if you have an answer that edits would invalidate it's good form to ask a new question instead of editing, but in this case that answer shouldn't have been given, as the question was close-eligible). – Charles Duffy Nov 09 '20 at 13:55
  • @CharlesDuffy thankyou for taking the time to help me, I've edited the question so that it has the parts you asked for, I'm stuck on python2.6.9 as i don't have root privileges and the people in charge of the university cluster have decided on that version. – Ciara Nov 09 '20 at 14:19
  • Could you include the _exact_ `call` and `check_call` invocations, not just the exceptions they throw? It's important to know (f/e) whether you still have `shell=True` there. ("No such file or directory" is a pretty clear error, and it means what it says; if you left out `shell=True`, your entire command is being treated as a filename, and that file doesn't exist; if you _used_ `shell=True`, we have something much more interesting -- maybe your Python script is being run in a sandbox that doesn't have a copy of `sh`?). – Charles Duffy Nov 09 '20 at 15:41
  • (...to explain that, `subprocess.call("something", shell=True)` is exactly the same as `subprocess.call(["sh", "-c", "something"], shell=False)`, so when you use `shell=True`, the executable being run, and thus to which a "No such file or directory" refers to, is `sh`). – Charles Duffy Nov 09 '20 at 15:44
  • My guess is that `madgraph/bin/mg5_aMC` doesn't exist. Can you check that from within your Python code? – tripleee Nov 09 '20 at 15:55
  • @tripleee, that would be a CalledProcessException, not an OSError, if the OP were using `shell=True`. Clearly, they aren't. – Charles Duffy Nov 09 '20 at 16:23
  • @Ciara, here's your problem: `subprocess.check_call('python ./madgraph/bin/mg5_aMC mg_run_iridis.txt')` -- if you don't use `shell=True` (and you shouldn't), you need it to be `subprocess.check_call(['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt'])` – Charles Duffy Nov 09 '20 at 16:24
  • @tripleee I've added my checks that all the files/directories exist – Ciara Nov 09 '20 at 18:02
  • @CharlesDuffy I tried your suggestion and unfortunately I get a new error, I've updated the question to reflect this – Ciara Nov 09 '20 at 18:03
  • 2
    The list should have three elements, not two: `['python', './madgraph/bin/mg5_aMC', 'mg_run_iridis.txt']` – that other guy Nov 09 '20 at 18:09
  • @Ciara, ...exactly what that other guy says above. Unquoted, unescaped spaces in shell become argument separators, so when you're operating with `shell=False`, they need to become boundaries between individual arguments there as well. – Charles Duffy Nov 09 '20 at 18:13
  • Thank-you both! This has solved the problem (and greatly improved my understanding of subprocess)! – Ciara Nov 09 '20 at 19:02

1 Answers1

1

Try:

subprocess.run('python ./madgraph/bin/mg5_aMC mg_run_iridis.txt', shell=True)

Note this run part. See documentaion for details. Currently, you are trying to call the subprocess module itself, rather then a particular function from the module.

Mikhail Vladimirov
  • 13,572
  • 1
  • 38
  • 40
  • Unfortunately I did originally have 'run' in the command, which outputs the following: `Traceback (most recent call last): File "MainIridis_script.py", line 42, in the_main_event() File "MainIridis_script.py", line 22, in the_main_event run_madgraph() File "MainIridis_script.py", line 39, in run_madgraph subprocess.run('python ./madgraph/bin/mg5_aMC mg_run_iridis.txt') AttributeError: 'module' object has no attribute 'run' ` – Ciara Nov 09 '20 at 11:21
  • According to the documentation, `run` function was added in Python 3.5. What Python version do you use? – Mikhail Vladimirov Nov 09 '20 at 11:23
  • I use python 3 on my laptop, but on the university cluster I have a problem as madgraph is written in python 2, I'm having difficulty with how I'd switch between them on there so I just tried to convert my script to be python2. – Ciara Nov 09 '20 at 11:43