0

I have a python code that is very intricate, it has manip, read, write, plot subfolders with their respective codes to do different tasks. The way is set up is that we have a "settings file" called IncludeFile.py which is imported as IncF in all the codes that require settings from it (basically all). At the very beginning of the code some of those variables are even rewritten with IncF.variable = some processing to original variable.

The script is run in a server with SLURM and the way the code is set up now I can only leave one job in queue, and I need to wait until it starts running so I can send another one with different settings. This because if I change the IncludeFile.py before it gets inside SLURM and starts running, I'd be rewriting the settings file and would end up with duplicate runs.

I am wondering if there is a way to make the IncludeFily.py be easily swapped. Like maybe when running the code I could do something like this to specify which file to use

python main.py IncludeFile.py.job1

But I don't see how that can be possible. It would be very annoying to have to change the header of 20+ files to change which file gets imported as IncF but that's the only way I can come up with fixing this issue. Is there a better way?

The whole point that I need is be able to leave a bunch of jobs in queue without them being duplicate runs as they would be now.

jonrsharpe
  • 115,751
  • 26
  • 228
  • 437
M.O.
  • 476
  • 7
  • 19
  • There are many approaches. If you have a lot of include files, why not consider importing main into each include file and have it run from there? This may also allow you to run several configuations at the same time – 576i Sep 29 '21 at 21:13
  • As a second option, look how to use a variable to define the import. as shown here: https://stackoverflow.com/questions/8718885/import-module-from-string-variable – 576i Sep 29 '21 at 21:16
  • @576i Thanks! I 've read both links and I think that pretty much solves my issue, I just need to copy/paste a bit of code in all modules but after that it will be set forever so that it can take different files. I ended up having to use `exec("import %s as IncF" % IncludeFile.replace(".py", ""))` because `IncF=__import__(IncludeFile)` didn't work. `IncludeFile` being equal to `sys.argv[1]` – M.O. Sep 29 '21 at 21:38
  • @576i Hey, could I just rewrite the file `IncludeFile.py` at the very beginning of the main code (before any modules get imported) by `sys.argv[1]`? Would that allow me to leave a bunch of jobs queued? I know that after compilation it's safe to edit the python source codes, but I am not sure if changing a file that gets imported at the beginning (before anything else is imported) would fit as changing before or after compilation. Hope that makes sense. Thanks! – M.O. Sep 29 '21 at 22:03
  • while rewriting the include file could work, it's a super hacky and I recommend not trying this in production, where there may be auto-reloading code. – 576i Sep 30 '21 at 13:14

0 Answers0