0

I have python module that should run python scripts (let's call it launcher)

I have list of scripts. Each of them has it's own virtual environment.

Launcher's input:

  • name of the script to launch
  • path to script
  • arguments to pass to script

I need to come up with solution, so that launcher was able to run scripts without creating new processes.

I tried to use __import__() function, but the main problem is that I don't know how to use script's own virtual environment.

Liquid Cat
  • 35
  • 1
  • 5
  • Do you have the option to create one conda environment that satisfy all requirement? Alternatively, can you run them all with a bash script, and then pipe the outputs? – Xbel Apr 15 '20 at 10:23
  • related: [activate-a-virtualenv-with-a-python-script](https://stackoverflow.com/questions/6943208/activate-a-virtualenv-with-a-python-script) – log0 Apr 15 '20 at 10:29
  • @Xbel If I'll use one venv for each one of my scripts, there's could be a problem that two scripts would require same module, but different versions – Liquid Cat Apr 15 '20 at 10:30
  • @log0 I am afraid that this is not what I am looking for. If I am not mistaken, subprocess creates new prcosee, so launcher and scripts are in different processes. I need them to be in one process. – Liquid Cat Apr 15 '20 at 11:09
  • @LiquidCat one of the anwswer is about loading the env in the same process – log0 Apr 15 '20 at 11:29
  • @log0 Oh yea. Didn't see it. Thanks – Liquid Cat Apr 15 '20 at 14:44

2 Answers2

0

If every script is gonna need different venv, then your best choice is to create a bash file with the pipeline and connect the scripts through output files. That's what I would do.

You can use pickle to transfer numpy arrays or dictionaries or other python objects, but be sure that the pickle protocol is the same.

For example:

#!/usr/bin/env bash
# General conf
var1 = 1.
var2 = "text"

# for each script
cd mypath1/
conda activate venv1
python script1.py -a 3.141592 -b var1 # Outputs some file
conda deactivate venv1


cd mypath2/
conda activate venv2
python script2.py -a var2 -b "text" # Takes the previous output
conda deactivate venv2

...
Xbel
  • 735
  • 1
  • 10
  • 30
0

Based on Lie Ryan from activate-a-virtualenv-with-a-python-script You could try to alter the current interpreter and import the scripts

# Doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))
log0
  • 10,489
  • 4
  • 28
  • 62