run .sh script from python that initates another process

Question

I was looking for an answer on StackOverflow, but most of them don't solve my entire task.

I have a software called cellprofiler, which works with its own GUI and runs in conda environment on Ubuntu.

I want to automate running this cellprofiler externally with another python script.

My steps:

1) I create a bash script in python which runs the environment and software:

env_activate = 'path_to_sh/activate_env.sh'
with open(env_activate, 'w') as f:
    f.write('#!/bin/sh\n')
    f.write('. activate cellprofiler && cellprofiler')
    f.close()

2) In my python script I then do:

processCP = subprocess.Popen(env_activate, stdout=subprocess.PIPE)
processCP.wait()

but this results in running it in a system python interpreter 3.5 rather than in conda environment

Traceback (most recent call last):
  File "/home/**/.local/bin/cellprofiler", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/**/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3126, in <module>
    @_call_aside
  File "/home/**/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3110, in _call_aside
    f(*args, **kwargs)
  File "/home/**/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3139, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/**/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 581, in _build_master
    ws.require(__requires__)
  File "/home/**/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 898, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/**/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 784, in resolve
    raise DistributionNotFound(req, requirers)

    pkg_resources.DistributionNotFound: The 'CellProfiler' distribution was not found and is required by the application

Does anyone know why this happens?

UPDATE: The python interpreter I need is python2.7 which is already in the conda environment. Cellprofiler works correctly if calling it in the terminal like:

source activate cellprofiler
cellprofiler

@JonahBishop yes, `shell=True` results in the same error, moreover I'd like to run it possibly in the `shell=False` mode in the future — Ren, Nov 16 '18 at 13:29
From `pkg_resources.DistributionNotFound:`, it looks like that some dependancy is unfulfilled. Can you see if the command runs manually? — Swadhikar, Nov 16 '18 at 13:29
@SwadhikarC it works well if I run it manually in my environment — Ren, Nov 16 '18 at 17:22

score 0 · Answer 1 · answered Nov 16 '18 at 13:26

0

Run in the terminal

pip install cellprofiler

answered Nov 16 '18 at 13:26

artona

1,086
8
13

I have cellprofiler installed in my conda environment. It works and runs well, the task it to call it from the python script as a new process running in this existing environment – Ren Nov 16 '18 at 13:31

score 0 · Accepted Answer · answered Nov 19 '18 at 12:01

To answer my own question: it helped to read through Running subprocess within different virtualenv with python

So, one needs to use subprocess.Popen to create a process. Another trick was to use she-ban #!/bin/bash\n in the bash script. after all it was created as:

 env_activate = 'path_to/activate_env.sh'
 with open(env_activate, 'w') as f:
        f.write('#!/bin/bash\n')
        f.write('source activate cellprofiler\n')
        f.write('cellprofiler')
        f.close()

and run by:

processCP = subprocess.Popen(env_activate, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

run .sh script from python that initates another process

2 Answers2