Debugging Popen subprocesses with PyCharm

Question

I'm trying to debug a Python application that uses psutil.Popen objects. When I start a subprocess, PyCharm replaces my command line with the following:

python -m pydevd.py --multiproc --client 127.0.0.1 --port 52581 --file <myapplication>

which ends up in an error:

python.exe: Import by filename is not supported.

When I launch the same command without -m option, everything seems to be fine. Is there a way I can change PyCharm's debugger launch command?

I've updated to PyCharm Community Edition 4.0.3 and the new debugger command looks like:

python.exe "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 4.0.3\helpers\pydev\pydevd.py" 
--multiproc --client 127.0.0.1 --port 62661 
--file __main__.py local -c local.yml -f input/11_12.xls

where -c and -f are my module's command line arguments. The debugger launch command has changed, but it didn't solve the issue; I still get the Import by filename is not supported error.

A code example is available here at Bitbucket.org. Pycharm's run configuration should look like:

Script:            __main__.py
Script parameters: server
Working directory: %path to the repository%

How does `` look like? Is it only a name or full path? If it's full path then try changing it to name only and make sure the parent folder is in `sys.path`. Compare with http://stackoverflow.com/questions/14465473/ — Piotr Dobrogost, Oct 21 '14 at 15:56
Facing the same issue with Community Edition 3.4 (and my 2nd day of python development). Can anyone comment if it worked with previous versions ever? — Prabhjot, Oct 24 '14 at 14:43
I found a workaround for this. First run the script from PyCharm and then attach debugger to the processes (this should show all the subprocesses). Tools --> Attach To Process. Somewhere I read that subprocess debugging is not supported by PyCharm and is planned for V4. — Prabhjot, Oct 26 '14 at 09:50
@Prabhjot From https://www.jetbrains.com/pycharm/webhelp/python-debugger.html – *Attach to subprocess automatically while debugging: If this check box is selected, PyCharm will automatically attach all subprocesses of the process being debugged. Thus, if the parent process has subprocesses, their breakpoints will always work.* The feature was already available in PyCharm 2.7 (might be available even before 2.7). — Piotr Dobrogost, Oct 28 '14 at 12:27
@PiotrDobrogost `` is a python module `__main__.py` file, not the full path — Ivan Gromov, Dec 19 '14 at 16:13
Can you provide a code example which can be used to reproduce the problem? — yole, Dec 19 '14 at 17:34
@yole https://bitbucket.org/lgyanf/pycharm-multiprocess-debug — Ivan Gromov, Dec 22 '14 at 08:01
@PiotrDobrogost this will work for breakpoints, but not the debug console. See my related posts in https://stackoverflow.com/questions/63525050/how-to-debug-code-run-using-popen-in-pycharm and https://intellij-support.jetbrains.com/hc/en-us/community/posts/360009571680-Debugging-code-run-through-subprocess-Popen — user32882, Aug 24 '20 at 09:44

score 19 · Accepted Answer · answered Dec 23 '14 at 10:43

19

As Piotr mentioned, PyCharm 'Attach to subprocess automatically while debugging'. If subprocess is a Python process, PyCharm debugger change the process's startup arguments (see function patch_args at source). When you start subprocess in this way:

args = ['python',
        '-m', 'pycharm-multiprocess-debug',
        'worker']
worker = subprocess.Popen(args)

The actual startup command is like:

python.exe -m "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 4.0.3\helpers\pydev\pydevd.py"
--multiproc --client 127.0.0.1 --port 62661
--file pycharm-multiprocess-debug

So it went wrong. There are several workarounds I can find:

easiest way, if you don't need to debug subprocess, just turn off "Attach to subprocess automatically while debugging" inside PyCharm settings
change your args to:
```
args = ['python', '__main__.py', 'worker']
```
The disadvantage is you can only run a Python file, not a Python module.

I recommend the last solution for Python subprocess:

from multiprocessing import Process

def server():
    p = Process(target=worker)
    p.start()
    print 'worker pid: {}'.format(p.pid)
    p.join()

answered Dec 23 '14 at 10:43

ZZY

3,689
19
22

where should I write that code?? args = ['python', '__main__.py', 'worker'] – Marco Sanchez Mar 10 '15 at 01:08
@MARCO, the `args` is in Ivan's code sample: https://bitbucket.org/lgyanf/pycharm-multiprocess-debug/src/973bebe2e1c249ff75f5db1e8ddaad0720a0dec5/__main__.py?at=default – ZZY Mar 10 '15 at 06:16
My current command (that doesn't get the Pycharm debugger attached) looks like this: subprocess.Popen("cd /users/lewis/; ENV_VAR=99; python main.py") – user3180 Jan 14 '19 at 06:37
@user3180 you did not mention `shell=True`, but apparently, you have tried to run a shell command sequence. Subprocess debugging will not work with `shell=True`. The program to be run needs to be `'python'` (and I would recommend to use `sys.executable` instead of `'python'` to make sure it is the same Python as the parent). In your case you would also need to pass `cwd` and `env` parameters. – Adrian W May 06 '19 at 18:15

Debugging Popen subprocesses with PyCharm

1 Answers1

Linked