2

After many researches on the web, on many different topics, I wasn't able to find the solution answering my problem.

I am developing a tool (with graphic interface) allowing to process python scripts from different projects. To not have any problem of dependences, I ask the user to target the project directory as well as the script to process.

Thus to develop this tool I have in my IDE a project "benchmark_tool" which is the project integrating the GUI, and the project "test_project" which groups various testing scripts. Obviously, the final goal is the tool to be compiled and thus that the project "benchmark_tool" disappears from IDE.

Here is the IDE architecture:

benchmark_tool (python project)
    __init__.py
    main.py

test_project (python project)
    __init__.py
    module1
        __init__.py
        test_script.py
    module2
        __init__.py
        imports.py

How it works: The main.py shall call test_script.py. test_script.py calls imports.py at the first beggining of the script.

UPDATE

I tried some modifications in my main code:

import sys
import subprocess

subprocess.check_call([sys.executable, '-m', 'test_project.module1.test_script'], cwd='D:/project/python')

I got this error

Traceback(most recent call last):
    File "C:/Python31/lib/runpy.py", line 110, in run module as main
        mod_name, loader, code, fname = _get_module_detail(mod_name)
    File "C:/Python31/lib/runpy.py", line 91, in get module details
        code = loader.get_code(mod_name)
    File "C:/Python31/lib/pkgutil.py", line 272, in get code
        self.code = compile(source, self.filename, 'exec')
    File "D:/project/python/test_project/module1/test_script.py", line 474


SyntaxError: invalid syntax
Traceback (most recent call last):
    File "D:/other_project/benchmark_tool/main.py", line 187, in read
        subprocess.check_call([sys.executable, '-m', 'module1.test_script.py'], cwd='D:/project/python/test_project')
    File "C:/Python31/lib/subprocess.py", line 446, in check call
        raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:/Python31/python.exe', '-m', 'module1.test_script.py']' returned non-zero exit status 1

Note It works with subprocess.Popen(['python','D:/project/python/test_project/module1/test_script.py])

What's the main difference between both methods ? I'll also have to pass arguments to test_scripts.py, which one is the best to use to communicate with a python script (input and output datas are exchanged) ?

Thanks by advance

v_lfr
  • 63
  • 8
  • on your update: Do you understand that `module1.test_script` and `module1/test_script.py` are different strings? Do not modify `PYTHONPATH`. Where do you see that I modify the pythonpath in my answer? please, read my answer carefully—do not add any other code. For clarity, update your question and specify whether module1 and module2 are top-level Python modules or whether they are submodules of `test_project` (i.e., whether `test_project` is a Python module or just a directory). Depending on the answer, use the appropriate code example from my answer which covers both cases. – jfs Jun 14 '16 at 17:58
  • Sorry I did put `module1.test_script`but I didn't write it well (I can't copy/paste my code I have to write it). test_project is a python module not just a directory, and module1 and module2 are submodules of test_script. I removed the `PYTHONPATH`command but got the same error. – v_lfr Jun 15 '16 at 07:20
  • Why do you have to write it? If you don't know how to copy-paste from the command line then learn (it is useful)—ask a separate question about your specific environment. I can't help you unless your provide the exact command. Again, `'module1.test_script.py'` is different from `'module1.test_script'`. If `test_project` is a Python module then use the second variant from my answer i.e., `subprocess.check_call([sys.executable, '-m', 'test_project.module1.test_script'], cwd=r'D:\project\python')`. Unrelated: you could use raw-string literals for Windows paths: with `r''` prefix. – jfs Jun 15 '16 at 23:36
  • Actually my code is on a computer with a protected network so I can't get out anything from it. Ok I tried with all possible cases you've told me, and I got the same error message each time. See my updated post. Btw, it seems that `sys.path.append()`worked, I may have had an environment issue. J.F.Sebastian – v_lfr Jun 16 '16 at 14:33
  • `SyntaxError` is not `ImportError`—congratulation, you've fixed the import issue. It doesn't mean there are no other problems in your code. Find computer where you can copy-paste. Create [mcve]. Why do you try to run `module1.test_script` instead of `test_project.module1.test_script`? If there is `test_project/__init__.py` then you should use `test_project.module1.test_script` name. – jfs Jun 16 '16 at 14:59
  • Do not try to import `test_project` in the parent. Do not use `sys.path.append()`. Do you understand that `subprocess` starts a separate process? (importing modules in the parent or changing `sys.path` in the parent has no effect on the subprocess). Do you see `from test_project import *` in my answer? Try the code exactly as I've shown it—do not add or remove anything. – jfs Jun 16 '16 at 15:06
  • Ok I removed both `sys.path.append()` and `import test_project`. I wrote exactly the same code you've suggested, but I have exactly the same error. – v_lfr Jun 17 '16 at 12:38
  • the code looks correct. As I said SyntaxError is not ImportError. The likely reason is that you are trying to run Python 2 script using python3 executable (your parent script uses Python 3.1 judging by the traceback—`sys.executable` is the same python executable as used by the parent) Your last example uses `'python'` that may refer to Python 2 executable. Anyway, it is a separate question (the original issue that `test_script.py` is not found is fixed: if you replace the `test_script.py` content with `print("hello world")` you should see that the script is found and run successfully). – jfs Jun 17 '16 at 12:52
  • Also, it may be Python 3.1 issue (you shouldn't use this version—weird bugs or the lack of the expected functionality are possible). Use Python 2.7, Python 3.4, or the current Python 3.5 instead. – jfs Jun 17 '16 at 12:58

2 Answers2

0

To run an "external project script", install it e.g.: pip install project (run it in a virtualenv if you'd like) and then run the script as you would with any other executable:

#!/usr/bin/env python
import subprocess

subprocess.check_call(['executable', 'arg 1', '2'])

If you don't want to install the project and it has no dependencies (or you assume they are already installed) then to run a script from a given path as a subprocess (assuming there is module1/__init__.py file):

#!/usr/bin/env python
import subprocess
import sys

subprocess.check_call([sys.executable, '-m', 'module1.test_script'], 
                      cwd='/abs/path/to/test_project')

Using -m to run the script, avoids many import issues.

The above assumes that test_project is not a Python package (no test_project/__init__.py). If test_project is itself a Python package then include it in the module name and start from the parent directory instead:

subprocess.check_call([sys.executable, '-m', 'test_project.module1.test_script'], 
                      cwd='/abs/path/to')

You could pass the path to test_project directory as a command-line parameter (sys.argv) or read it from a config file (configparser,json). Avoid calculating it relative to the installation path of your parent script (if you have to then use pkgutil, setuptools' pkg_resources, to get the data). To find a place to store user data, you could use appdirs Python package.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • I don't understand why do I have to install the project ? – v_lfr Jun 13 '16 at 15:00
  • @v_lfr how do you expect project's dependencies to appear on your computer? – jfs Jun 13 '16 at 15:02
  • Actually I have 2 projects. One is going to be a compiled tool, which ask users to put other python project link with test scripts to execute. So i'm gonna have 1 compiled project with many python project which will already be on users computers. Corresponding project dependencies will be automatically add to `PYTHONPATH`when the user will put the project link. Hope I'm understandable.... @J.F.Sebastian – v_lfr Jun 13 '16 at 15:27
  • @v_lfr: 1- don't put the info necessary to answer your question into the comments. Update your question instead. 2- I don't understand what "compiled project" means. What does it mean *"with many python project which will already be on users computers"*? Are you running the scripts remotely via ssh? What does it mean "project link"? If you use your own terminology; you have to provide examples or define the terms. If you have to ask how to implement a `sys.path.append()`-based solution; you won't be able to debug the corresponding issues that arise. Use standard tools instead. – jfs Jun 13 '16 at 15:57
  • A compiled project is a python project compiled into a .exe programm. It means that people will have several projects in their own python IDE. Let's say I have 3 (p1, p2, p3) projects on my python IDE, and I want to run p1 main script. I have to launch the tool, and put the relative path of either p1 project and main script. Then the tool shall run the main script without any import issues regarding to p1 project. that's why I tried to fix that with `PYTHONPATH`and `sys.path.append()`@J.F.Sebastian – v_lfr Jun 14 '16 at 14:47
  • @v_lfr: does the code with `cwd` parameter work for you? If not; what is the exact error? (copy-paste it) Don't put it in the comments. Update your question with a minimal code example that causes the error instead. – jfs Jun 14 '16 at 15:12
  • It didn't work, I've just updated my question @J.F.Sebastian – v_lfr Jun 14 '16 at 15:53
0

There are 3 problems I see that you have to fix:

  1. In order to import from module2, you need to turn that directory into a package by placing an empty *__init__.py file in it
  2. When executing test_script.py, you need the full path to the file, not just file name
  3. Fixing up sys.path only work for your script. If you want to propagate that to test_script.py, set the PYTHONPATH environment variable

Therefore, my solution is:

import os
import sys
import subprocess

# Here is one way to determine test_project
# You might want to do it differently
script_dir = os.path.abspath(os.path.dirname(__file__))
test_project = os.path.normpath(os.path.join(script_dir, '..', 'test_project'))

python_path = '{}:{}'.format(os.environ.get('PYTHONPATH', ''), test_project)
python_path = python_path.lstrip(':')  # In case PYTHONPATH was not set

# Need to determine the full path to the script
script = os.path.join(test_project, 'module1', 'test_script.py')
proc = subprocess.Popen(['python', script], env=dict(PYTHONPATH=python_path))
Hai Vu
  • 37,849
  • 11
  • 66
  • 93
  • 1. I didn't mention it but I actually already have a init.py in my module2. 2. I also tried to put the full path but it didn't work anyway. 3.if i'm right, setting the PYTHONPATH environment variable is the same as `sys.path.append(test_project)` ? I tried your solution but it didn't work, I still have the same error @HaiVu – v_lfr Jun 13 '16 at 15:05
  • For #2, does **didn't work** mean script not found, or not being able to import? For #3, `sys.path.append(...)` is not the same as setting up PYTHONPATH, I have tested and confirmed that in my script. – Hai Vu Jun 13 '16 at 15:52
  • There are at least two issues: 1- how does it install `test_project` dependencies? 2- it runs a script from *inside* a Python package directly (that you should not do). if a person can't configure PYTHONPATH by themselves then it would be hard to resolve [the import issues](https://www.python.org/dev/peps/pep-0395/#traps-for-the-unwary) that arise too. Unrelated: you could use `os.pathsep` and [`get_script_dir()`](http://stackoverflow.com/a/22881871/4279) (though fishing for files relative to your installation directory doesn't look nice—perhaps, `appdirs` may provide more appropriate paths). – jfs Jun 13 '16 at 16:08
  • @HaiVu it means it was not able to import. @J.F.Sebastian the idea is that once you enter the project relative path, `PYTHONPATH`is automatically updated with that path. The tool shall be able to run any script in this path. For instance, when you run a python script from the shell, there is no dependencies problem, all imports are managed to make this script work. Let's see my tool as a shell for python scripts – v_lfr Jun 14 '16 at 14:58