1

Asked a similar question before, but I marked it answered, and I have other information.

Here is the structure:

engine-load-tests\
    |-src\
        |- __init__.py
        |- main.py
            |- on_locust_init()
        |
        |- win_perf_counters
                |- __init__.py
                |- main.py
                    |- main([])
    |- __init__.py
    |- .gitlab-ci.yml

engine-load-tests\src\main.py:

    import src.win_perf_counters.main as wmi_counters

...
    def on_locust_init():
        pid = wmi_counters.main([conf_file, database])
Also tried ...
    from src.win_perf_counters.main import main
...
    pid = main([conf_file, database])

win_perf_counters:
    class WinPerf:
        def x:
            ...
        def y:
            ...

    def main(args):
        // code

    if __name__ == "__main__":
        import sys

        main(sys.argv[1:])

This may not be good form, but it always used to work, and it continues to work when I run in PyCharm.

When I run in Gitlab CI, I get:

  File "C:\GitLab-Runner\builds\eTJPJvW5\0\qa\engine-automation\engine-load-tests\src\main.py", line 14, in <module>
    import src.win_perf_counters.main as wmi_counters
ModuleNotFoundError: No module named 'src.win_perf_counters.main'

I've spent all day reading the PythonDoc, but I am still not seeing what I am missing.

I have a line in the CI YML set PYTHONPATH=%PYTHONPATH%:., but that does not help.

I would think that if there is a fundamental flaw in my imports, it should fail in PyCharm as well.

I wonder if it has to do with the fact that I am trying to include main.

Suggestions for better Python design welcome and appreciated.

Also works on DOS command line.

Gitlab YAML

(I know it would be much faster to make this a Docker image instead of running pip every time.)

test:
  before_script:
    - python -V
    - pip install virtualenv
    - virtualenv venv
    - .\venv\Scripts\activate.ps1
    - refreshenv
    - python -m pip install --upgrade pip
    - python -m pip install locust
    - locust -V
    - python -m pip install multipledispatch
    - python -m pip install pycryptodome
    - python -m pip install pandas
    - python -m pip install wmi
    - python -m pip install pywin32
    - python -m pip install influxdb_client    
  script:
  - set LOAD_TEST_CONF=load_test.conf
  - set PYTHONPATH=%PYTHONPATH%:%CI_PROJECT_DIR%\src
  - echo "**** about to run locust ******"
  - locust -f ./src/main.py --host $TARGET_HOST -u $USERS -t $TEST_DURATION -r $RAMPUP -s 1800 --headless --csv=./LoadTestsData_VPOS --csv-full-history --html=./LoadTestsReport_VPOS.html  --stream-file ./data/stream_jsons/streams_vpos.json --database=csv
  
  - Start-Sleep -s $SLEEP_TIME

  variables:
    LOAD_TEST_CONF: load_test.conf
    PYTHON_VERSION: 3.8.0
    TARGET_HOST: http://x.x.x.x:9000

  tags:
    - win2019
  artifacts:
    paths:
      - ./LoadTests*
      - public
  only:
    - schedules

  after_script:
    - mkdir .public
    - cp -r ./LoadTests* .public 
    - cp metrics.csv .public -ErrorAction SilentlyContinue
    - mv .public/* public
Guy
  • 666
  • 1
  • 10
  • 34
  • where is gitlabs yaml file placed at? – sudden_appearance Feb 06 '22 at 12:53
  • @sudden_appearance - At root, updated stick diagram. \engine-load-tests\.gitlab-ci.yml. Interesting question - would be neat if that was it. – Guy Feb 06 '22 at 13:36
  • Please include the YAML for your GitLab job. PyCharm adds source directories to PythonPath automatically. So that's probably why it works in PyCharm. `set PYTHONPATH=%PYTHONPATH%:.` won't work in the GitLab because your source is in `src` -- not `.` -- Maybe try `set PYTHONPATH=%PYTHONPATH%:%CI_PROJECT_DIR%\src` – sytech Feb 08 '22 at 00:49
  • @sytech - thanks for the suggestion, but that didn't get it right either :(. I'll include the YAML. – Guy Feb 08 '22 at 07:21
  • the `main.py` is inside `src`. Why do you adding `src`? Isn't it should be: `import win_perf_counters.main as wmi_counters` – rzlvmp Feb 08 '22 at 07:37
  • @rzlvmp - You would think so, but when I use `import win_perf_counters.main as wmi_counters`, I get ` File "C:\git\engine-load-tests\src\main.py", line 194, in on_locust_init from win_perf_counters.main import main ModuleNotFoundError: No module named 'win_perf_counters' ` – Guy Feb 08 '22 at 07:44
  • What happens if change `cwd` to `src` (`cd ./src`) and try to import wothout `src` again? – rzlvmp Feb 08 '22 at 07:48
  • When I do that, there is an error from a different module saying no module named `src`. – Guy Feb 08 '22 at 08:03
  • Also your `PYTHONPATH` is seems to be not correct. [separator mark](https://stackoverflow.com/a/21433154/13946204) is `;` not `:`. – rzlvmp Feb 08 '22 at 08:04
  • `different module saying no module named src` → this is expected :) If you removed `src` in one place, you should update all other parts in the same way – rzlvmp Feb 08 '22 at 08:10
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/241812/discussion-between-guy-and-rzlvmp). – Guy Feb 08 '22 at 08:46

1 Answers1

1

Since you're doing

set PYTHONPATH=%PYTHONPATH%:%CI_PROJECT_DIR%\src

It will try and execute from /src. You're also doing

src.win_perf_counters.main as wmi_counters

So in turn you're looking for wmi_counters in /src/src/

Python imports are dependant on where the script is ran from. So either remove the \src in PYTHONPATH or in the import

Hampus
  • 276
  • 2
  • 8
  • I follow your logic. I attempted both. (a) I removed `\src` from the *PYTHONPATH*. I kept the command the same. (b) I retained `\src` in *PYTHONPATH*. Removed `src.` from the import. I either got `src.win.... not found` or `win_perf... not found`. – Guy Feb 14 '22 at 13:30
  • OK. I branched, played around, and finally got it to work **by first `cd src` then running**. Had to change several other imports. But what is frustrating is that PyCharm shows errors, when I could run it fine on the command line outside of PyCharm. But that is a different issue, which I am sure is dealt with many times. – Guy Feb 14 '22 at 14:24