0

I have a Jenkins job pulling python files from different Gitlab repositories within my Jenkins project WORKSPACE. Some of those scripts are supposed to be imported by others but I'm facing errors when using relative imports.

ValueError: attempted relative import beyond top-level package

script1.py

from ..script2/script2 import foo

script2.py

def foo():
    print('Foo!')

File/directory hierarchy

/var/lib/jenkins/workspace/project_name
    script1/
        script1.py
    script2/
        # script2.py contains 'foo' function
        script2.py

Jenkins 'Execute shell' build step

#!/bin/bash

python3 -m venv ${WORKSPACE}/venv
. ${WORKSPACE}/venv/bin/activate

python ${WORKSPACE}/script1/script1.py

What have I missed?

donmelchior
  • 893
  • 3
  • 13
  • 31
  • What I do in my projects is to define a setup.py installing the package and then writing separate tests where I can now import the scripts without problems. (```from mypackage import script1```) - if this sounds interesting I can write an answer with links to the resources I used – clotodex Mar 11 '20 at 10:47
  • @clotodex: This seems interesting, could you post the specifics? Thanks. – donmelchior Mar 11 '20 at 12:52
  • @napuzba: Your website is interesting to understand the mechanics behing imports. How could this could applied in the current use case? For obvious reasons I can't modify the Jenkins project workspace directory and all my python files come from git clone within their own (sub)subdirectory. – donmelchior Mar 11 '20 at 12:57
  • Unfortunately, I do not have a solution without changing the directory structure :-( – napuzba Mar 11 '20 at 18:40
  • also you should change ```from ..script2/script2 import foo``` to ```from ..script2.script2 import foo``` – clotodex Mar 12 '20 at 14:08

1 Answers1

1

Tests should ideally be done in a "clean environment". So instead of running script tests with relative imports, one solution is to write a setup.py to install your project locally as a module. Then you can write separate tests where you can do from mymodule import script1.script1.

If you go that path you need to decide on a project structure. There is no best structure out there (see here). So as an example, here is my typical structure:

├── src/
│   ├── script1/
│   │   └── script1.py
│   └── script2/
│       └── script2.py
├── tests/
│   ├── test_script1.py
│   └── test_script2.py
└── venv/

and a setup.py like this:

import os

from setuptools import find_packages, setup

# load desc from readme
def read(fname):
    return open(os.path.join(os.path.dirname(__file__), fname)).read()

setup(
    name="mymodule",
    version="0.0.1",
    author="you",
    author_email="<you@you.com>",
    description=("short description"),
    license="<none>",
    packages=find_packages("src"),
    package_dir={"": "src"},
    long_description=read("README.md"),
    classifiers=[
        "Development Status :: 2 - Pre-Alpha",
        "Topic :: Utilities",
        # "License :: OSI Approved :: MIT License",
    ],
    install_requires=[],
)

with this example you can now run python setup.py install to install mymodule and then run all tests with pytest tests. This not only tests your tests but also checks that your package ist correctly installed/installable. When using a virtual environment the setup.py will install here instead of the full system.
Finally you can use relative imports in all of your files since you will have a top level module "mymodule" and your error will disappear.

PS: If you dont like to import script1.script1 you can write __init__.py files in the directories to be able to do from mymodule import script1 or even from mymodule import ClassFromScript1

clotodex
  • 193
  • 2
  • 10