How does one create a Python virtualenv in Jenkins?

Question

I am using a Makefile to provide consistent single commands for setting up a virtualenv, running tests, etc. I have configured my Jenkins instance to pull from a mercurial repo and then run "make virtualenv", which does this:

virtualenv --python=/usr/bin/python2.7 --no-site-packages . && . ./bin/activate && pip install -r requirements.txt

But for some reason it insists on using the system-installed pip and trying to install my package dependencies in the system site-packages rather than the virtualenv:

error: could not create '/usr/local/lib/python2.7/dist-packages/flask': Permission denied

If I add some debugging commands and explicitly point to the pip in my virtualenv, things get even more confusing:

virtualenv --python=/usr/bin/python2.7 --no-site-packages . && . ./bin/activate && ls -l bin && which pip && pwd && ./bin/pip install -r requirements.txt

Which generates the following output:

New python executable in ./bin/python2.7
Not overwriting existing python script ./bin/python (you must use ./bin/python2.7)
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python2.7

It appears Jenkins doesn't rebuild the environment from scratch for each build, which strikes me as an odd choice, but shouldn't effect my immediate issue

The output from the "ls -l bin" shows pip to be installed in the virtualenv and executable:

-rw-r--r-- 1 jenkins jenkins    2248 Apr  9 21:14 activate
-rw-r--r-- 1 jenkins jenkins    1304 Apr  9 21:14 activate.csh
-rw-r--r-- 1 jenkins jenkins    2517 Apr  9 21:14 activate.fish
-rw-r--r-- 1 jenkins jenkins    1129 Apr  9 21:14 activate_this.py
-rwxr-xr-x 1 jenkins jenkins     278 Apr  9 21:14 easy_install
-rwxr-xr-x 1 jenkins jenkins     278 Apr  9 21:14 easy_install-2.7
-rwxr-xr-x 1 jenkins jenkins     250 Apr  9 21:14 pip
-rwxr-xr-x 1 jenkins jenkins     250 Apr  9 21:14 pip2
-rwxr-xr-x 1 jenkins jenkins     250 Apr  9 21:14 pip2.7
lrwxrwxrwx 1 jenkins jenkins       9 Apr 10 19:31 python -> python2.7
lrwxrwxrwx 1 jenkins jenkins       9 Apr 10 19:31 python2 -> python2.7
-rwxr-xr-x 1 jenkins jenkins 3349512 Apr 10 19:31 python2.7

The output of "which pip" seems to want to use the correct one:

/var/lib/jenkins/jobs/Run Tests/workspace/bin/pip

My current working directory is what I expect it to be:

/var/lib/jenkins/jobs/Run Tests/workspace

But... wtf?

/bin/sh: 1: ./bin/pip: Permission denied
make: *** [virtualenv] Error 126
Build step 'Execute shell' marked build as failure
Finished: FAILURE

score 16 · Answer 1 · edited Jul 20 '23 at 21:13

16

I have been using python virtualenvs with Jenkins every day in the last two years, at multiple companies and for small side projects and cannot say I found "THE" answer. Still, I hope that sharing my experience will help others save time. Hopefully I will get further feedback in order to make the decision easier.

Avoid ShiningPanda - it's not well maintained, incompatible with Jenkins2 pipelines and prevents execution of jobs in parallel. Also it has the bad habit of leaving orphan environments on disk.
DIY via bash and virtualenv is my current favourite. Create it inside $WORKSPACE and, if not always cleaning, run relocatable before activating them. This is because jenkins workspace folder disk location can change between executions of job N and N+1.

If you use multiple builders that do need the same virtualenv, the easiest way is to dump your environment to a file and source it at the beginning of the new builder.

To ease the maintenance I am planning to investigate these:

direnvm
virtualenv-wrapper (mkvirtualenv)
pyenv

If you hit the shebang command line limits the best thing to do is to change your jenkins home directory to just /j.

edited Jul 20 '23 at 21:13

Adrian W

4,563
11
38
52

answered Mar 29 '17 at 19:07

sorin

161,544
178
535
806

Agreed, same deal for windows.. simple batch file directly entered into the "run windows batch file" works great. Then just use python to grab environment variables or arguments and take it from there. – Duane Sep 01 '18 at 00:22
4

I don't get how to do it. Do you have a basic example? I try it like this: sh 'python3 -m virtualenv env' sh '. ./env/bin/activate && pip install conan' but it doesnt install anything... – Zack Mar 18 '19 at 19:17
4

Please add an example to the post, otherwise the post is not useful at all. – voliveira89 Sep 16 '21 at 12:20
4

This is not a useful stackoverflow answer - it's more like bragging "i can do it but i won't tell you poor guys how to do it" There should be a multi-downvote option for answers like this. – Wolfgang Fahl Feb 26 '22 at 10:11
1

the relocatable link is broken – Wolfgang Fahl Feb 26 '22 at 11:49
I have resurrected the link to "relocatable". But please note that this refers to a "legacy" version of the documentation and "relocatable" has always been marked as "experimental" and never been guaranteed to work in all circumstances. In the [current version](https://virtualenv.pypa.io/en/stable/user_guide.html), I couldn't find any reference to "relocatable" at all. – Adrian W Jul 20 '23 at 21:26

score 14 · Answer 2 · answered Jun 08 '17 at 04:00

Jenkins pipelines can be made to run with virtual environments but there are multiple things to consider.

The default shell that Jenkins uses is /bin/sh - this is configurable in Manage Jenkins -> Configure System -> Shell -> Shell executable. Setting this to /bin/bash will make source work.
An activated venv simply changes environment variables, and environment variables do not persist between stages in jenkins. See withEnv
If you are using version controlled multibranch pipelines jenkins creates a workspace with the branch name and a commit hash in the path - which can be quite long. venv scripts (e.g. pip) all start with a hashbang line which includes the full path to the python interpreter in the venv (the python interpreter itself is a symlink). E.g.,
```
~/workspace/ink_feature-use-jenkinsfile-VGRPYD53GGGDDSBIJDLSUDYPJ34QR63ITGMC5VJNB56W6ID244AA/env/bin$ cat pip
#!/var/jenkins_home/workspace/ink_feature-use-jenkinsfile-VGRPYD53GGGDDSBIJDLSUDYPJ34QR63ITGMC5VJNB56W6ID244AA/env/bin/python3.5
```
Bash only reads the first N characters of any executable file - which I found did not quite include the full venv path:
```
bash: ./pip: /var/jenkins_home/workspace/ink_feature-use-jenkinsfile-VGRPYD53GGGDDSBIJDLSU: bad interpreter: No such file or directory
```
This particular problem can be avoided by executing the script with Python instead. E.g. python3.5 ./pip

score 6 · Answer 3 · edited Jun 02 '20 at 02:13

I'd recommend avoiding ShiningPanda.

I set up my virtual environments with Anaconda/Miniconda. When installing conda make sure you're running as jenkins user.

your_user@$ sudo -u jenkins sh
jenkins@$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
jenkins@$ bash Miniconda3-latest-Linux-x86_64.sh

Since Jenkins runs sh rather than bash, I added conda path to /etc/profile:

export PATH="/var/lib/jenkins/miniconda3/bin:$PATH"

Then in Jenkinsfile you can create and delete conda environments. Here's an example that creates a new environment for each build:

pipeline {
    agent any
    stages {
        stage('Unit tests') {
            steps {
            sh '''
                conda create --yes -n ${BUILD_TAG} python
                source activate ${BUILD_TAG}
                // example of unit test with nose2
                pip install nose2
                nose2
            '''
            }
        }
    }
    post {
        always {
            sh 'conda remove --yes -n ${BUILD_TAG} --all'
        }
    }
}

score 2 · Answer 4 · answered Aug 05 '16 at 23:48

2

I've got same problem. As I can see - you project named 'Run Tests'. So, this name contain space. That was the problem for me. I just renamed project, like RunTests - and venv working now! Attention - jenkins ask you about confirmation renaming project.

answered Aug 05 '16 at 23:48

Talkerbox

385
1
4
15

2

This solved my problem... Now it is the time for a big WTF. – davide Jun 21 '17 at 09:01

score 1 · Answer 5 · edited Jun 23 '22 at 17:26

There are some issues with venv-python plugin with different OS environments.

Here is how I call python method manually. Not best practice but it work.

// Put this stage on top of pipeline
stage('Prepare venv') {
    steps {
        script {
            if (isUnix()) {
                env.ISUNIX = "TRUE" // cache isUnix() function to prevent blueocean show too many duplicate step (Checks if running on a Unix-like node) in python function below
                sh 'python3 -m venv pyenv'
                PYTHON_PATH =  sh(script: 'echo ${WORKSPACE}/pyenv/bin/', returnStdout: true).trim()                        
            }
            else {
                env.ISUNIX = "FALSE"
                powershell(script:"py -3 -m venv pyenv") // windows not allow call python3.exe with venv. https://github.com/msys2/MINGW-packages/issues/5001
                PYTHON_PATH =  sh(script: 'echo ${WORKSPACE}/pyenv/Scripts/', returnStdout: true).trim()
            }

            try  {
                // Sometime agent with older pip version can cause error due to non compatible plugin.
                Python("-m pip install --upgrade pip")
            } 
            catch (ignore) { } // update pip always return false when already lastest version
            // After this you can call Python() anywhere from pipeline
            Python("-m pip install -r requirements.txt")
        }                
    }
}

// Several plugins like WithPyenv is not working perfectly accross platform when using Virtual Env.
// Put this method outside pipeline
def Python(String command) {
    if (env.ISUNIX == "TRUE") {
        sh script:"source ${WORKSPACE}/pyenv/bin/activate && python ${command}", label: "python ${command}"
    }
    else {
        powershell script:"${WORKSPACE}\\pyenv\\Scripts\\Activate.ps1 ; python ${command}", label: "python ${command}"
    }
}

score 0 · Answer 6 · answered Oct 21 '20 at 22:49

After acticating the virtualenv, try to run pip as a module:

python -m pip install ...

python -m pip vs pip

python -m pip: executes python interpreter binary that reads module pip.py from site packages directory
pip: executes pip binary / script picked up from $PATH

I have found that using python -m pip solved most of the pip permission problems encountered.

score 0 · Answer 7 · answered Aug 08 '22 at 13:00

@hardbyte's answer

The default shell that Jenkins uses is /bin/sh - this is configurable in Manage Jenkins -> Configure System -> Shell -> Shell executable. Setting this to /bin/bash will make source work.

plus:

got me working

sudo apt install python3.10-venv

and then in jenkins in the execute shell step:

python3.10 -m venv .venv
source .venv/bin/activate
...

How does one create a Python virtualenv in Jenkins?

7 Answers7