4

I'm looking for others that have experience with python venv and deployment of applications via rpm. My goals are to:

  1. Use pip for python dependency
  2. Use venv to keep application environments/deps separate
  3. Use rpm for deployment (required by our companies internal audit etc).

I have a build server (jenkins slave) for each architecture (read: distro) that we deploy on. My original (and only) plan on the jenkins slave(build server) via a jenkins job was to:

  1. Create a venv
  2. Activate venv
  3. python setup.py build/install (within rpm spec)
  4. archive rpm as artifact
  5. Rejoice

I never got to step 2 or 3 so I do not know the dragons there, however the main issue comes with "Create a venv" step. Since venvs are not "relocatable" and RPM uses a RPM_BUILD_ROOT, which is a self contained filesystem in a tmpdir that we package from, I can't install the venv into the rpm_build_root. I'd have to install the venv into the ACTUAL location on the build server it was going to be when we deploy (install rpm). This is not ideal for a lot of reasons you may be able to guess (collisions with other applications, other stuff running on build servers etc).

I don't want to run setup.py on my production box and download packages at install time. I want to make sure all is well, have everything downloaded and packaged before deploy happens.

The closest thing I have found is dh-virtualenv from this so question. This looks promising and from what I can tell installs directly into the final directory (not a temp build). It cleans up after itself but still seems bad practice. Is there a better way? Am I missing something? Seems I'm stuck doing it the spotify way.

mikew
  • 912
  • 2
  • 11
  • 22
  • Why do you want to bother with pip and venv? You can use pyp2rpm that will build directly rpm from pypi. You will "just" check if it has some missing dependencies and build them as well. Again using pyp2rpm. – msuchy Sep 24 '19 at 23:26
  • 1
    @msuchy have not researched this in depth, but I did not because I do not one rpm per pypi package. I do not want rpm and pip managing deps. Just pip. I dont want 20 rpms to support my package. I'm sure I'd have to munge some of the specs to support the non-os python deployed etc. – mikew Sep 25 '19 at 13:22

1 Answers1

3

I don't know what is the best way, but your idea is very close to what we do at our company:

  • create a relative virtualenv (that can be relocated and is still functional). I have rendered the script public in this gist
  • then package the whole virtualenv, changing also the shebang of scripts such that they are invoked with the correct virtualenv.

We would have something like this in our spec files:

%build
create-relative-virtualenv -p . -v 3.6 -r Pipfile

%install
# Create directories
install -d -m 0755 "${RPM_BUILD_ROOT}/usr/lib/application"

cp -r virtualenv ${RPM_BUILD_ROOT}/usr/lib/application/
cp script.py ${RPM_BUILD_ROOT}/usr/lib/application/
# set the correct shebang for script
sed -i "1s@.*@#\!/usr/lib/application/virtualenv/bin/python3@" ${RPM_BUILD_ROOT}/usr/lib/application/script.py
ln -s /usr/lib/application/script.py ${RPM_BUILD_ROOT}/usr/bin/script

%check
./virtualenv/bin/python3 -m coverage run --branch tests/run_tests.py

%files
/usr/lib/application
/usr/bin/script

Nice detail here is that the tests run with the same virtualenv that is packaged.

Chris Maes
  • 35,025
  • 12
  • 111
  • 136
  • This is interesting. I don't think it was your gist i had run across while researching because yours looks much cleaner than what I remember, but I am not familiar enough with the details to know that updating the shebang lines was enough. I had read there was more to it than that when I was reading about the old '--relocatable' flag ([relevant so question](https://stackoverflow.com/questions/6628476/renaming-a-virtualenv-folder-without-breaking-it)) that used to be available before venv. Have you run into any edge cases where this solution doesn't work? – mikew Sep 25 '19 at 13:24
  • [Relevant docs](https://virtualenv.pypa.io/en/stable/userguide/#making-environments-relocatable) for the reloctable flag from the old virtualenv system. It did not carry over to venv so I'm wondering why? This is what drove me away from this solution – mikew Sep 25 '19 at 13:32
  • 1
    We are using this sucessfully for over a year, we don't know any edge cases for the moment. But I can't guarantee you this is the best possible way... – Chris Maes Sep 25 '19 at 14:04