2

The installation of the virtual environment of my python application takes too much time during deployment due to a large amount of dependencies. In order to minize that time, I want to include the dependencies residing in the virtual environment in git, so that they are already there on deployment.

The main issue with that is that dependencies with C code need to be rebuild because of architecture differences between machines.

Is there a way to rebuild all dependencies that need compilation in my virtual environment?

Dan Milon
  • 2,509
  • 2
  • 18
  • 17

1 Answers1

1

wheel format is what you need

Popular example is lxml, which when installing from source on Linux takes about 3 minutes to get downloaded, compiled and installed.

Using wheel format and installing from local wheel file for lxml installs within fraction of a second.

For detailed instructions how I use it see corrected link to Detailed SO answer how to configure pip incl. instructions how to take advantage of wheels

For more information:

Some notes:

  • pure python packages can be distributed in wheel format regardless of target platform (apart from being possibly python version dependent).

  • compiled python packages shall be build on the same platform, where you are going to install them. There might be some cross-compile options, but I do not have real experience with that.

  • some do consider wheel of "package format of the future", others claim, that it is supposed to be build on your own side and use your own wheels. The later case is for lxml not being provided as a wheel - see launchpad issue related to lxml in wheel format. Consider adding there yourself as an affected person, if you care.

Once you manage using wheels the first time, you will love it.

Community
  • 1
  • 1
Jan Vlcinsky
  • 42,725
  • 12
  • 101
  • 98
  • Thanks for the information, but that does not seem to solve the issue. A .whl file is already compiled, right? Thus I cant distribute it directly to other machines with different architecture. – Dan Milon Apr 23 '14 at 12:26
  • wheel allows different distributions for different platforms, the platform becomes part of wheel file name, see http://legacy.python.org/dev/peps/pep-0427/#id11 . Having multiple platforms, compilations must happen somewhere. You can try cross-compiling, or build wheels on respective platforms. All the wheels for all supported platforms can be then stored in your repo and used to install on specific platform, where it will pick proper platform variant of a wheel. – Jan Vlcinsky Apr 23 '14 at 13:01
  • I integrated wheel format with our CI server which is the same architecture as our production machines. While it isn't the answer to the question, installation time is much faster. Thank you. – Dan Milon Apr 24 '14 at 08:01