0

I have a python script that I use to analyze data. I rely on number crunching packages like numpy and others to work with my data. However, the packages constantly evolve and some functions depreciate, etc. This forces me to go through the script several times per year to fix errors and make it work again.

One of the solutions is to keep an older version of numpy. However, there are other packages that require a new version of numpy.

So the question I have is: Is there a way to 1) keep multiple versions of a package installed or 2) have a local copy the package located in the directory of my script so I am in control what I am importing. For example, I can have my own package where I will have all the different packages and versions I need.

Later, I can simply import libraries I want

from my_package.numpy_1_15 as np115
from my_package.numpy_1_16_4 as np1164

and later in my code, I can decide which function to use from which numpy version. For example:

index = np115.argwhere(x == 0)

This is my vision of the solution to my problem where I want to keep using old functions from previous versions of numpy (or other libraries). In addition, in this way, I can always have all the libraries needed with me in my script directory. So, if I need to run the script on a different machine I don't need to spend hours figuring out if everything is compatible.

Here are possible proposed solutions and why they do not solve my problem.

  1. Virtual Environments in Python or Anaconda.

There are a bunch of introductions (for example) available that explain how to use them. However, virtual environments require maintenance and initial setup. Imagine, if I can just have a python code that performs well a specific computational task independent on what year it is and what packages are installed on any machine. The code can be shared among different research groups and will always work.

  1. python create standalone executable linux

I can create standalone executable (example). However, it will be compiled​ and cannot be dynamically changed the really nice feature of Python

Valentyn
  • 659
  • 1
  • 7
  • 28
  • 1
    Do you know about venvs or e.g. Anaconda? This tools provide exactly what you are looking for. Multiple environments with different packages installed and dependency graph tracking. – Dschoni Apr 02 '19 at 14:37
  • Mixing of different versions of the same package however seems to be a bad idea (citation needed). – Dschoni Apr 02 '19 at 14:38
  • I understand that the code might become messy if I have multiple versions of numpy (for example) used in one code. However, it will make it last forever and I will not need to fix all errors and warning that appear with every new package version. – Valentyn Apr 02 '19 at 14:51
  • The problem I see in this approach, that dependency management of this code basis is going to be horrible. Let's say, you have not only numpy, but also any other package, that depends on numpy. Sometimes this package will rely on having a certain version of numpy. You might break your programm if you import different versions at the same time. The other way woul be version pinning. You could fix a version of any package using a `requirements.txt` with e.g. `anaconda`. Do you have a real-world example where you need a deprecated function? – Dschoni Apr 02 '19 at 14:59
  • With your edit, I'm pretty sure you don't fully understand what Anaconda is aiming at. It allows you to reproduce a complete python environment with all dependencies and correct matching versions on any other pc. At any time. You could also deploy your package to e.g. Pypi with version pinning. – Dschoni Apr 02 '19 at 15:02
  • Dschoni, I don't want to force my choice of python distribution on someone else. I need to be able to go to a different place open my script on a different computer and simply start crunching data. – Valentyn Apr 02 '19 at 15:06
  • Okay, I see. What you can do is, you simply take every function (and C module...) you want from the packages you are referring to. So, instead of importing a module, you would copy the source code to your source code. So basically you embed all functions from external modules and packages directly in your source. That works as long as you do not rely on compiled code (e.g. numpy). – Dschoni Apr 02 '19 at 15:10

0 Answers0