How to make Python package update within a code to take effect on-the-fly

Question

I have the following line of codes:

from pip import main as pipmain

# initial installation
pipmain(["install", "pyscenic==0.10.0"])
import pyscenic
pyscenic.__version__

# return 0.10.0
# Some large code here

# second installation
pipmain(["install", "install", "pyscenic==0.10.4"])
import pyscenic
pyscenic.__version__
# still return 0.10.0

# Another large chunk that required new version

There I want to upgrade the pyscenic package on-the-fly inside my code. However as I noted above, in the second installation the version still doesn't change. I expect it to change to 0.10.4. How can I do it properly?

I also tried this, still no avail:

import os
import importlib
os.system('pip install pyscenic==0.10.0')
import pyscenic
pyscenic.__version__
os.system('pip install pyscenic==0.10.4')
import pyscenic
pyscenic.__version__
importlib.reload(pyscenic)
pyscenic.__version__

All code tested on IPython (interactive). If I exit the IPython and re-enter again it will take effect. But that's not what I want.

Answer 1

As several previous answers (1, 2) and recent requests have mentioned, pip and Python weren't really designed with this in mind.

But with some clever hacking of Python's name system, and some knowledge of the package you want to work with, you could install two versions next to each other:

# Copyright © 2021 Alexander L. Hayes
# MIT License

git clone git@github.com:aertslab/pySCENIC pySCENIC100
git clone git@github.com:aertslab/pySCENIC pySCENIC104

(
  cd pySCENIC100
  git checkout 0.10.0
  sed -i "s/pyscenic/pyscenic100/g" setup.py
  sed -i "s/pyscenic/pyscenic100/g" MANIFEST.in
  sed -i "s/pyscenic/pyscenic100/g" setup.cfg
  (
    cd src
    mv pyscenic pyscenic100
    (
      cd pyscenic100
      sed -i "s/pyscenic/pyscenic100/g" binarization.py
      sed -i "s/pyscenic/pyscenic100/g" __init__.py
      sed -i "s/pyscenic/pyscenic100/g" _version.py
    )
  )
  python setup.py install
)

(
  cd pySCENIC104
  git checkout 0.10.4
  sed -i "s/pyscenic/pyscenic104/g" setup.py
  sed -i "s/pyscenic/pyscenic104/g" MANIFEST.in
  sed -i "s/pyscenic/pyscenic104/g" setup.cfg
  (
    cd src
    mv pyscenic pyscenic104
    (
      cd pyscenic104
      sed -i "s/pyscenic/pyscenic104/g" binarization.py
      sed -i "s/pyscenic/pyscenic104/g" __init__.py
      sed -i "s/pyscenic/pyscenic104/g" _version.py
      (
        cd cli
        sed -i "s/pyscenic/pyscenic104/g" *.py
      )
    )
  )
  python setup.py install
)

This bash script clones the repository twice, checks out versions 0.10.0 and 0.10.4, does some renaming via sed, and finally installs two libraries named pyscenic100 and pyscenic104:

import pyscenic100
import pyscenic104

print(pyscenic100.__version__)
print(pyscenic104.__version__)
# 0.10.0+0.g3de37cb.dirty
# 0.10.4+0.g436561f.dirty

I don't know what happens during "# Some large code here", but it looks like examples from the documentation/tests work:

from pyscenic100.featureseq import Feature as Feature100
from pyscenic104.featureseq import Feature as Feature104


f1 = Feature100.from_string('chr1 12 50 feature1 10.0 +')
f2 = Feature100.from_string('chr1 40 60 feature2 10.0 -')
print(f1.has_overlap_with(f2))
# True

f1 = Feature104.from_string('chr1 12 50 feature1 10.0 +')
f2 = Feature104.from_string('chr1 40 60 feature2 10.0 -')
print(f1.has_overlap_with(f2))
# True

Answer 2

This is not the preferred way to install packages, ie installing them within the scripts. Most of the packages even after updating have all the features that were in the older versions and since your version change is minor it shouldn't create a lot of issues.

However if you want to install packages within a script you can use os.system as you mentioned, but before installing the other version, first remove the previous version and you should be good to go.

If that doesn't solve your problem then you can look at this code. This should solve your problem of installing and uninstalling packages within a script.

import subprocess
import sys

def install(package):
    subprocess.check_call([sys.executable, "-m", "pip", "install", package])

def uninstall(package):
    subprocess.check_call([sys.executable, "-m", "pip", "uninstall", package])

Do let me know if it works

Answer 3

If you are able to, a simple solution would be to write a bash file running sequentially two python files, with a change of version of the package pyscenic between the two python runs using pip. You can do something similar to this, like:

#!/usr/bin/env bash
pip install pyscenic==0.10.0
python first_script.py
pip install pyscenic==0.10.4
python second_script.py