2

I am trying to dockerize a python project. The problem is that docker is unable to find many of the packages that I am importing from requirements.txt file.

My OS is Xubuntu 20.04. My docker version informations are as follow:

Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu1~20.04.1
 Built:             Wed Aug  4 22:52:25 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.7-0ubuntu1~20.04.1
  Built:            Wed Aug  4 19:07:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:

This is my dockerfile:

FROM pypy:3

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "pypy3", "./index.py" ]

The requirements.txt file is :

apturl==0.5.2
attrs==20.3.0
Automat==20.2.0
beautifulsoup4==4.9.3
blinker==1.4
blis==0.7.4
bs4==0.0.1
catalogue==2.0.3
catfish==1.4.13
certifi==2019.11.28
cffi==1.14.5
chardet==3.0.4
click==7.1.2
colorama==0.4.3
command-not-found==0.3
constantly==15.1.0
cryptography==3.4.7
cssselect==1.1.0
cupshelpers==1.0
cymem==2.0.5
dbus-python==1.2.16
defer==1.0.6
distro==1.4.0
distro-info===0.23ubuntu1
elasticsearch==7.13.0
entrypoints==0.3
h2==3.2.0
hpack==3.0.0
httplib2==0.14.0
hyperframe==5.2.0
hyperlink==21.0.0
idna==2.8
incremental==21.3.0
itemadapter==0.2.0
itemloaders==1.0.4
Jinja2==2.11.3
jmespath==0.10.0
joblib==1.0.1
keyring==18.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
lightdm-gtk-greeter-settings==1.2.2
lxml==4.6.3
MarkupSafe==1.1.1
menulibre==2.2.1
mugshot==0.4.2
murmurhash==1.0.5
netifaces==0.10.4
numpy==1.20.2
oauthlib==3.1.0
olefile==0.46
onboard==1.4.1
packaging==20.9
pandas==1.2.4
parsel==1.6.0
pathy==0.5.2
pexpect==4.6.0
Pillow==7.0.0
preshed==3.0.5
priority==1.3.0
Protego==0.1.16
psutil==5.5.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycparser==2.20
pycups==1.9.73
pydantic==1.7.3
PyDispatcher==2.0.5
PyGObject==3.36.0
PyJWT==1.7.1
pymacaroons==0.13.0
pymongo==3.11.3
PyNaCl==1.3.0
pyOpenSSL==20.0.1
pyparsing==2.4.7
python-apt==2.0.0+ubuntu0.20.4.5
python-dateutil==2.7.3
python-debian===0.1.36ubuntu1
pytz==2021.1
PyYAML==5.3.1
queuelib==1.6.1
reportlab==3.5.34
requests==2.22.0
requests-unixsocket==0.2.0
scikit-learn==0.24.1
scipy==1.6.3
Scrapy==2.5.0
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
service-identity==18.1.0
sgt-launcher==0.2.5
simplejson==3.16.0
six==1.14.0
sklearn==0.0
smart-open==3.0.0
soupsieve==2.2.1
spacy==3.0.6
spacy-legacy==3.0.5
srsly==2.4.1
systemd-python==234
thinc==8.0.3
threadpoolctl==2.1.0
tqdm==4.60.0
Twisted==21.2.0
typer==0.3.2
ubuntu-advantage-tools==27.0
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.25.8
w3lib==1.22.0
wadllib==1.3.3
wasabi==0.8.2
xcffib==0.8.1
xkit==0.0.0
zope.interface==5.4.0

I receive the error ERROR: Could not find a version that satisfies the requirement for the following packages:

apturl==0.5.2
catfish==1.4.13
command-not-found==0.3
cupshelpers==1.0
defer==1.0.6
distro-info===0.23ubuntu1
language-selector==0.1
lightdm-gtk-greeter-settings==1.2.2
menulibre==2.2.1
mugshot==0.4.2
onboard==1.4.1
PyGObject==3.36.0
python-apt==2.0.0+ubuntu0.20.4.5
python-debian===0.1.36ubuntu1
screen-resolution-extra==0.0.0
sgt-launcher==0.2.5
systemd-python==234
ubuntu-advantage-tools==27.0
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
xkit==0.0.0

(I tried to eliminate them one by one until the container could be compiled, but of course it did not run because of the absence of these packages).

I also tried to replace pypy by the normal python image, but I received the same error.

I tried to use the following dockerfile based on an ubuntu image :

FROM ubuntu:focal

SHELL ["/bin/bash", "-xe", "-c"]

ARG DEBIAN_FRONTEND=noninteractive

COPY . /code

ADD requirements.txt ./

RUN apt-get update -q \
 && apt-get install -y -q --no-install-recommends \
        python3-wheel \
        python3-pip \
        gunicorn \
 && if [ -e requirements.txt ]; then \
        python3 -m pip install --no-cache-dir \
            --disable-pip-version-check \
            -r requirements.txt; \
    fi \
 && python3 -m pip install \
        --no-cache-dir --disable-pip-version-check \
        /code/ \
 && apt-get remove -y python3-pip python3-wheel \
 && apt-get autoremove -y \
 && apt-get clean -y \
 && rm -rf /var/lib/apt/lists/* \
 && useradd _gunicorn --no-create-home --user-group

USER _gunicorn
WORKDIR /code

CMD ["gunicorn", \
     "--bind", "0.0.0.0:8000", \
     "hello_world:app"]

I also got the same result.

I tried to edit the docker DNS options by :

  • Adding DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4" to the file /etc/default/docker.

  • Adding { "dns": ["192.168.1.254", "8.8.8.8"] } to the file /etc/docker/daemon.json.

I feel that I ran out of propositions :(. Does anyone have an idea of what can I do to make pip install all these packages in a python image container?

Thanks.

tammuz
  • 407
  • 5
  • 14
  • Which packages specifically is raising the "ERROR: Could not find a version that satisfies the requirement"? The pip logs usually indicate which one it failed to install and where it aborted. Check the lines before that error message. Also, how did you generate your requirements.txt file? Did it come from a virtual environment where you are running your python project? – Gino Mempin Aug 23 '21 at 23:06
  • I used: pip freeze > requirements.txt to generate the file in pycharm environement. The packages that raise the error are listed after the requirements file: apturl, catfish, ... Each time a package raised the error I eliminate it from the requirements file just to identify them one by one. – tammuz Aug 23 '21 at 23:11
  • 2
    Spot-checking a couple of those packages, they aren't on pypi.org, but they do show up on packages.ubuntu.org. Have you somehow gotten a list of Debian (OS) packages in your Python requirements file? What's the complete error for one of these packages? – David Maze Aug 23 '21 at 23:27
  • Does this packages exist for pypy3 ? Did your new Ubuntu-dicker-image had python3 installed ? Have you changed docker file entry: CMD [ "pypy3", "./index.py" ] entry to run index.py with python3? – Rustam A. Aug 23 '21 at 23:54
  • I think that you are right, they are not a pypy3 packages, they are ubuntu packages. This is the error for the first one: ERROR: Could not find a version that satisfies the requirement apturl==0.5.2 ERROR: No matching distribution found for apturl==0.5.2 WARNING: You are using pip version 20.3.4; however, version 21.2.4 is available. You should consider upgrading via the '/opt/pypy/bin/pypy3 -m pip install --upgrade pip' command. – tammuz Aug 23 '21 at 23:57
  • I think [What is “pkg-resources==0.0.0” in output of pip freeze command](https://stackoverflow.com/questions/39577984/what-is-pkg-resources-0-0-0-in-output-of-pip-freeze-command) describes this situation, though its workarounds seem specific to a `pkg-resources` non-Python package. – David Maze Aug 24 '21 at 00:13
  • I would highly suggest creating a fresh virtual environment, `pip install` only the packages that are absolutely necessary to run your code, and then `pip freeze > requirements.txt`. That list clearly contains ubuntu packages, e.g. `python-debian===0.1.36ubuntu1` – Paul P Aug 29 '21 at 18:06

1 Answers1

0

I think David Maze nailed it in the comments for some of the failures: a lot of the Python packages in the fail-list are installed via apt together with the Ubuntu packages they are part of. If you look them up (e.g. distro-info, ufw, command-not-found) on https://packages.ubuntu.com/, you'll see that many apt package ships with a Python library. (In fact, python-apt seems to be an outlier for being in PyPI as most of them are missing completely, and this answer starts with explaining why.)

The above only applies to packages that fail with the following exact message:

  ERROR: Could not find a version that satisfies the 
         requirement <package-name> (from versions: none)
  ERROR: No matching distribution found for <package-name>

Found it important to emphasize this because in my case (see at the bottom) this was the issue, and once I simply omitted the "Ubuntu-shipped packages", I just had to resolve other dependency issues, such as PyGObject (which was a tough one).

Also, there are other variants of the above error message (e.g., here and here), and this Stackoverflow thread, Could not find a version that satisfies the requirement <package>, has a lot of suggestions for those cases.

My case

Inherited a legacy Django app, and the dev did a pip freeze before leaving the project 2 years ago. I simply removed all the qualifiers (i.e., == + parts until the end of the line), and started the pip install process when I got these errors.

I have a theory that needs to be confirmed, but it was true in my case:

The pip freeze was issued on the production server, and it caught every Python package that could be pulled into the project (regardless whether it was used by the project or not).

So kind of like capturing the output of pip list vs. pip list --local.

I also had ufw on my list (among many others) that didn't make any sense as it is a firewall and this project was a simple, internal CMS. From then on, I kept hammering pip install, and crossed out errored out packages which were there because of (probably) unrelated Ubuntu installs.

(As for PyGObject and co., I was lucky that I used nix-shell to resolve dependency issues, so I only needed to issue

nix-shell -p cairo gobject-introspection

and it found the needed headers automatically. Even the database was dropped in a nix-shell.)

toraritte
  • 6,300
  • 3
  • 46
  • 67