-1

I am trying to build a Docker application that uses Python's gensim library, version 3.8.3, which is being installed via pip from a requirements.txt file.

However, Docker seems to have trouble while trying to do RUN pip install -r requirements.txt

My Requirement.txt for reference -

boto==2.49.0
boto3==1.14.33
botocore==1.17.33
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
Cython==0.29.14
docutils==0.15.2
Flask==1.1.2
gensim==3.8.3
idna==2.10
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
MarkupSafe==1.1.1
numpy==1.19.1
python-dateutil==2.8.1
requests==2.24.0
s3transfer==0.3.3
scipy==1.5.2
six==1.15.0
smart-open==2.1.0
urllib3==1.25.10
Werkzeug==1.0.1

dockerFile

FROM python:3.8.2-alpine
WORKDIR /project
ADD . /project
RUN set -x && apk add --no-cache build-base && apk add --no-cache libexecinfo-dev
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
CMD ["python","similarity.py"]

error:

(venv) C:\Users\verma\PycharmProjects\flaskTest>docker image build -t similarity-flask-api  .
Sending build context to Docker daemon  302.7MB
Step 1/7 : FROM python:3.8.2-alpine
 ---> 6c32e2504283
Step 2/7 : WORKDIR /project
 ---> Using cache
 ---> 554b6bda89ad
Step 3/7 : ADD . /project
 ---> d085a645ecb1
Step 4/7 : RUN set -x && apk add --no-cache build-base && apk add --no-cache libexecinfo-dev
 ---> Running in e7117c1e18ff
+ apk add --no-cache build-base
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/18) Installing libgcc (9.2.0-r4)
(2/18) Installing libstdc++ (9.2.0-r4)
(3/18) Installing binutils (2.33.1-r0)
(4/18) Installing libmagic (5.37-r1)
(5/18) Installing file (5.37-r1)
(6/18) Installing gmp (6.1.2-r1)
(7/18) Installing isl (0.18-r0)
(8/18) Installing libgomp (9.2.0-r4)
(9/18) Installing libatomic (9.2.0-r4)
(10/18) Installing mpfr4 (4.0.2-r1)
(11/18) Installing mpc1 (1.1.0-r1)
(12/18) Installing gcc (9.2.0-r4)
(13/18) Installing musl-dev (1.1.24-r2)
(14/18) Installing libc-dev (0.7.2-r0)
(15/18) Installing g++ (9.2.0-r4)
(16/18) Installing make (4.2.1-r2)
(17/18) Installing fortify-headers (1.1-r0)
(18/18) Installing build-base (0.5-r1)
Executing busybox-1.31.1-r9.trigger
OK: 182 MiB in 52 packages
+ apk add --no-cache libexecinfo-dev
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/2) Installing libexecinfo (1.1-r1)
(2/2) Installing libexecinfo-dev (1.1-r1)
OK: 183 MiB in 54 packages
Removing intermediate container e7117c1e18ff
 ---> 9e7a97f8bddc
Step 5/7 : RUN pip install --upgrade pip
 ---> Running in 0286591e9e70
Collecting pip
  Downloading pip-20.2.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1
    Uninstalling pip-20.1:
      Successfully uninstalled pip-20.1
Successfully installed pip-20.2.1
Removing intermediate container 0286591e9e70
 ---> ca837786d695
Step 6/7 : RUN pip install -r requirements.txt
 ---> Running in 7f124c100c0b
Collecting boto==2.49.0
  Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB)
Collecting boto3==1.14.33
  Downloading boto3-1.14.33-py2.py3-none-any.whl (129 kB)
Collecting botocore==1.17.33
  Downloading botocore-1.17.33-py2.py3-none-any.whl (6.5 MB)
Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet==3.0.4
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting click==7.1.2
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting Cython==0.29.14
  Downloading Cython-0.29.14.tar.gz (2.1 MB)
Collecting docutils==0.15.2
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting Flask==1.1.2
  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting gensim==3.8.3
  Downloading gensim-3.8.3.tar.gz (23.4 MB)
Collecting idna==2.10
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting itsdangerous==1.1.0
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting Jinja2==2.11.2
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting jmespath==0.10.0
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting MarkupSafe==1.1.1
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Processing /root/.cache/pip/wheels/df/b2/64/111c431ca7f7d49afb42126b7351fe1a4894803d75026360de/numpy-1.19.1-cp38-cp38-linux_x86_64.whl
Collecting python-dateutil==2.8.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting requests==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting s3transfer==0.3.3
  Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting scipy==1.5.2
  Downloading scipy-1.5.2.tar.gz (25.4 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpoyjzx5wb
         cwd: /tmp/pip-install-r078skp_/scipy
    Complete output (139 lines):
    lapack_opt_info:
    lapack_mkl_info:
    customize UnixCCompiler
      libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    openblas_lapack_info:
    customize UnixCCompiler
    customize UnixCCompiler
      libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    openblas_clapack_info:
    customize UnixCCompiler
    customize UnixCCompiler
      libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    flame_info:
    customize UnixCCompiler
      libraries flame not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    atlas_3_10_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
      NOT AVAILABLE

    atlas_3_10_info:
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_3_10_info'>
      NOT AVAILABLE

    atlas_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_threads_info'>
      NOT AVAILABLE

    atlas_info:
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/lib/
    <class 'numpy.distutils.system_info.atlas_info'>
      NOT AVAILABLE

    accelerate_info:
      NOT AVAILABLE

    lapack_info:
    customize UnixCCompiler
      libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    lapack_src_info:
      NOT AVAILABLE

      NOT AVAILABLE

    setup.py:460: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-ujofw06w'), proceeding with generating Cython sources
and expanding templates
      warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
    Running from SciPy source directory.
    /tmp/pip-build-env-mw61mr08/overlay/lib/python3.8/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
        Lapack (http://www.netlib.org/lapack/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [lapack]) or by setting
        the LAPACK environment variable.
      if getattr(self, '_calc_info_{}'.format(lapack))():
    /tmp/pip-build-env-mw61mr08/overlay/lib/python3.8/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
        Lapack (http://www.netlib.org/lapack/) sources not found.
        Directories to search for the sources can be specified in the
        numpy/distutils/site.cfg file (section [lapack_src]) or by setting
        the LAPACK_SRC environment variable.
      if getattr(self, '_calc_info_{}'.format(lapack))():
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/tmp/pip-build-env-mw61mr08/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/tmp/pip-build-env-mw61mr08/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 248, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/tmp/pip-build-env-mw61mr08/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 142, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 583, in <module>
        setup_package()
      File "setup.py", line 579, in setup_package
        setup(**metadata)
      File "/tmp/pip-build-env-mw61mr08/overlay/lib/python3.8/site-packages/numpy/distutils/core.py", line 137, in setup
        config = configuration()
      File "setup.py", line 477, in configuration
        raise NotFoundError(msg)
    numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_bu
ild_wheel /tmp/tmpoyjzx5wb Check the logs for full command output.
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1

I tried this thread - Docker unable to install numpy, scipy, or gensim As suggested I added line 4 and 5 in my dockerFile but it is still not working.

davidism
  • 121,510
  • 29
  • 395
  • 339
Kush Verma
  • 35
  • 1
  • 7

2 Answers2

1

in the post you mension, thye install libc-dev to compile packs ...
you dont.

RUN apt-get -y install libc-dev
RUN apt-get -y install build-essential

I have problems trying to use "alpine" with Python... so we choose "slim-buster" as docker image for Python.

so if you can...

1 - I would try slim-buster if you can
2 - Try a numpy ready docker image and install your python packages.

Carlos Leite
  • 172
  • 5
  • /bin/sh: apt-get: not found The command '/bin/sh -c apt-get -y install libc-dev' returned a non-zero code: 127 I tried doing that but getting this – Kush Verma Aug 11 '20 at 13:22
  • 2
    The image in question is an `alpine` image which doesn't use `apt` as the package manager – C.Nivs Aug 11 '20 at 16:00
  • as I told you... Alpine is really "dry" image. Can you consider the option I told you so... slim-buster https://hub.docker.com/_/python – Carlos Leite Aug 11 '20 at 21:38
  • 1
    @CarlosLeite Just used slim-buster instead alpine and that made my image size reduced by ~600 mbs, 849 mb from 1.4 GB. You mentioned it very rightly that alpine is a "dry" image and while using that i have to use some extra layers in docker file which results in larger image. slim-buster looks like a better choice for my case. Thank you :) – Kush Verma Aug 18 '20 at 15:17
  • glad it helps. Glad to help. – Carlos Leite Aug 20 '20 at 11:58
0

To install numpy on an alpine image, you typically need a few more dependencies:

RUN apk update && apk add gfortran build-base openblas-dev libffi-dev

Namely the openblas-dev, which you are missing. That will at least get numpy to install

C.Nivs
  • 12,353
  • 2
  • 19
  • 44
  • Although this works but now my image size is 1.39 GB, can you suggest what should I do minimize the size and building time. – Kush Verma Aug 11 '20 at 13:17
  • My advice would be to use an image with these dependencies installed. Unfortunately, a lot of the dependencies are being compiled from source, hence the build time, so that's a no-starter. For size, I'd recommend looking into [the builder pattern](https://medium.com/@kaperys/create-lean-docker-images-using-the-builder-pattern-37fe2b5d97d4) – C.Nivs Aug 11 '20 at 15:59