hnswlib package issue while installing chromadb in ubuntu

Question

I am using ubuntu 20.04 focal and trying to install chromadb by using 'pip install chromdb' but I am getting following error

  Building wheel for hnswlib (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for hnswlib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      running bdist_wheel
      running build
      running build_ext
      creating tmp
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/namanc/codes/langchian_chromadb/venv/include -I/usr/include/python3.10 -c /tmp/tmp822lkln8.cpp -o tmp/tmp822lkln8.o -std=c++14
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/namanc/codes/langchian_chromadb/venv/include -I/usr/include/python3.10 -c /tmp/tmpw33wg22s.cpp -o tmp/tmpw33wg22s.o -fvisibility=hidden
      building 'hnswlib' extension
      creating build
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/python_bindings
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/pip-build-env-izlfj5h4/overlay/lib/python3.10/site-packages/pybind11/include -I/tmp/pip-build-env-izlfj5h4/overlay/lib/python3.10/site-packages/numpy/core/include -I./hnswlib/ -I/home/namanc/codes/langchian_chromadb/venv/include -I/usr/include/python3.10 -c ./python_bindings/bindings.cpp -o build/temp.linux-x86_64-cpython-310/./python_bindings/bindings.o -O3 -fopenmp -DVERSION_INFO=\"0.7.0\" -std=c++14 -fvisibility=hidden
      In file included from /tmp/pip-build-env-izlfj5h4/overlay/lib/python3.10/site-packages/pybind11/include/pybind11/detail/../attr.h:13,
                       from /tmp/pip-build-env-izlfj5h4/overlay/lib/python3.10/site-packages/pybind11/include/pybind11/detail/class.h:12,
                       from /tmp/pip-build-env-izlfj5h4/overlay/lib/python3.10/site-packages/pybind11/include/pybind11/pybind11.h:13,
                       from /tmp/pip-build-env-izlfj5h4/overlay/lib/python3.10/site-packages/pybind11/include/pybind11/functional.h:12,
                       from ./python_bindings/bindings.cpp:2:
      /tmp/pip-build-env-izlfj5h4/overlay/lib/python3.10/site-packages/pybind11/include/pybind11/detail/../detail/common.h:266:10: fatal error: Python.h: No such file or directory
        266 | #include <Python.h>
            |          ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hnswlib
Failed to build hnswlib
ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I have tried to update the gcc complier too but error still persists

I tried to update the ubuntu and its gcc complier

score 2 · Answer 1 · answered May 30 '23 at 12:46

2

The error message suggests that the installation of chromdb is failing due to a problem with building the hnswlib wheel. The error specifically mentions that it cannot find the Python.h header file.

To resolve this issue, you need to install the development headers for Python. You can do this by running the following command (since you are on ubuntu):

sudo apt-get install python3-dev

This command will install the necessary development files for Python. Afterward, you can try installing chromdb again using pip install chromdb.

If you still encounter the same error, you can try updating your pip version to the latest version by running:

pip install --upgrade pip

Then attempt the installation again using pip install chromdb.

If the issue persists, please provide any additional error messages or logs you receive during the installation process, as they might provide further insight into the problem. I hope this helps :)

answered May 30 '23 at 12:46

Eldinur the Kolibri

460
1
13

I tried to install python3-dev and after I also tried to upgrade pip but the same error is persists – Sinister _ May 30 '23 at 17:47
1

@Sinister_ Hmm I don't know. Maybe try `sudo apt-get install build-essential` as well and pip install `numpy` and `sudo apt-get install libomp-dev`. If nothing helps (then I can't help you any further) and there might be a compatibility issue with chromdb and your ubuntu version (but I doubt that). – Eldinur the Kolibri May 30 '23 at 17:51
1

I tried everything but nothing worked then I tried to locate Python.h by using 'locate Python.h' and I found it in 'include' folder inside my python package but there were no Python.h file in my virtual env include folder so I copied all the files from include folder to the virtual env include folder and then it worked – Sinister _ May 30 '23 at 18:21
Can you please explain your solution a bit more like how did you copy the file and where did you copy it from @Sinister_ ? – Shahriar Dhruvo Jul 16 '23 at 17:33
On RHEL, install like that: sudo yum install python3-devel. It solved my issue – Olivier Aug 16 '23 at 15:10

score 1 · Answer 2 · edited Jul 10 '23 at 10:35

1

As suggested by @Eldinur the Kolibri in their answer, installing development headers for Python does solve this. Although You may have to run python version-specific installation.

For python3.8, it will be like:

sudo apt-get install python3.8-dev

edited Jul 10 '23 at 10:35

moken

3,227
8
13
23

answered Jul 05 '23 at 17:51

MasterRohit

11
2

hot_pan_stand_back · Answer 3 · 2023-08-24T18:27:51.367

I was having almost the exact same error message when trying pip install chromadb (the only differences were I'm using python3.11, looks like you're on 3.10. Second, it looks like you're using Langchain and I'm running from a python script).

What ended up working for me was sudo apt install python3.11-dev, you should try installing python3.10. I had also tried sudo apt install python3-dev and this didn't work (which I think means I don't have 3.11 set as default, not sure).

I'd recommend trying this and if that doesn't work, try running a script outside of langchain (chromadb's homepage has a quick example). If it works standalone but not in langchain then you know it's langchain that's now the issue.

score 0 · Answer 4 · answered Aug 28 '23 at 17:29

0

This worked for me:

sudo apt-get install python3-dev
python3.10 -m pip install --upgrade pip
sudo apt install build-essential
python3.10 -m pip install hnswlib

answered Aug 28 '23 at 17:29

John Ostrowick

75
3

score -1 · Answer 5 · answered Aug 24 '23 at 07:46

The error message indicates that the installation of 'chromdb' is encountering a failure attributed to the construction of the 'hnswlib' wheel. The error explicitly states its inability to locate the 'Python.h' header file.

To rectify this matter, the resolution involves procuring the development headers tailored for Python. This can be executed by executing the ensuing command (presuming you're utilizing Ubuntu):

sudo apt-get install python3-dev

This command's execution will facilitate the acquisition of the requisite development files pertinent to Python. Once accomplished, make another attempt at installing 'chromdb' using the subsequent pip command:

pip install chromdb

Should the identical error persist, consider elevating your 'pip' version to the most current iteration through the command:

pip install --upgrade pip

Subsequently, endeavor once more to carry out the installation of 'chromdb' utilizing 'pip'.

I hope this helps!

hnswlib package issue while installing chromadb in ubuntu

5 Answers5