How does one install pygraphviz on a HPC cluster without errors even when graphviz seems to be installed?

Question

I was trying to pip and conda install pygraphviz. I believe grahviz is installed since

(metalearningpy1.7.1c10.2) miranda9~/ML4Coq $ conda install -y graphviz
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

but when I try installing pygraphviz with pip I get the following:

(metalearningpy1.7.1c10.2) miranda9~/ML4Coq $ pip install pygraphviz
Collecting pygraphviz
  Using cached pygraphviz-1.7.zip (118 kB)
Building wheels for collected packages: pygraphviz
  Building wheel for pygraphviz (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-b61bb2uf
       cwd: /tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/
  Complete output (55 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/pygraphviz
  copying pygraphviz/scraper.py -> build/lib.linux-x86_64-3.8/pygraphviz
  copying pygraphviz/testing.py -> build/lib.linux-x86_64-3.8/pygraphviz
  copying pygraphviz/graphviz.py -> build/lib.linux-x86_64-3.8/pygraphviz
  copying pygraphviz/agraph.py -> build/lib.linux-x86_64-3.8/pygraphviz
  copying pygraphviz/__init__.py -> build/lib.linux-x86_64-3.8/pygraphviz
  creating build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_attribute_defaults.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_subgraph.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_readwrite.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_unicode.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_drawing.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_string.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_close.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_scraper.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_edge_attributes.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_graph.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_html.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_clear.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/__init__.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_node_attributes.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_layout.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
  running egg_info
  writing pygraphviz.egg-info/PKG-INFO
  writing dependency_links to pygraphviz.egg-info/dependency_links.txt
  writing top-level names to pygraphviz.egg-info/top_level.txt
  reading manifest file 'pygraphviz.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.png' under directory 'doc'
  warning: no files found matching '*.txt' under directory 'doc'
  warning: no files found matching '*.css' under directory 'doc'
  warning: no previously-included files matching '*~' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '.svn' found anywhere in distribution
  no previously-included directories found matching 'doc/build'
  writing manifest file 'pygraphviz.egg-info/SOURCES.txt'
  copying pygraphviz/graphviz.i -> build/lib.linux-x86_64-3.8/pygraphviz
  copying pygraphviz/graphviz_wrap.c -> build/lib.linux-x86_64-3.8/pygraphviz
  running build_ext
  building 'pygraphviz._graphviz' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/pygraphviz
  gcc -pthread -B /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/include/python3.8 -c pygraphviz/graphviz_wrap.c -o build/temp.linux-x86_64-3.8/pygraphviz/graphviz_wrap.o
  In file included from /usr/include/graphviz/gvc.h:17:0,
                   from pygraphviz/graphviz_wrap.c:2712:
  /usr/include/graphviz/types.h:49:20: fatal error: cgraph.h: No such file or directory
   #include <cgraph.h>
                      ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pygraphviz
  Running setup.py clean for pygraphviz
Failed to build pygraphviz
Installing collected packages: pygraphviz
    Running setup.py install for pygraphviz ... error
    ERROR: Command errored out with exit status 1:
     command: /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-mu112lx1/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/include/python3.8/pygraphviz
         cwd: /tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/
    Complete output (55 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/pygraphviz
    copying pygraphviz/scraper.py -> build/lib.linux-x86_64-3.8/pygraphviz
    copying pygraphviz/testing.py -> build/lib.linux-x86_64-3.8/pygraphviz
    copying pygraphviz/graphviz.py -> build/lib.linux-x86_64-3.8/pygraphviz
    copying pygraphviz/agraph.py -> build/lib.linux-x86_64-3.8/pygraphviz
    copying pygraphviz/__init__.py -> build/lib.linux-x86_64-3.8/pygraphviz
    creating build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_attribute_defaults.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_subgraph.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_readwrite.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_unicode.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_drawing.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_string.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_close.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_scraper.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_edge_attributes.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_graph.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_html.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_clear.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/__init__.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_node_attributes.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    copying pygraphviz/tests/test_layout.py -> build/lib.linux-x86_64-3.8/pygraphviz/tests
    running egg_info
    writing pygraphviz.egg-info/PKG-INFO
    writing dependency_links to pygraphviz.egg-info/dependency_links.txt
    writing top-level names to pygraphviz.egg-info/top_level.txt
    reading manifest file 'pygraphviz.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.png' under directory 'doc'
    warning: no files found matching '*.txt' under directory 'doc'
    warning: no files found matching '*.css' under directory 'doc'
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '.svn' found anywhere in distribution
    no previously-included directories found matching 'doc/build'
    writing manifest file 'pygraphviz.egg-info/SOURCES.txt'
    copying pygraphviz/graphviz.i -> build/lib.linux-x86_64-3.8/pygraphviz
    copying pygraphviz/graphviz_wrap.c -> build/lib.linux-x86_64-3.8/pygraphviz
    running build_ext
    building 'pygraphviz._graphviz' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/pygraphviz
    gcc -pthread -B /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/include/python3.8 -c pygraphviz/graphviz_wrap.c -o build/temp.linux-x86_64-3.8/pygraphviz/graphviz_wrap.o
    In file included from /usr/include/graphviz/gvc.h:17:0,
                     from pygraphviz/graphviz_wrap.c:2712:
    /usr/include/graphviz/types.h:49:20: fatal error: cgraph.h: No such file or directory
     #include <cgraph.h>
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rc_2rzek/pygraphviz_904720f7fa8b49f4a7006870a0d5e479/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-mu112lx1/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/include/python3.8/pygraphviz Check the logs for full command output.

with conda:

(metalearningpy1.7.1c10.2) miranda9~/ML4Coq $ conda install -y -c anaconda pygraphviz

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: / 
Found conflicts! Looking for incompatible packages.                                                                                                                                                                                                                                                                                                                                                                           failed                                                                                                                                                                                                                                                                                                                                                                                                                            

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - pygraphviz -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0']

Your python: python=3.8

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

what are the sources of my errors? [note I am on a linux machine, not mac os, my mac phygraphviz works fine]

I saw the following related questions:

but they didn't help because I cannot run apt-get

(metalearningpy1.7.1c10.2) miranda9~/ML4Coq $ apt-get install python-dev graphviz libgraphviz-dev pkg-config
-bash: apt-get: command not found

Edit:

note I also did try to install grapviz with pip but that didn't solve my problems:

(metalearningpy1.7.1c10.2) miranda9~/ML4Coq $ pip install graphviz
Collecting graphviz
  Downloading graphviz-0.16-py2.py3-none-any.whl (19 kB)
Installing collected packages: graphviz
Successfully installed graphviz-0.16

Edit 2:

I found out that graphviz is there but why can't it find it?

(metalearningpy1.7.1c10.2) miranda9~ $ rpm -qa |grep graphviz
graphviz-gd-2.30.1-21.el7.x86_64
graphviz-2.30.1-21.el7.x86_64
graphviz-devel-2.30.1-21.el7.x86_64

Edit 3:

I'd also like this to work with pip install -e . e.g. see installation of ultimate utils (https://github.com/brando90/ultimate-utils):

Due to a depedency on pygraphviz you will likely need to install graphviz first. Next, install graphviz. On mac, brew install graphviz. On Ubuntu, sudo apt install graphviz. After graphviz is install, install uutils in edibable mode and all it's depedencies with pip:

pip install -e ~/ultimate-utils/ultimate-utils-proj-src

as a gitissue: https://github.com/brando90/ultimate-utils/issues/6

seems the answer will be to download the original graphviz[https://graphviz.org/download/] (not the pytorch package with the same name...confusing!) by the sys admins on all the nodes in the HPC. — Charlie Parker, May 13 '21 at 16:39

score 1 · Answer 1 · answered May 13 '21 at 21:40

Well, it seems that using conda-forge worked!

(metalearningpy1.7.1c10.2) miranda9~ $ conda install -y -c conda-forge pygraphviz

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2

  added / updated specs:
    - pygraphviz


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libgcc-ng-9.3.0            |      h2828fa1_19         7.8 MB  conda-forge
    libgomp-9.3.0              |      h2828fa1_19         376 KB  conda-forge
    pygraphviz-1.7             |   py38h0d738da_0         125 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         8.3 MB

The following NEW packages will be INSTALLED:

  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  libgomp            conda-forge/linux-64::libgomp-9.3.0-h2828fa1_19
  pygraphviz         conda-forge/linux-64::pygraphviz-1.7-py38h0d738da_0

The following packages will be UPDATED:

  certifi            pkgs/main::certifi-2020.12.5-py38h06a~ --> conda-forge::certifi-2020.12.5-py38h578d9bd_1
  libgcc-ng           pkgs/main::libgcc-ng-9.1.0-hdf63c60_0 --> conda-forge::libgcc-ng-9.3.0-h2828fa1_19

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge
  ca-certificates    pkgs/main::ca-certificates-2021.4.13-~ --> conda-forge::ca-certificates-2020.12.5-ha878542_0
  openssl              pkgs/main::openssl-1.1.1k-h27cfd23_0 --> conda-forge::openssl-1.1.1k-h7f98852_0



Downloading and Extracting Packages
libgcc-ng-9.3.0      | 7.8 MB    | ##################################################################################################################################################################################################################################################################################################################################################################################### | 100% 
pygraphviz-1.7       | 125 KB    | ##################################################################################################################################################################################################################################################################################################################################################################################### | 100% 
libgomp-9.3.0        | 376 KB    | ##################################################################################################################################################################################################################################################################################################################################################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Note:

However, notice that pygraphviz doesn't recommend using conda to install their pkg for some reason

We recommend avoiding Anaconda and conda-forge to install Graphviz and PyGraphviz.

however since pip keeps failing for me - despite graphviz being installed in the cluster - see the output of rpm:

[rundblom@vision-12 ~]$ rpm -qa |grep graphviz
graphviz-devel-2.30.1-21.el7.x86_64
graphviz-2.30.1-21.el7.x86_64
graphviz-gd-2.30.1-21.el7.x86_64

so idk what is wrong with using pip. I would have preferred to load a module in the usual HPC way but that isn't an option for me right now...

does someone know why using `conda-forge` worked? – Charlie Parker Sep 15 '21 at 18:17 — Charlie Parker, Sep 15 '21 at 18:17

score 1 · Answer 2 · answered Jun 23 '21 at 16:01

I was able to compile it on CentOS with graphviz-devel installed. Originally I got the same error as above, the missing cgraph.h, but by providing a header lookup path to the compiler I could make it build:

CPPFLAGS="-I/usr/include/graphviz" pip install pygraphviz

When I tried to use the package though I had symbol lookup errors. So in the end I gave up and went with conda.

How does one install pygraphviz on a HPC cluster without errors even when graphviz seems to be installed?

2 Answers2