gfortran linking c libraries with conda

Question

I am trying to compile a C++/Fortran program using conda on an Ubuntu 18.04 server where I do not have superuser rights.

I am able to compile correctly the program with the same source code on my Ubuntu 18.04 PC (using conda too), but on the server I get a bunch of errors. At the moment, I am stuck with a "library not found - undefined reference" error:

gfortran -o glm -Wl,--export-dynamic obj/glm_globals.o obj/glm_util.o obj/glm_csv.o obj/glm_mobl.o obj/glm_mixu.o obj/glm_wqual.o obj/glm_layers.o obj/glm_surface.o obj/glm_input.o obj/glm_plot.o obj/glm_output.o obj/glm_ncdf.o obj/glm_lnum.o obj/glm_init.o obj/glm_flow.o obj/glm_mixer.o obj/glm_deep.o obj/glm_stress.o obj/glm_bird.o obj/glm_model.o obj/glm_types.o obj/glm_const.o obj/glm_debug.o obj/glm_main.o obj/glm_zones.o -L/local/XXX/my_name/PycharmProjects/glm/glm_source/GLM/../libutil/lib -lutil -L/local/XXX/my_name/anaconda3/lib -lnetcdf -L/usr/lib -L/local/XXX/my_name/PycharmProjects/glm/glm_source/GLM/../libplot/lib -lplot -lgd -lpng -ljpeg -lm -lX11    -lgfortran

/local/XXX/my_name/anaconda3/bin/ld: warning: libpthread.so.0, needed by /local/XXX/my_name/anaconda3/lib/libnetcdf.so, not found (try using -rpath or -rpath-link)
/local/XXX/my_name/anaconda3/bin/ld: warning: libxcb.so.1, needed by /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libX11.so, not found (try using -rpath or -rpath-link)
/local/XXX/my_name/anaconda3/bin/ld: warning: libdl.so.2, needed by /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libX11.so, not found (try using -rpath or -rpath-link)
/local/XXX/my_name/anaconda3/bin/ld: warning: librt.so.1, needed by /local/XXX/my_name/anaconda3/lib/./libhdf5_hl.so.100, not found (try using -rpath or -rpath-link)
/local/XXX/my_name/anaconda3/bin/ld: warning: libresolv.so.2, needed by /local/XXX/my_name/anaconda3/lib/././libgssapi_krb5.so.2, not found (try using -rpath or -rpath-link)
/local/XXX/my_name/anaconda3/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libX11.so: undefined reference to `dlopen@GLIBC_2.2.5'
/local/XXX/my_name/anaconda3/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libX11.so: undefined reference to `xcb_wait_for_event'
/local/XXX/my_name/anaconda3/bin/ld: /local/XXX/my_name/anaconda3/lib/././libkrb5.so.3: undefined reference to `__res_nsearch@GLIBC_2.2.5'
...

collect2: error: ld returned 1 exit status
Makefile:277: recipe for target 'glm' failed
make: *** [glm] Error 1

I can see the shared libraries libpthread.so.0, librt.so.1, libdl.so.2, libresolv.so.2 not found by the linker are part of glibc. So I assume there is some kind of problem linking to that library.

In a previous attempt I tried installing glibc with conda. This resulted in conda crashing with a "segmentation fault (core dumped)" when I tried to compile. I had to reinstall anaconda to make conda work again.

In the last days I also have tried to add -rpath as suggested in the warnings, to add an LD_LIBRARY_PATH, to add the directory containing the shared libraries with -L. Nothing worked.

I feel quite lost at the moment. Any idea of what could have gone wrong?

PS: The difference between compiling on my computer and on the server is that I installed missing libraries in the system on my computer, but on conda on the server. Thus, I had to add the location of those libraries with -L when compiling on the server.

Answer 1

This answer is a suggested alternative workflow that aims at avoiding the problem rather than an exact diagnosis of the issue in OP.

In my experience, I have found the Conda Forge compiler packages help simplify the creation and use of custom environments for compilation. As an example, here is the YAML definition for the environment I use to build the kallisto software. For your case, I would expect something like

fortran-compiler.yaml

name: fortran-compiler  # name it what you want
channels:
  - conda-forge
  - defaults
dependencies:
  - fortran-compiler
  - cxx-compiler
  - libpng
  - libgd
  - jpeg
  - libnetcdf
  - openlibm
  - xorg-libx11
# include the other required libraries

Create the environment with

conda env create -f fortran-compiler.yaml

then run your compilation with this environment activated. Activating the environment should automatically manage where the linker will look so that the libraries in the environment will be found (this is, in part, what the Conda Forge *-compiler packages provide). The idea is to have as many libraries as possible come from the environment itself.

I find this approach keeps the amount of manual locating and including of library paths to a minimum. However, it does require having to track down the libraries in Conda Forge (e.g., search for libx11), which unfortunately is not always a one-to-one mapping from library name to package name. The advantage is that a compilation environment defined in this way can facilitate transferring across platforms - e.g., the example YAML I have for kallisto works on both osx-64 and linux-64 without any changes - because it explicitly defines that Conda will provide the shared libraries.

Perhaps there are simpler ways to do this, but this is at least what I've found works after having multiple bad experiences trying to use the gcc or clang Anaconda packages directly.