Building libgit2, openssl_stream.c and "error: dereferencing pointer to incomplete type" for BIOs

Question

I am building Julia from source on a new installed OS,Ubuntu 14.04.1.

git clone https://github.com/JuliaLang/julia.git 
make -C deps getall
make -j 12

the openssl error occurs when building libgit2, one of Julia's dependencies.

The error messages are:

[ 11%] Building C object CMakeFiles/libgit2_clar.dir/src/pathspec.c.o
/home/guo/Github/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/src/openssl_stream.c: In function ‘bio_create’:
/home/guo/Github/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/src/openssl_stream.c:159:3: error: dereferencing pointer to incomplete type
  b->init = 1;
   ^
/home/guo/Github/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/src/openssl_stream.c:160:3: error: dereferencing pointer to incomplete type
  b->num = 0;
   ^
/home/guo/Github/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/src/openssl_stream.c:161:3: error: dereferencing pointer to incomplete type
  b->ptr = NULL;
   ^
/home/guo/Github/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/src/openssl_stream.c:162:3: error: dereferencing pointer to incomplete type
  b->flags = 0;
   ^
/home/guo/Github/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/src/openssl_stream.c: In function ‘bio_destroy’:
/home/guo/Github/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/src/openssl_stream.c:172:3: error: dereferencing pointer to incomplete type
  b->init = 0;

After make VERBOSE=1, the error messages are:

guo@dllab:~/Github/libgit2/build$ make  VERBOSE=1
/home/guo/data/software/cmake-3.6.1-Linux-x86_64/bin/cmake -H/home/guo/Github/libgit2 -B/home/guo/Github/libgit2/build --check-build-system CMakeFiles/Makefile.cmake 0
/home/guo/data/software/cmake-3.6.1-Linux-x86_64/bin/cmake -E cmake_progress_start /home/guo/Github/libgit2/build/CMakeFiles /home/guo/Github/libgit2/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory `/home/guo/Github/libgit2/build'
make -f CMakeFiles/libgit2_clar.dir/build.make CMakeFiles/libgit2_clar.dir/depend
make[2]: Entering directory `/home/guo/Github/libgit2/build'
cd /home/guo/Github/libgit2/build && /home/guo/data/software/cmake-3.6.1-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/guo/Github/libgit2 /home/guo/Github/libgit2 /home/guo/Github/libgit2/build /home/guo/Github/libgit2/build /home/guo/Github/libgit2/build/CMakeFiles/libgit2_clar.dir/DependInfo.cmake --color=
make[2]: Leaving directory `/home/guo/Github/libgit2/build'
make -f CMakeFiles/libgit2_clar.dir/build.make CMakeFiles/libgit2_clar.dir/build
make[2]: Entering directory `/home/guo/Github/libgit2/build'
[  0%] Building C object CMakeFiles/libgit2_clar.dir/src/openssl_stream.c.o
/usr/bin/cc  -DCLAR_FIXTURE_PATH=\"/home/guo/Github/libgit2/tests/resources/\" -DCLAR_RESOURCES=\"\" -DCLAR_TMPDIR=\"libgit2_tests\" -DGIT_ARCH_64 -DGIT_CURL -DGIT_OPENSSL -DGIT_THREADS -DGIT_USE_NSEC -DGIT_USE_STAT_MTIM -DHAVE_FUTIMENS -DHAVE_QSORT_R -DOPENSSL_SHA1 -D_FILE_OFFSET_BITS=64 -I/home/guo/Github/libgit2/src -I/home/guo/Github/libgit2/include -I/usr/local/include -I/home/guo/Github/libgit2/deps/http-parser -I/home/guo/Github/libgit2/tests  -D_GNU_SOURCE -Wall -Wextra  -fvisibility=hidden -fPIC -Wno-missing-field-initializers -Wstrict-aliasing=2 -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-function -g   -o CMakeFiles/libgit2_clar.dir/src/openssl_stream.c.o   -c /home/guo/Github/libgit2/src/openssl_stream.c
/home/guo/Github/libgit2/src/openssl_stream.c: In function ‘bio_create’:
/home/guo/Github/libgit2/src/openssl_stream.c:159:3: error: dereferencing pointer to incomplete type
  b->init = 1;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c:160:3: error: dereferencing pointer to incomplete type
  b->num = 0;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c:161:3: error: dereferencing pointer to incomplete type
  b->ptr = NULL;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c:162:3: error: dereferencing pointer to incomplete type
  b->flags = 0;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c: In function ‘bio_destroy’:
/home/guo/Github/libgit2/src/openssl_stream.c:172:3: error: dereferencing pointer to incomplete type
  b->init = 0;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c:173:3: error: dereferencing pointer to incomplete type
  b->num = 0;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c:174:3: error: dereferencing pointer to incomplete type
  b->ptr = NULL;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c:175:3: error: dereferencing pointer to incomplete type
  b->flags = 0;
   ^
/home/guo/Github/libgit2/src/openssl_stream.c: In function ‘bio_read’:
/home/guo/Github/libgit2/src/openssl_stream.c:182:35: error: dereferencing pointer to incomplete type
  git_stream *io = (git_stream *) b->ptr;
                                   ^
/home/guo/Github/libgit2/src/openssl_stream.c: In function ‘bio_write’:
/home/guo/Github/libgit2/src/openssl_stream.c:188:35: error: dereferencing pointer to incomplete type
  git_stream *io = (git_stream *) b->ptr;
                                   ^
/home/guo/Github/libgit2/src/openssl_stream.c: At top level:
/home/guo/Github/libgit2/src/openssl_stream.c:217:1: error: variable ‘git_stream_bio_method’ has initializer but incomplete type
 static BIO_METHOD git_stream_bio_method = {
 ^
/home/guo/Github/libgit2/src/openssl_stream.c:218:2: warning: excess elements in struct initializer [enabled by default]
  BIO_TYPE_SOURCE_SINK,
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:218:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:219:2: warning: excess elements in struct initializer [enabled by default]
  "git_stream",
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:219:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:220:2: warning: excess elements in struct initializer [enabled by default]
  bio_write,
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:220:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:221:2: warning: excess elements in struct initializer [enabled by default]
  bio_read,
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:221:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:222:2: warning: excess elements in struct initializer [enabled by default]
  bio_puts,
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:222:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:223:2: warning: excess elements in struct initializer [enabled by default]
  bio_gets,
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:223:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:224:2: warning: excess elements in struct initializer [enabled by default]
  bio_ctrl,
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:224:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:225:2: warning: excess elements in struct initializer [enabled by default]
  bio_create,
  ^
/home/guo/Github/libgit2/src/openssl_stream.c:225:2: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c:227:1: warning: excess elements in struct initializer [enabled by default]
 };
 ^
/home/guo/Github/libgit2/src/openssl_stream.c:227:1: warning: (near initialization for ‘git_stream_bio_method’) [enabled by default]
/home/guo/Github/libgit2/src/openssl_stream.c: In function ‘verify_server_cert’:
/home/guo/Github/libgit2/src/openssl_stream.c:342:4: warning: ‘ASN1_STRING_data’ is deprecated (declared at /usr/local/include/openssl/asn1.h:553) [-Wdeprecated-declarations]
    const char *name = (char *) ASN1_STRING_data(gn->d.ia5);
    ^
/home/guo/Github/libgit2/src/openssl_stream.c:397:4: warning: ‘ASN1_STRING_data’ is deprecated (declared at /usr/local/include/openssl/asn1.h:553) [-Wdeprecated-declarations]
    memcpy(peer_cn, ASN1_STRING_data(str), size);
    ^
/home/guo/Github/libgit2/src/openssl_stream.c: In function ‘openssl_connect’:
/home/guo/Github/libgit2/src/openssl_stream.c:450:5: error: dereferencing pointer to incomplete type
  bio->ptr = st->io;
     ^
make[2]: *** [CMakeFiles/libgit2_clar.dir/src/openssl_stream.c.o] Error 1
make[2]: Leaving directory `/home/guo/Github/libgit2/build'
make[1]: *** [CMakeFiles/libgit2_clar.dir/all] Error 2
make[1]: Leaving directory `/home/guo/Github/libgit2/build'
make: *** [all] Error 2

Run `make -V` or `make V=1` so we can see the compiler invocation. — jww, Sep 04 '16 at 13:56
@Lating - My bad; its `VERBOSE=1`; not `V=1` (I think `V=1` is for `kbuild`). Also see [Using CMake with GNU Make: How can I see the exact commands?](http://stackoverflow.com/q/2670121) (And see what a dumb idea it is to hide information by default...) — jww, Sep 05 '16 at 09:48
Building Git from sources sucks... if you think the Git command is bad dues to its penchant for taking every simple workflow and making it difficult, then brace for a Git configuration and build from sources. Git will silent fail its configuration and build stages, but march on as if nothing happened. Also see [Git 2.4.8 built from kernel.org sources and missing HTTP/HTTPS helpers](http://stackoverflow.com/q/32192181). — jww, Sep 05 '16 at 10:40
Thanks, @jww. I reinstalled the ubuntu and install dependencies via apt-get . Now everything is fine. There were other troubles caused by an unstable NVIDIA driver. — Lanting Guo, Sep 08 '16 at 10:32

HarmonicaMuse · Answer 1 · 2016-09-04T09:50:51.207

2

You need to installl libssl-dev acording to the Julia README.md:

Required build tools and external libraries

openssl — needed for HTTPS support in libgit2 on Linux, install via apt-get install libssl-dev or yum install openssl-devel.

So, in order to install it in Ubuntu, issue te following command in a terminal:

sudo apt-get install libssl-dev

After that you may attempt to recompile Julia again.

edited Sep 04 '16 at 09:50

answered Sep 04 '16 at 09:45

HarmonicaMuse

7,633
37
52

Thanks for your answer. I have already installed `libssl-dev` and also built another version `openssl` from source manually. It is probably `libssl` that causes the error. But I can't find it. My v2.10 git can't use https which should be the same reason. `https://github.com/JuliaLang/julia.git Cloning into 'julia'... fatal: Unable to find remote helper for 'https'` – Lanting Guo Sep 04 '16 at 10:08
@Lating - *"Unable to find remote helper for 'https'..."* - Even though its a Git error, it usually means curl was built *without* SSL/TLS support. Git is awful to build from sources. It silently swallows errors and continues, like forgetting to stop because HTTPS support and configuration failed. – jww Sep 05 '16 at 10:13

jww · Accepted Answer · 2016-09-05T10:18:33.063

/usr/bin/cc  -DCLAR_FIXTURE_PATH=\"/home/guo/Github/libgit2/tests/resources/\" -DCLAR_RESOURCES=\"\"
-DCLAR_TMPDIR=\"libgit2_tests\" -DGIT_ARCH_64 -DGIT_CURL -DGIT_OPENSSL -DGIT_THREADS
-DGIT_USE_NSEC -DGIT_USE_STAT_MTIM -DHAVE_FUTIMENS -DHAVE_QSORT_R -DOPENSSL_SHA1
-D_FILE_OFFSET_BITS=64 -I/home/guo/Github/libgit2/src -I/home/guo/Github/libgit2/include
-I/usr/local/include -I/home/guo/Github/libgit2/deps/http-parser -I/home/guo/Github/libgit2/tests
-D_GNU_SOURCE -Wall -Wextra -fvisibility=hidden -fPIC -Wno-missing-field-initializers
-Wstrict-aliasing=2 -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-function -g 
-o CMakeFiles/libgit2_clar.dir/src/openssl_stream.c.o   -c /home/guo/Github/libgit2/src/openssl_stream.c
/home/guo/Github/libgit2/src/openssl_stream.c: In function ‘bio_create’:
/home/guo/Github/libgit2/src/openssl_stream.c:159:3: error: dereferencing pointer to incomplete type
  b->init = 1;
...

It looks like Julia is mis-configured.

Based on your comment, "I have already installed libssl-dev and also built another version openssl from source manually....", you should see a -I option somewhere with something like -I /usr/local/ssl/include. If OpenSSL is installed at /usr/local/ssl, then the header location is /usr/local/ssl/include and the library location is /usr/local/ssl/lib.

You should probably include -Wl,-rpath,/usr/local/ssl/lib in your CFLAGS to ensure runtime linking picks up the proper OpenSSL library. Or, you may have to use LD_LIBRARY_PATH (but its usually easier to use an RPATH).

Julia does not have much to say about configuring for OpenSSL (other than its needed):

$ cat README.md | grep -B 1 -A 1 -i ssl
- **[cmake]**                   — needed to build `libgit2`.
- **[openssl]**                 — needed for HTTPS support in `libgit2` on Linux, install via `apt-get install libssl-dev` or `yum install openssl-devel`.
- **[pkg-config]**              - needed to build libgit2 correctly, especially for proxy support

--

[libunwind]:    http://www.nongnu.org/libunwind
[openssl]:      https://www.openssl.org
[libssh2]:      https://www.libssh2.org

If Julia is a well-behaved project and it honors CFLAGS, then this is all you should have to do:

$ export CFLAGS="-I/usr/local/ssl/include -L/usr/local/ssl/lib -Wl,-rpath,/usr/local/ssl/lib"
$ CFLAGS="$CFLAGS" make -C deps getall
$ CFLAGS="$CFLAGS" make VERBOSE=1

If there are any C++ components, then you will want to use:

$ CFLAGS="$CFLAGS" CXXFLAGS="$CFLAGS" make -C deps getall
$ CFLAGS="$CFLAGS" CXXFLAGS="$CFLAGS" make VERBOSE=1

For the most part, it looks like it honors your CFLAGS:

$ CFLAGS="$CFLAGS" make VERBOSE=1
make[1]: Entering directory '/Users/jwalton/julia/deps'
...
echo make -C build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6/ CC="clang  -mmacosx-version-min=10.6 -m64" FC="gfortran -m64" RANLIB="ranlib" FFLAGS=" -O2 -fPIC" TARGET= BINARY=64 USE_THREAD=1 GEMM_MULTITHREADING_THRESHOLD=50 NUM_THREADS=16 NO_AFFINITY=1 DYNAMIC_ARCH=1 INTERFACE64=1 SYMBOLSUFFIX="64_" LIBPREFIX="libopenblas64_" OBJCONV=/Users/jwalton/julia/deps/build/objconv/objconv MAKE_NB_JOBS=0 # echo first, so we only print the error message below in a failure case
make -C build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6/ CC=clang  -mmacosx-version-min=10.6 -m64 FC=gfortran -m64 RANLIB=ranlib FFLAGS= -O2 -fPIC TARGET= BINARY=64 USE_THREAD=1 GEMM_MULTITHREADING_THRESHOLD=50 NUM_THREADS=16 NO_AFFINITY=1 DYNAMIC_ARCH=1 INTERFACE64=1 SYMBOLSUFFIX=64_ LIBPREFIX=libopenblas64_ OBJCONV=/Users/jwalton/julia/deps/build/objconv/objconv MAKE_NB_JOBS=0
make[2]: Entering directory '/Users/jwalton/julia/deps/build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6'
make[3]: Entering directory '/Users/jwalton/julia/deps/build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6/interface'
clang  -mmacosx-version-min=10.6 -m64 -I/usr/local/ssl/include -L/usr/local/ssl/lib -Wl,-rpath,/usr/local/ssl/lib -O2 -DMAX_STACK_ALLOC=2048 -DEXPRECISION -Wall -m64 -DF_INTERFACE_GFORT  -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=16 -DASMNAME=_ -DASMFNAME=__ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I. -O2 -DMAX_STACK_ALLOC=2048 -DEXPRECISION -Wall -m64 -DF_INTERFACE_GFORT  -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=16 -DASMNAME=_saxpy -DASMFNAME=_saxpy_ -DNAME=saxpy_ -DCNAME=saxpy -DCHAR_NAME=\"saxpy_\" -DCHAR_CNAME=\"saxpy\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX -c axpy.c -o saxpy.o
clang: warning: -Wl,-rpath,/usr/local/ssl/lib: 'linker' input unused
clang: warning: argument unused during compilation: '-L/usr/local/ssl/lib'
In file included from <built-in>:320:
<command line>:22:9: warning: 'ASMNAME' macro redefined [-Wmacro-redefined]
#define ASMNAME _saxpy
        ^

I don't know about the other Julia issues, like 'ASMNAME' macro redefined. I'm guessing (and its purely a guess), its because I am testing it on OS X, but there's no real support for it (RANLIB="ranlib" is a giveaway something is wrong because OS X uses Apple's libtool, not ranlib).

Building libgit2, openssl_stream.c and "error: dereferencing pointer to incomplete type" for BIOs

2 Answers2