7

Intro: I've developed an application in C++ that utilizes LAPACK(LAPACKE) and MPI, all on Windows. Works OK in Windows (compiling and linking are taken care of through Code::Blocks IDE), but the execution is way too slow. Hence, I want to migrate the code to our small "supercomputer" running under CentOS Linux where we have already installed GNU C++, MPICH2 and LAPACK.

Questions: How to COMPILE/LINK and RUN a C++ code that calls LAPACKE on Linux/CentOS? Do I have to have GNU Fortran installed on the CentOS machine in order to compile/link/run C++ with LAPACK(LAPACKE)?

THANKS A LOT!!!

Boki
  • 193
  • 1
  • 3
  • 15

1 Answers1

9

I assume that my Debian is close enough to your CentOS for these tricks to work...

1) Check that LAPACKE is installed on your computed.

  • You can search locations like /usr/include, /usr/local/include for files lapacke.h, lapacke_config.h, lapacke_mangling.h, lapacke_mangling_with_flags.h and lapacke_utils.h.
  • You can search locations like /usr/lib or /usr/local/lib for the static library lapacke.a or the dynamic library lapacke.so or lapacke.so.3.

If these files are missing, consider installing the packages liblapacke and liblapacke-dev. Alternatively, (in particular if you don't have root privileges), you can download source of netlib's lapack and lapacke at http://www.netlib.org/lapack/#_lapack_version_3_6_1 To compile LAPACKE, rename make.inc.example to make.inc, then type:

    make
    make lapackelib

The include files will be located in lapack-3.6.1/LAPACKE/include and the library will be in lapack-3.6.1. gcc and gfortran are useful to recompile lapack and lapacke from scratch.

2) Let's compile a simple code based on this example:

#include <iostream>
#include <string>
#include <fstream>

#include <stdlib.h>
#include <stdio.h>
#include <math.h>

#include <lapacke.h>
#include "mpi.h"

void print_matrix_rowmajor(const char* desc, lapack_int m, lapack_int n, double* a, lapack_int lda );

int main(int argc, char *argv[])
{
    MPI_Init(&argc,&argv);
    std::cout << "Start..." << std::endl;
    //std::string fn_VALS;

    /* Locals */
    double A[5][3] = {1,1,1,2,3,4,3,5,2,4,2,5,5,4,3};
    double b[5][2] = {-10,-3,12,14,14,12,16,16,18,16};
    lapack_int info,m,n,lda,ldb,nrhs;

    /* Initialization */
    m = 5;
    n = 3;
    nrhs = 2;
    lda = 3;
    ldb = 2;

    /* Print Entry Matrix */
    print_matrix_rowmajor( "Entry Matrix A", m, n, *A, lda );
    /* Print Right Rand Side */
    print_matrix_rowmajor( "Right Hand Side b", n, nrhs, *b, ldb );
    printf( "\n" );
    /* Executable statements */
    printf( "LAPACKE_dgels (row-major, high-level) Example Program Results\n" );
    /* Solve least squares problem*/
    info = LAPACKE_dgels(LAPACK_ROW_MAJOR,'N',m,n,nrhs,*A,lda,*b,ldb);

    /* Print Solution */
    print_matrix_rowmajor( "Solution", n, nrhs, *b, ldb );
    printf( "\n" );


    std::cout << "info = " << info << std::endl;




    std::cout << "Done :-) !!!" <<std::endl;
    MPI_Finalize();
    return 0;
}


////////////////////////////////////////////////////////* Auxiliary routine: printing a matrix */
void print_matrix_rowmajor(const char* desc, lapack_int m, lapack_int n, double* a, lapack_int lda )
{
    lapack_int i, j;
    printf( "\n %s\n", desc );
    for( i = 0; i < m; i++ )
    {
        for( j = 0; j < n; j++ )
        {
            printf( " %6.2f", a[i*lda+j]);
        }
        printf( "\n" );
    }
}
//=======================================

The command to compile is:

mpiCC main.cpp -o main -llapacke -llapack -lblas -lm -Wall

If the include files are in a particular folder, use -I/usr/pathtolapackedoth. Similarly, if the library is in a particula folder, try -L/usr/lib/pathtoliblapackedota. Depending on how MPICH2 was installed, it is likely that mpiCC wraps g++. You can type mpiCC --version to learn more. To run it using 2 processes:

mpirun -np 2 main

Finally, you do not have to install GNU Fortran installed on the CentOS machine in order to compile/link/run C++ with LAPACK(LAPACKE). Indeed, it is only required if you wish to recompile LAPACK from scratch.

francis
  • 9,525
  • 2
  • 25
  • 41
  • Thanks a lot. Let me work on it, and I'll post what I did. – Boki Dec 12 '16 at 17:46
  • First, I've searched for the installation of LAPACK(E): – Boki Dec 12 '16 at 18:17
  • Typing **locate lapack** gave me following (just part of the list) – Boki Dec 12 '16 at 18:19
  • - /usr/lib64/liblapack.a - /usr/lib64/liblapack.so 3. /usr/lib64/liblapack.so.3 4. /usr/lib64/liblapack.so.3.2 5. /usr/lib64/liblapack.so.3.2.1 6. /usr/lib64/liblapack_pic.a – Boki Dec 12 '16 at 18:22
  • So, it looks like that LAPACK is installed indeed. Also I've used **yum info lapack** and got the package installation confirmation. – Boki Dec 12 '16 at 18:23
  • 1
    Indeed, LAPACK is installed, but not LAPACKE. LAPACK is the linear algebra library and LAPACKE is a small wrapper written so as to easily call LAPACK functions from C/C++. It's something different. You may try to `yum install lapack-devel` to get the headers of LAPACK. See https://www.centos.org/forums/viewtopic.php?t=20984 for instance. I don't know whether LAPACKE is in the developper's package: you may have to recompile LAPACKE from scratch... – francis Dec 12 '16 at 19:04
  • In folder **/usr** there is no any file/folder with word **lapacke** in it. If I compile your test program it gives me bunch of errors, all of them pointing to some LAPACKE_ command not being recognized. So, I'll follow your advice and will post my results. – Boki Dec 12 '16 at 20:25
  • Btw, I've tested separately MPI with C++ on this CentOS machine and they work just fine. LAPACKE is the problem..... will post later what I did. – Boki Dec 12 '16 at 20:33
  • I've downloaded Lapack and did **make** and **make lapackelib**. The second went well, but the first one gave two errors: – Boki Dec 12 '16 at 21:12
  • make[1]: Entering directory `........./lapack-3.6.1/BLAS/TESTING' gfortran -O2 -frecursive -c sblat1.f -o sblat1.o gfortran sblat1.o \ ../../librefblas.a -o ../xblat1s gfortran: ../../librefblas.a: No such file or directory make[1]: *** [../xblat1s] Error 1 make[1]: Leaving directory `........./lapack-3.6.1/BLAS/TESTING' make: *** [blas_testing] Error 2 – Boki Dec 12 '16 at 21:14
  • I've tried to compiled the program again with one change in the code, for the "lapacke.h" header file I've included the new path where the LAPACKE headers are: "....../lapack-3.6.1/LAPACKE/include/lapacke.h". Now, I only get error for linking with with lapacke library (can not find -llapacke). – Boki Dec 12 '16 at 21:28
  • Now, if I add the path for linking to the libraries when compiling: **mpiCC mainX.cpp -o mainX -L/home/......./lapack-3.6.1 -llapacke -llapack -lblas -lm -Wall** then I get following errors: /usr/bin/ld: skipping incompatible /usr/local/lib/libm.so when searching for -lm /usr/bin/ld: skipping incompatible /usr/local/lib/libpthread.so when searching for -lpthread /usr/bin/ld: skipping incompatible /usr/local/lib/libc.so when searching for -lc – Boki Dec 12 '16 at 21:49
  • Hmmmm, it's seems like **it's working!** I still get some error (or warring) messages like: **3 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics** and **Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages** but the output of the test program is OK - i.e. it properly uses LAPACK(E). – Boki Dec 12 '16 at 21:58
  • **THANKS SO MUCH FRANCIS!** – Boki Dec 12 '16 at 21:58
  • Although the test example works, my actual program doesn't pass the compilation stage on the Linux server (it work on Windows with MPI and LAPACKE). The compiler complains about **previous declarations of complex numbers**: error: previous declaration of 'float __complex__ lapack_make_complex_float(float, float)' with 'C++' linkage error: conflicts with new declaration with 'C' linkage ---> Here it complains about declaration: lapack_complex_float lapack_make_complex_float( float re, float im ); error: expected unqualified-id before '__complex__' – Boki Dec 12 '16 at 23:47
  • 1
    You're welcome! Could you try to include the header of lapacke as if it were a C header, using `extern "C"{#include "lapacke.h"}` ? See http://stackoverflow.com/questions/3329159/including-c-headers-inside-a-c-program – francis Dec 13 '16 at 09:54
  • Excellent, that solved half of the errors, type of which is: conflicting/repeated definition of complex number. I used your advice on all LapackE headers: **extern "C" { #include #include #include #include #include }** – Boki Dec 13 '16 at 17:21
  • The other type of error relates to my header with auxiliary math functions: **error: expected unqualified-id before '__complex__'** . I tried to include it under the **extern "C" {...}** but it didn't help. – Boki Dec 13 '16 at 17:25
  • I'll keep on working on this error.... if you have advice, I would greatly appreciate. Merci beaucoup! – Boki Dec 13 '16 at 17:28
  • This type of error repeats three times in my header file with auxiliary math functions whenever I declare complex number/matrix: **const std::complex i1(0.0,1.0);** and with two more similar cases. – Boki Dec 13 '16 at 17:42
  • 1
    Have you read this hint? http://eigen.tuxfamily.org/bz/show_bug.cgi?id=452 Could you try to `#include ` instead of `#include ` (if you use such a thing...) ? See also http://stackoverflow.com/questions/23414270/c-complex-and-complex-h-in-the-same-file : it might help to understand the problem. – francis Dec 13 '16 at 18:58
  • I've seen the first hint you sent, but not the second. If I strip down the code to empy "main" and header files without LAPACKE headers, then the problem is solved by replacing **#include ** with **#include **. However, when just adding the LAPACKE headers with **extern "C" {...}** brings back the problem. Even worse, the type of errors depend weather the **extern...** is before or after the **#include **. – Boki Dec 13 '16 at 23:05
  • lapacke.h has **#include **. This might be inconflict with **#include ** – Boki Dec 14 '16 at 00:58
  • I've casted this as another question on: http://stackoverflow.com/questions/41133269/order-of-include-matters-bug – Boki Dec 14 '16 at 01:23
  • Hey Francis, it seems that the problem is with different versions of GCC compilers on the Win and the Linux machines. They spit different preprocessing content. I can see that **complex** is replace by **_Complex** on the places where Linux GCC has reported error messages. So, I have question for you: what is the version of GCC you are using and what does your compiler spit out in preprocessor file (option **-E** when compiling) when encounters **complex** and **lapack_complex_double** in the original code? **THANKS SO MUCH!!!!** – Boki Dec 15 '16 at 22:32
  • 1
    Hi, my gcc version is 4.9. According to the answer and comments in http://stackoverflow.com/questions/23414270/c-complex-and-complex-h-in-the-same-file , using a version of gcc higher than 4.7 could help. Try to add `-std=c++11` to the compiler's line. – francis Dec 18 '16 at 17:43
  • Thanks Francis! I've tried that and it doesn't recognize std=c++11. They have installed GCC 4.9.3, so there are two GCC compilers on the same machine. Now, the question is how to set the default GCC compiler to be the new one 4.9.3. – Boki Dec 19 '16 at 21:01
  • @ Francis: In MPICH 3.2 Installation Guide it says to select the Intel compilers instead of the GNU compilers on a system with both, use one should use `./configure CC=icc CXX=icpc F77=ifort FC=ifort ... ` . Now, is there a way to discriminate between different version of GCC compilers in `./configure` ? – Boki Dec 20 '16 at 00:29