0

I'm trying compile a large model using ifort and with the -qopenmp flag: 

FC = mpif90 and FCFLAGS = -g -m64 -qopenmp -O3 -xHost -fp-model \
precise -convert big_endian -traceback -r8
FCDEFS = BLAS LITTLE LINUX INTEGER_IS_INT
LFLAGS = -qopenmp

CC = mpicc
CFLAGS = -g -O3
CCDEFS = BLAS LITTLE LINUX INTEGER_IS_INT _ABI64

OMP_NUM_THREADS=2
OMP_STACKSIZE=1000M
OMP_SCHEDULE=STATIC

ulimit -s unlimited

mpprun -n 192 master.exe -e "exp1" -f d1 -t 2700

However, when I try and run the model I get:

mpprun info: Starting impi run on 12 node ( 192 rank X 1 th ) for 22

==================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 19619 RUNNING AT n457
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================
APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault 
(signal 11)
mpprun info:   Job terminated with error

Now the thing is, if I compile this model without the OpenMP flag and run it with TotalView, there are no errors and the model executes without error. I'm trying to find a way to track down what is going wrong. Does anyone have any ideas? Where do I start? how can I do simple tests to see why OpenMP exited with a segmentation fault?

Appreciate the help

Vladimir F Героям слава
  • 57,977
  • 4
  • 76
  • 119
Shejo284
  • 4,541
  • 6
  • 32
  • 44
  • 2
    There are many ways OpenMP can trigger a segfault. If the program does not even start, one can think of http://stackoverflow.com/questions/3070587/segfault-with-fopenmp-for-a-trivial-program http://stackoverflow.com/questions/13870564/gfortran-openmp-segmentation-fault-occurs-on-basic-do-loop http://stackoverflow.com/questions/13264274/why-segmentation-fault-is-happening-in-this-openmp-code They should be probably linked as duplicates – Vladimir F Героям слава Nov 19 '16 at 14:44
  • Can linked libraries cause the program to crash at the start? How can I get error messages apart from "segmentation fault"? – Shejo284 Nov 19 '16 at 16:20
  • 3
    Such an excessive setting for OMP_STACKSIZE is likely to exhaust even "unlimited" stack. Working applications using as much as 40M (thread stack) are rare. Note that mpif90 would not be the correct wrapper for Intel MPI (where it gives you gfortran) nor for OpenMPI (unless you aliased it to the correct underlying name). So you need to be sure that you haven't crossed incompatible MPI implementations. – tim18 Nov 19 '16 at 19:05
  • 1
    Have you considered generating a core file and exploring it, also? – Harald Nov 21 '16 at 12:18
  • @Harald That sounds like a good idea. How can I generate one? I've also reduced the stack size to 128M but the problem persists. – Shejo284 Nov 21 '16 at 16:51
  • @Shejo284, take a look at https://software.intel.com/pt-br/forums/intel-fortran-compiler-for-linux-and-mac-os-x/topic/269362 – Harald Nov 21 '16 at 22:14

0 Answers0