How to solve UPC Runtime error: out of shared memory

Question

I am trying to run a Berkeley UPC code on a computer with 64 cores and 256 GB RAM. However the code fails to run because it cannot find enough memory. The following should work because 51 x 5 = 255 GB < 256 GB

upcrun -n 51 -shared-heap=5GB xcorupc_sac inputpgas_sac{$rc1}.txt
..
UPCR: UPC thread  3 of 51 on range (pshm node 0 of 1, process  3 of 51, pid=191914)
UPCR: UPC thread 16 of 51 on range (pshm node 0 of 1, process 16 of 51, pid=191927)
UPC Runtime warning: Requested shared memory (5120 MB) > available (2515 MB) on node 0 (range): using 2515 MB per thread instead

UPC Runtime error: out of shared memory
  Local shared memory in use:  1594 MB per-thread,  81340 MB total
  Global shared memory in use:    0 MB per-thread,     1 MB total
  Total shared memory limit:   2515 MB per-thread,  128281 MB total
upc_alloc unable to service request from thread 0 for 1672245248 more bytes

NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.

I don't understand why the Total shared memory limit is 128 GB which is half of the total physical memory present. I cannot over-ride it even with a the shared-heap flag where I am clearly asking for 5 GB per thread. Any suggestions ?

cat /proc/meminfo 
MemTotal:       263378836 kB

The UPC build was compiled using flags --with-sptr-packed-bits=20,9,35 that allows up to 2^35 = 32 GB of shared memory per thread.

EDIT1: Following is the output of the command upcc --version

[avinash@range jointinvsurf5_cajoint_compile]$ upcc --version
This is upcc (the Berkeley Unified Parallel C compiler), v. 2019.4.4
  (getting remote translator settings...)
----------------------+---------------------------------------------------------
 UPC Runtime          | v. 2019.4.4, built on Feb 11 2020 at 23:31:40
----------------------+---------------------------------------------------------
 UPC-to-C translator  | v. 2.28.0, built on Jul 19 2018 at 20:29:47
                      | host aphid linux-x86_64/64
                      | gcc v4.2.4 (Ubuntu 4.2.4-1ubuntu4)
----------------------+---------------------------------------------------------
 Translator location  | http://upc-translator.lbl.gov/upcc-2019.4.0.cgi
----------------------+---------------------------------------------------------
 networks supported   | smp udp mpi ibv
----------------------+---------------------------------------------------------
 default network      | ibv
----------------------+---------------------------------------------------------
 pthreads support     | available (if used, default is 2 pthreads per process)
----------------------+---------------------------------------------------------
 Configured with      | '--with-translator=http://upc-translator.lbl.gov/upcc-2
                      | 019.4.0.cgi' '--with-sptr-packed-bits=20,9,35'
                      | '--prefix=/usr/local/berkeley_upc/opt'
                      | '--with-multiconf-magic=opt'
----------------------+---------------------------------------------------------
 Configure features   | trans_bupc,pragma_upc_code,driver_upcc,runtime_upcr,
                      | gasnet,upc_collective,upc_io,upc_memcpy_async,
                      | upc_memcpy_vis,upc_ptradd,upc_thread_distance,upc_tick,
                      | upc_sem,upc_dump_shared,upc_trace_printf,
                      | upc_trace_mask,upc_local_to_shared,upc_all_free,
                      | upc_atomics,pupc,upc_types,upc_castable,upc_nb,nodebug,
                      | notrace,nostats,nodebugmalloc,nogasp,nothrille,
                      | segment_fast,os_linux,cpu_x86_64,cpu_64,cc_gnu,
                      | packedsptr,upc_io_64
----------------------+---------------------------------------------------------
 Configure id         | range Tue Feb 11 23:18:39 PST 2020 gnome-initial-setup
----------------------+---------------------------------------------------------
 Binary interface     | 64-bit x86_64-unknown-linux-gnu
----------------------+---------------------------------------------------------
 Runtime interface #  | Runtime supports 3.0 -> 3.13: Translator uses 3.6
----------------------+---------------------------------------------------------
                      |  --- BACKEND SETTINGS (for ibv network) ---
----------------------+---------------------------------------------------------
 C compiler           | /usr/bin/gcc
                      |   GNU/4.8.5/4.8.5 20150623 (Red Hat 4.8.5-39)
                      |   gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) Copyright
                      |   (C) 2015 Free Software Foundation, Inc.
----------------------+---------------------------------------------------------
 C compiler flags     | -O3 --param max-inline-insns-single=35000 --param
                      | inline-unit-growth=10000 --param
                      | large-function-growth=200000 -Wno-unused
                      | -Wunused-result -Wno-unused-parameter -Wno-address
                      | -std=gnu99
----------------------+---------------------------------------------------------
 linker               | /data/seismo82/avinash/Programs/openmpiinstall/bin/mpic
                      | c
                      |   GNU/4.8.5/4.8.5 20150623 (Red Hat 4.8.5-39)
                      |   gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) Copyright
                      |   (C) 2015 Free Software Foundation, Inc.
----------------------+---------------------------------------------------------
 linker flags         | -D_GNU_SOURCE=1 -O3 --param
                      | max-inline-insns-single=35000 --param
                      | inline-unit-growth=10000 --param
                      | large-function-growth=200000 -Wno-unused
                      | -Wunused-result -Wno-unused-parameter -Wno-address
                      | -std=gnu99 -L/data/seismo82/avinash/Programs/myupc/opt
                      | -L/data/seismo82/avinash/Programs/myupc/opt/umalloc
                      | -lupcr-ibv-seq -lumalloc
                      | -L/data/seismo82/avinash/Programs/myupc/opt/gasnet/ibv-
                      | conduit -lgasnet-ibv-seq -libverbs -lpthread -lrt
                      | -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -lgcc -lm
----------------------+---------------------------------------------------------

EDIT2: Following is the output of df -h /dev/shm command

[avinash@range jointinvsurf5_cajoint_compile]$ df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           126G   21M  126G   1% /dev/shm

Please provide the full output of `upcc --version`. Assuming this is Linux, please also provide the output of `df -h /dev/shm` (if that directory doesn't exist, then /var/shm or /run/shm) — Dan Bonachea, Mar 26 '20 at 18:44
@DanBonachea please see the output of the `upcc --version` and `df -h /dev/shm` commands. Following your answer it seems that the kernel-provided POSIX shared memory space is indeed limited to `126 GB`. Any suggestion on how to increase this limit ? I will try to limit it to 85% as suggested. The OS is `Red Hat Enterprise Linux Server 7.7 3.10.0-1062.12.1.el7.x86_64` — Guddu, Mar 28 '20 at 03:00
ok i read that section on GASNET readme webpage. it seems that it requires administrator privileges and skills to actually make the changes. i have neither. thanks for your help to understand this. — Guddu, Mar 28 '20 at 03:08
Yes absolutely you need admin to reconfigure the kernel's /dev/shm allocation. Given you lack that your best recourse is probably to either convince an admin to make the change for you, or limit your usage to the 126GB available. — Dan Bonachea, Mar 28 '20 at 21:57

Dan Bonachea · Accepted Answer · 2020-03-28T21:55:03.993

By default, Berkeley UPC uses kernel shared memory services to cross-map the UPC shared segments between co-located processes. For smp-conduit, this is the only mode of operation.

Assuming this is a Linux system with configure defaults, the most likely explanation is exhaustion of the kernel-provided POSIX shared memory space. You can confirm this by looking at the virtual file system where that resides. Here's an example from a system configured for up to 20G of shared memory:

$df -h /dev/shm /var/shm /run/shm
df: '/var/shm': No such file or directory
df: '/run/shm': No such file or directory
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            20G  504K   20G   1% /dev/shm

This value limits the total per-node shared memory segment space. This limit can usually be raised by an administrator adjusting kernel settings, although the details vary with distribution.

For more info, see the section 'System Settings for POSIX Shared Memory' in https://gasnet.lbl.gov/dist-ex/README

Finally, note that even once the above issue is addressed, asking for 255 GB of shared memory heap on a system with 256 GB of physical DRAM (99.6%) may be inadvisable. This leaves very little space for the non-shared portions of application memory (stack, static data, malloc heap) and for memory overheads of the kernel and daemon processes. Depending on your kernel settings this may trigger an out-of-memory panic to start killing processes. We generally recommend a safe rule-of-thumb limit of 85% of physical memory (assuming the system is otherwise idle), and "proceed with caution" beyond that.

How to solve UPC Runtime error: out of shared memory

1 Answers1