1

I am trying to modify a boost array in a function. The code runs fine in serial and produces the expected results, whereas with MPI I get an error. Here is a minimal reproducible example:

#include <stdio.h>
#include <iostream>
#include "boost/multi_array.hpp"


static void myfunc(boost::multi_array<double, 3>& foo){
            std::cout<<foo[0][0][0]<<std::endl;
        }

int main(){
    typedef boost::multi_array<double, 3> array_type;
    typedef array_type::index index;
    array_type foo(boost::extents[1][1][1]);
    myfunc(foo);
  return 0;
}

The error I get is:

.../src/boost/boost/multi_array/base.hpp:135: Reference boost::detail::multi_array\
::value_accessor_n<T, NumDims>::access(boost::type<Reference>, boost::detail::multi_array::value_accessor_n<T, NumDims>::index, TPtr, const size_type\
*, const index*, const index*) const [with Reference = boost::detail::multi_array::sub_array<double, 2>; TPtr = double*; T = double; long unsigned in\
t NumDims = 3; boost::detail::multi_array::value_accessor_n<T, NumDims>::index = long int; boost::detail::multi_array::multi_array_base::size_type = \
long unsigned int]: Assertion `idx - index_bases[0] >= 0' failed.

The root rank prints out foo[0][0][0], but the other ranks do not, which is when the error occurs. My naive interpretation of this behavior would be that the other ranks do not carry foo[0][0][0], therefore I get something like an "index out of bound error". Any ideas on how I can fix this?

Edit: When I print out the size of the boost array each rank prints a size of (initial size of array)/(number of ranks).

Suyama87
  • 85
  • 10
  • As you're mentioning MPI, what makes you sure, that [`boost::multi_array`](https://www.boost.org/doc/libs/1_63_0/libs/multi_array/doc/user.html#sec_introduction) should be handled in a thread safe manner (AFAIK that's also not the case for standard containers)? I can't spot any guarantee for that in their documentation. But I could be totally wrong about that, I never worked with MPI myself. – πάντα ῥεῖ Feb 05 '21 at 16:21
  • Naively I would guess they are thread-safe, extrapolating from the second answer here on boost vectors: https://stackoverflow.com/questions/9042571/is-stdvector-or-boostvector-thread-safe. But I also may be wrong. In any case, I would still need to modify a boost array with MPI, so if passing it to another object would lead thread-safely in this direction, I'm all ears. – Suyama87 Feb 05 '21 at 16:28
  • Might be solvable with a thin template wrapper, and c++ standard read / write locks to guard the access operations, in case it isn't a thread safe container. – πάντα ῥεῖ Feb 05 '21 at 16:31
  • Could you please provide your solution in a minimal reproducible example? Thank you :) – Suyama87 Feb 05 '21 at 16:33
  • Isn't MPI is multi-processing? I think it sends data using serialization. (So no threads, locking shouldn't matter) I'm just not seeing any MPI related code in the question – sehe Feb 05 '21 at 17:32
  • 1
    Works for me with Open MPI 4.0.5 on Arch Linux: `$ mpic++ -o foo foo.cc && mpiexec -n 4 ./foo` produces `0` four times. `foo.cc` is copy-paste of your code. There is nothing related to MPI in your program so all ranks execute the exactly same instructions. – Hristo Iliev Feb 05 '21 at 21:40
  • @HristoIliev Thank you, please see edit, I seem to have missed a crucial detail. – Suyama87 Feb 07 '21 at 08:02

1 Answers1

0

Indeed, the problem is that not all ranks carry all indices. In particular, if the total number of ranks is N_ranks, each individual rank is rank and the dimensions of the boost array are NxNxN, each rank carries [rank * N / N_ranks:(rank + 1) * N / N_ranks, 0:N, 0:N]. Therefore, if one wants to access the foo array, it should be indexed in a manner dependent on the number of ranks.

Suyama87
  • 85
  • 10
  • 1
    I don't see how this could be the case when `boost::multi_array` is not a distributed array type. My suspicion is there is more code that you aren't showing us. As I said, the example in your question works fine with four ranks on my computer, which shouldn't be the case if what you write here is true. Also, can't find anything in Boost's documentation about distributed arrays. – Hristo Iliev Feb 07 '21 at 15:29
  • There might well be some piece of code I don't realize adds to the initial question and therefore have omitted. Do you have any suspicion what condition could lead to such a behavior? – Suyama87 Feb 08 '21 at 10:51