5

I am launched an MPI program with MVAPICH2 and got this error:

Fatal error in PMPI_Gather:
Invalid buffer pointer, error stack:
PMPI_Gather(923): MPI_Gather() failed
PMPI_Gather(857): Buffers must not be aliased

There are two ways I think I could solve this:

  1. Rewrite my MPI program (use different buffers)
  2. Disable checking buffer aliasing

Do someone know how I could do this with MVAPICH2? Some compiler option, parameter, environmental variable, etc?

Something like MV2_NO_BUFFER_ALIAS_CHECK, but it does not work.

Wesley Bland
  • 8,816
  • 3
  • 44
  • 59
xijhoy
  • 167
  • 2
  • 5
  • 10
  • FYI, when you're writing your next question or answer, take a look at how to format it properly. Generally on StackOverflow, you shouldn't use HTML formatting to enforce line breaks. You can look at how I've edited your question to see how to do the formatting correctly. There's also buttons at the top of the text box that help you format more easily. – Wesley Bland Jul 23 '14 at 17:30

1 Answers1

8

What you're doing is an incorrect program and you should rewrite your code to use separate buffers

Alternatively, you might be able to use MPI_IN_PLACE if you want to use the same buffer as both the input and output values of your MPI_GATHER. Without seeing your code, I can't tell you how you could do that. You can check out some documentation about MPI_GATHER and read more about how MPI_IN_PLACE works and see if that solves your problem.

Wesley Bland
  • 8,816
  • 3
  • 44
  • 59
  • It's shocking that even in an "industrial-level" benchmarking suite such as SPEC MPI2007 there are several benchmarks that fail to run on modern systems due to operations such as `MPI_Allgather(&array[rank], 1, MPI_INT, array, 1, MPI_INT, ...);`. I guess, the "works for me" syndrome is more widespread that it might look like. – Hristo Iliev Jul 23 '14 at 22:29
  • That's a loophole that was just recently closed in MPICH. There's an environment variable you can set to not check it though. – Wesley Bland Jul 23 '14 at 22:31
  • In our case the problem runs deeper. The MPI library does not explicitly check for overlapping buffers (only tests for pointer equality) but then the `memcpy()` it uses to copy the local chunk into the local gather buffer call barfs. Rewriting the code to be compliant with the MPI standard is the only way to go. – Hristo Iliev Jul 23 '14 at 23:04
  • That's what we did until recently. We have more comprehensive checks now but it's probably gonna catch quite a few incorrect uses. – Wesley Bland Jul 23 '14 at 23:05
  • Nice to see someone from ANL around here :) – Hristo Iliev Jul 24 '14 at 07:44
  • We're lurking everywhere. :) – Wesley Bland Jul 24 '14 at 11:14
  • Thank you for reply. Unfortunately I can't correct code because it is very big file written in FORTRAN. I know about MPI_IN_PLACE. I was trying to correct calls MPI_GATHER by this way. But it is hard for me, it is not my code. And as a result of my fixes program crashed. I looking for some option on Mvapich which can help me. But seems it is wrong way. – xijhoy Jul 24 '14 at 14:22
  • There was an option added to MPICH (which is the basis for MVAPICH) called `MPICH_COLL_ALIAS_CHECK` that you can set to `0`. I doubt that's made it into MVAPICH yet though since those were just added in 3.1 I believe and I don't think that MVAPICH has pulled that down yet. – Wesley Bland Jul 24 '14 at 14:25
  • @WesleyBland Unfortunately it doesn't help. Anyway thanks a lot for reply! – xijhoy Jul 25 '14 at 10:26