0

Here is a function I made with the intent to run a function func with arguments inputs across all processors, one by one. This is mostly geared towards println by order, but not limited to.

Edit: I forgot to include the next/prev processor call, so I embedded it in. The ranks (ID's) for current, next and previous processors have been thoroughly verified.

function MPI_OneByOne(::Val{my_ID},::Val{Total_Proc},func::Function,inputs...) where {my_ID,Total_Proc}

    if !Base.isequal(my_ID,zero(my_ID))
        # Wait for previous processor to finish
        MPI.recv(comm; source=mod( MPI.Comm_rank(comm)-1, MPI.Comm_size(comm)))
    end

    println(my_ID," ======================================")
    func(inputs...)

    if !Base.isequal(my_ID,Total_Proc-one(Total_Proc))
        # All but last processor need to push the next one to act
        MPI.send(nothing, comm; dest= mod(MPI.Comm_rank(comm)+1, MPI.Comm_size(comm)))
    end

    MPI.Barrier(comm)

end

The idea was to kickstart with the first proc, while the others wait with recv. The first finishes, trigerring a send to #2. #2 does it's thing, trigerring #3 with a send, and so on.

As you can expect, this didn't work as expected and the output of the function (especially when using some longer input than just a single value) is not controlled or by order. What am I doing wrong?

TheWhitestOfFangs
  • 403
  • 3
  • 14
  • 2
    If I understand you correctly (I'm a Julia novice, but have a lot of MPI experience) you can't do it this way, assuming you are talking about ordering the I/O in process order. Basically while you can sync the computational processes this way, you can't order the OS processes that deal with the I/O, and more often than not you will observe random ordering. The only sensible ways to ensure ordering of the I/O is either one processes does it all, or MPI-IO (or, debatably, something like HDF5, but there the structure is a bit different) – Ian Bush Jul 29 '23 at 07:29
  • Thank you. While it was not my intent to split screen prints, file access and generic operations apart, based on what you are saying I might have to do just that. So for file access (read/write) I can use the MPI-IO, but what about running non-file related functionalities? I assume the simplest example would be println, but I also had more complex functionality in mind. What would you suggest to do about (1) screen printing? (2) other types of operations not related to file I/O ? – TheWhitestOfFangs Jul 29 '23 at 15:38
  • Do you need it to be printed in real time from each rank? If not, you can just collect them all (i.e. via gather) after they have all finished and then print them ordered by rank. I'm not sure how to get around the system I/O giving a somewhat random order. – kirklong Aug 02 '23 at 16:50

1 Answers1

1

If you want to perform an action one rank at a time, you can do something like

function sequentially(f, comm)
  s, r = MPI.Comm_size(comm), MPI.Comm_rank(comm)
  for i in 0:s-1
    i == r && f()
    MPI.Barrier(comm)
  end
end

which can then be used as

sequentially(MPI.COMM_WORLD) do
  # do something
end

However, this may not give the expected results if you’re performing IO such as println, as MPI does not specify how the output from different ranks is combined. Printing, perhaps followed by flush(stdout), may work for your specific MPI implementation and can be a good option for debugging. If you want the sequential output to work reliably across implementations, you would have to send the data to a single process or rely on MPI-IO.

msch
  • 178
  • 2
  • 8