mpi processes working in a burst

Question

I am using mpi4py to model a distributed application.

I have n processes accessing a shared file and writing some logs into the shared file during their execution. I notice that the logs are not uniformly written. Here is an example of how logs are written into the shared file:

process0.log0
process0.log1
process0.log2
process0.log3
process0.log4
process2.log0
process2.log1
process2.log2
process1.log0
process1.log1

Ideally it should be like:

process0.log0
process1.log0
process2.log0
process0.log1
process2.log1
process1.log1
process0.log2

Can anyone tell me what is possibly wrong with my implementation? I am writing into the file using Pickle module.

following is the function which dumps the log:

import pickle

log_file_name = "store.log"

def writeLog(data):
  try:
    with open(log_file_name,"a") as fp:
        pickle.dump(obj=data,file=fp)
  except:
    with open(log_file_name,"w") as fp:
        pickle.dump(obj=data,file=fp)

def readLog():
 data = []
 try:
    with open(log_file_name,"r") as fp:
        while True:
            data.append(pickle.load(fp))
    return data
 except EOFError:
    return data

All n processes access this function to dump the data

please share your implementation code.so that we can find how to fix it — sundar nataraj, Aug 19 '14 at 07:43
Is your MPI distribution strategy a `worker pool` or a `scatter-gather`? — Mike McKerns, Aug 19 '14 at 18:27

score 3 · Answer 1 · edited May 23 '17 at 12:03

There are lots of questions/answers out there that explain the phenomenon you're seeing here:

Even though these are (mostly) talking about printing to the screen, the problem is the same. MPI is a distributed model which means that some processes will execute faster than others and it will probably be a different order every time depending on the workload/ordering of each process.

If ordering is important, you can use synchronization functions to enforce it or you can use something more fancy like MPI I/O for writing to files (not my specialty so I can't tell you much more about it).

mpi processes working in a burst

1 Answers1

Linked