file writing in mpi using python without race condition

Question

I am using mpi4py to model a distributed application and I want all the processes to write to a common file. Is there any function which allows this without the race condition ?

We need more details about your problem. What do you wish to write in your file ? Could you use `MPI_File_write()` [doc mpich](http://www.mpich.org/static/docs/v3.1/www3/MPI_File_write.html) or [example](http://mpi.deino.net/mpi_functions/MPI_File_write.html) ? Show us what you have tried ! — francis, Aug 19 '14 at 17:42
possible duplicate of [mpi processes working in a burst](http://stackoverflow.com/questions/25378022/mpi-processes-working-in-a-burst) — Wesley Bland, Aug 19 '14 at 21:34

score 0 · Answer 1 · answered Aug 19 '14 at 17:44

0

You should check out one of the many tutorials out there for using MPI I/O. I'm sure there's some way to use it in mpi4py.

answered Aug 19 '14 at 17:44

Wesley Bland

8,816
3
44
59

there seems to be I/O tutorials for MPI but not for mpi4py. – Pranav Raj Aug 20 '14 at 17:10
Of course. But the semantics are the same. It's just the bindings that change between MPI and MPI4PY right? – Wesley Bland Aug 20 '14 at 17:20

score 0 · Answer 2 · answered Sep 19 '14 at 10:49

0

It might be worth looking at h5py's parallel implementation

http://docs.h5py.org/en/latest/mpi.html

Which works very well with mpi4py as long as you build it correctly

answered Sep 19 '14 at 10:49

Mark

93
1
9

score 0 · Answer 3 · answered Feb 11 '15 at 15:50

I had a similar problem. For me, the easiest way to work around this was to have each process write out to its own file, and include a time stamp. This file can then be processed afterwards to put everything in order.

For example, include (python3-style) prints like:

print("Process %d just received point %r at %s" % (rank, point,str(datetime.datetime.now())))

Just include datetime at the top. mpi4py seems to buffer some of it's I/O in an interesting fashion, so each process maintaining its own output is the most robust solution.

file writing in mpi using python without race condition

3 Answers3