4

In my parallel program, there was a big matrix. Each process computed and stored a part of it. Then the program wrote the matrix to a file by letting each process wrote its own part of the matrix in the correct order. The output file is in "unformatted" form. But when I tried to read the file in a serial code (I have the correct size of the big matrix allocated), I got an error which I don't understand.

My question is: in an MPI program, how do you get a binary file as the serial version output for a big matrix which is stored by different processes?

Here is my attempt:

    if(ThisProcs == RootProcs) then
        open(unit = file_restart%unit, file = file_restart%file, form = 'unformatted')
        write(file_restart%unit)psi
        close(file_restart%unit)
    endif
#ifdef USEMPI
    call mpi_barrier(mpi_comm_world,MPIerr)
#endif
    do i = 1, NProcs - 1
        if(ThisProcs == i) then
            open(unit = file_restart%unit, file = file_restart%file, form = 'unformatted', status = 'old', position = 'append')
            write(file_restart%unit)psi
            close(file_restart%unit)
        endif
#ifdef USEMPI
        call mpi_barrier(mpi_comm_world,MPIerr)
#endif
    enddo

Psi is the big matrix, it is allocated as:

Psi(N_lattice_points, NPsiStart:NPsiEnd)

But when I tried to load the file in a serial code:

open(2,file=File1,form="unformatted")
read(2)psi


forrtl: severe (67): input statement requires too much data, unit 2 (I am using MSVS 2012+intel fortran 2013)

How can I fix the parallel part to make the binary file readable for the serial code? Of course one can combine them into one big matrix in the MPI program, but is there an easier way?

Edit 1

The two answers are really nice. I'll use access = "stream" to solve my problem. And I just figured I can use inquire to check whether the file is "sequential" or "stream".

francescalus
  • 30,576
  • 16
  • 61
  • 96
FortCpp
  • 906
  • 2
  • 12
  • 28
  • The error suggests the file is larger than `psi`. What is the size of the file and what is the size of `psi`? – Carey Gregory Jun 11 '15 at 20:30
  • @CareyGregory In my test case the file is 8.03 MB. In the same test case Psi is split into 2 parts for two processes. I've checked the file and it seems if you do it in the serial code you get the same sized file. – FortCpp Jun 11 '15 at 20:32
  • When each process writes out it does to a record for its part. You then try to read the whole matrix from a single record. You may want to read about record-directed and stream output. – francescalus Jun 11 '15 at 20:32
  • @CareyGregory complex(8) Psi(4096, 16, 8). I forgot to mention that the file is 8.03MB which include Psi and another array d(4096). – FortCpp Jun 11 '15 at 20:45
  • @francescalus So you mean I should write a loop to read the file to the correspond part of the matrix in the same order of the processes write the file, correct? – FortCpp Jun 11 '15 at 20:47
  • I'm shockingly unable to find a duplicate-target. I'll look further, but I may just write an answer. – francescalus Jun 11 '15 at 20:48
  • http://stackoverflow.com/q/23552597 is as close as I can get. It isn't a duplicate-target, but it's still useful. – francescalus Jun 11 '15 at 20:52
  • consider you might be better off gathering up the data and having your root process write it all at once. – agentp Jun 12 '15 at 14:32
  • That's right. But "stream" is more flexible. – FortCpp Jun 12 '15 at 16:19

3 Answers3

4

This isn't a problem specific to MPI, but would also happen in a serial program which took the same approach of writing out chunks piecemeal.

Ignore the opening and closing for each process and look at the overall connection and transfer statements. Your connection is an unformatted file using sequential access. It's unformatted because you explicitly asked for that, and sequential because you didn't ask for anything else.

Sequential file access is based on records. Each of your write statements transfers out a record consisting of a chunk of the matrix. Conversely, your input statement attempts to read from a single record.

Your problem is that while you try to read the entire matrix from the first record of the file that record doesn't contain the whole matrix. It doesn't contain anything like the correct amount of data. End result: "input statement requires too much data".

So, you need to either read in the data based on the same record structure, or move away from record files.

The latter is simple, use stream access

open(unit = file_restart%unit, file = file_restart%file,  &
     form = 'unformatted', access='stream')

Alternatively, read with a similar loop structure:

do i=1, NPROCS
  ! read statement with a slice
end do

This of course requires understanding the correct slicing.

Alternatively, one can consider using MPI-IO for output, which is very similar to using stream output. Read this back in with stream access. You can find about this concept elsewhere on SO.

francescalus
  • 30,576
  • 16
  • 61
  • 96
3

Fortran unformatted sequential writes in record files are not quite completely raw data. Each write will have data before and after the record in a processor dependent form. The size of your reads cannot exceed the record size of your writes. This means if psi is written in two writes, you will need to read it back in two reads, you cannot read it in at once.

Perhaps the most straightforward option is to instead use stream access instead of sequential. A stream file is indexed by bytes (generally) and does not contain record start and end information. Using this access method you can split the write but read all at once. Stream access is a feature of Fortran 2003.

If you stick with sequential access, you'll need to know how many MPI ranks wrote the file and loop over properly sized records to read the data as it was written. You could make the user specify the number of ranks or store that as the first record in the file and read that first to determine how to read the rest of the data.

casey
  • 6,855
  • 1
  • 24
  • 37
1

If you are writing MPI, why not MPI-IO? Each process will call MPI_File_set_view to set a subarray view of the file, then each process can collectively write the data with MPI_FILE_WRITE_ALL . This approach is likely to scale really well on big machines (though your approach will be fine up to oh, maybe 100 processors.)

Rob Latham
  • 5,085
  • 3
  • 27
  • 44
  • I am using windows version MPICH2. But the version on the website is 1.4.1p. As I can remember (maybe I was wrong), the IO functions there are not good enough. So I didn't learn the MPI-IO part. Plus, I think physically there is only one process is allowed to write the data to the HDD. So I'd stick to my approach since it is not so complicated either. Thanks for bringing it up. – FortCpp Jun 13 '15 at 17:39