0

I have some iterative Fortran code which at each integration step produces some output. What is the best practice in terms of speed/accuracy for getting each of these steps saved to disk?

My current approach involves declaring some large array, at each integration step saving the output to a row of the array, and then finally saving a cropped version of the total array to file. A psuedo-example is shown below. 

program IO_example

integer, parameter :: dp = selected_real_kind(33,4931) 

integer(kind=dp) :: nrows = 1e6, ncols = 6

real(kind=dp), dimension(nrows,ncols) :: BigDataArray
real(kind=dp), dimension(ncols) :: RowVector
real(kind=dp), dimension(:,:), allocatable :: SmallDataArray

integer(kind=dp) :: i !for iterating


i = 1
do while (condition)

!Update RowVector

BigDataArray(i,:) = RowVector
i = i+1

enddo


!First reallocate to create a smaller array
allocate(SmallDataArray(i,ncols))
SmallDataArray = BigDataArray(1:i, :)

!Now save
open(unit=10,file=BinaryData,status='replace',form='unformatted')
write(10) SmallDataArray
close(10)

end program IO_example

Now this works fine, but my question is is this the best way to do this, or is some other approach more favourable? By best I am particularly referring to speed (how much does writing to array and writing to file slow down the code), although accuracy issues are also important (I understand these are avoided by writing in binary unformatted. See this StackOverflow answer).

Some potential issues I can foresee is the SmallDataArray being greater than the RAM (especially in quad precision) and so unable to write to disk. In addition, the number of iterations could become greater than nrows (in this case I suppose one can just increase nrows, but at what point does this start to impact performance?)

Thanks in advance for any help.

user1887919
  • 829
  • 2
  • 9
  • 24
  • Is `AllData(1:i,:)` supposed to be `BigDataArray`? Also, why bother allocating and transferring values into `SmallDataArray` at all? – Matt P Aug 07 '19 at 16:18
  • Yes it was - edited! Re second point, my logic was that otherwise we save all of `BigDataArray` to file, including the zero (i.e. un-populated) rows. If `i` << `nrows` this could significantly reduce the file size. Or I have misunderstood something? – user1887919 Aug 07 '19 at 16:27
  • 1
    Since you have `i`, perhaps you can just write to file directly from `BigDataArray(1:i,:)`. – Matt P Aug 07 '19 at 16:29
  • Yep - that is a much smarter approach and I am a fool! Thanks! – user1887919 Aug 07 '19 at 16:30

1 Answers1

2

This is probably an extended comment, taking advantage of some formatting, and verges close to an opinion, but there are one or two matters which are amenable to measurement and which you might care to test for yourself.

I'm not sure what role BigDataArray plays in your code, since you don't seem to need all the data in memory after it has been computed. You could probably drop it altogether and simply accumulate results into SmallDataArray. If BigDataArray has 10^6 rows, then maybe give SmallDataArray 10^5 rows, and fill it up 10 times. Or, if you're not certain at the outset how many rows to allocate to Big, then don't, just set Small to 10^5 and fill it up as many times as necessary, exiting when the computation converges.

(And don't get hung up on the numbers I've chosen, the best size for Small is something you probably ought to experiment with.)

Once the code has filled Small write it to file, go back to row 1 and carry on.

If you follow this approach you will eliminate at least a couple of potential performance issues; the repeated allocation of Small (not sure what that's about anyway), and the movement of data when you copy a bunch of rows from Big to Small (which gains you nothing in terms of computation performance and is unnecessary for writing the data to the file).

As you seem to know, the rule when writing data to file (which is very slow computationally) is to write large volumes in one go, but it's difficult to state how large that volume should be without at least some measurements and some testing, so go measure and test.

By dropping Big altogether you remove that burden from the memory while the code runs. And if you do need all of Big at the end of the calculation, you could always read it back in (subject to memory being available of course).

Finally, let me get some retaliation in first: if your response to this 'answer' is something akin to Oh, that doesn't answer my real question, it only answers the simplified question I asked but I have all these other issues to consider would you mind taking a look at these too ... then you can take it that my response to that is (a) unprintable and (b) boils down to Yes, I would mind

High Performance Mark
  • 77,191
  • 7
  • 105
  • 161