Fortran runtime error: I/O past end of record on unformatted file

Question

I am having a problem with a Fortran 90 code. The code reads a huge amount of data using the read statement. The error message points to the last line of this piece of code:

        open(1,file=trim(filename),action='read',form='unformatted',status='old')
        read(1) np    ! np is a 6 element array of integers*4
        npart = np(species)

        if (sum(np(1:species-1))>0) then
            read(1) (xempty,yempty,zempty,i=1,sum(np(1:species-1)))
        end if
        allocate(x(npart),y(npart),z(npart),mass(npart))

        write(*,*) "number of particles: ", npart
        read(1) (x(i),y(i),z(i),i=1,npart

The error message appears while running the code. It is:

Fortran runtime error: I/O past end of record on unformatted file

I think this problem is related with the record length limit in Fortran 90, but I am not sure how to solve this problem, and how to find or modify the record limit in gfortran. The error message does not show up if I use a smaller filename. In other words, when npart is smaller.

I am using the following gfortran compiler:

Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-
6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-
5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-
c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-
build-id --libexecdir=/usr/lib --without-included-gettext --enable-
threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --
enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-
time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object -
-disable-vtable-verify --enable-libmpx --enable-plugin --with-system-
zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo 
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-
java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --
with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-
arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --
enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 
--with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --
with-tune=generic --enable-checking=release --build=x86_64-linux-gnu -
-host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)

The input file is in binary format, and it was created using a C code. It is not possible for me to know the compiler used to create the file. The last line of the code : read(1) (x(i),y(i),z(i),i=1,npart) reads positions of particles which are on real( 4 bytes) format. Additionally, the error does not appear when I used an input file with smaller number of particles, in other words, when npart is smaller.

might be related to this: https://stackoverflow.com/questions/19464417/reading-strings-from-unformatted-files-variable-record-length — piiipmatz, Sep 30 '17 at 08:03
Please use tag `fortran` for all Fortran questions. Add tag fortran90 to questions specific to the old Fortran 90 (not this question). — Vladimir F Героям слава, Sep 30 '17 at 12:56
There is no inherent "record length limit in Fortran 90". How was this file written, using which compiler on which platform? The code you show assumes that all the data was in a single record - what values of npart fail and what values work? What are the declarations of x, y and z? — Steve Lionel, Sep 30 '17 at 15:08
Hi, the code works while using a input file with a smaller number of particles, that is a smaller npart, or when I reduce the reading interval, for example, If I do read(1) (x(i),y(i),z(i),i=1,int(npart/100)) , there is not error message — Guido, Sep 30 '17 at 16:36
I thought this issue is related with the following problem: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=20257 , are you sure there is not "record length limit" in Fortran 90 when the code is compiled with gfortran ? — Guido, Sep 30 '17 at 16:50
C binary files are NOT compatible with way you are reading them unless the C code was specially (and unusually) adopted to write Fortran files. C files are compatible with Fortran 2003 stream files. However, you put back the [tag:fortran90] files which I removed. Are you **really 100% sure** you need Fortran 90? That is highly unusual these days. Fortran 90 is old and obsolete. — Vladimir F Героям слава, Sep 30 '17 at 17:16
Possible duplicate of [Fortran unformatted file format](https://stackoverflow.com/questions/8751185/fortran-unformatted-file-format) — Vladimir F Героям слава, Sep 30 '17 at 17:17
There may be a bug in the implementation, there may be an incompatibility with the data file you have, but "Fortran 90" (or Fortran in general) has no specific limit on the length of a record. If you want to ask if gfortran has a limit, that's a separate question, but the answer that it has no practical limit in that it supports records with lengths in excess of many gigabytes. — Steve Lionel, Sep 30 '17 at 23:18
Hello, @SteveLionel, I am using gfrotran as a compiler and each of the position arrays (x , y, z) of the problematic read line is a little bit larger than 1 GB. Do you where I could check what is the length of record when using the gfortran compiler? — Guido, Oct 03 '17 at 03:35
@VladimirF Yes, I am sure that the code is a Fortran 90 code. The code didn't show error messages while running with input file obtained with the same code, therefore, I think that the C binary files incompatibility might not be the problem. Do you have a source for that? Where can I find more about hat incompatibility? — Guido, Oct 03 '17 at 03:41
The point is whether you **need** to use Fortran 90 features only in year 2017, whatever the present code uses. C stream files are supported in Fortran 2003. Have you noticed the close vote and the link https://stackoverflow.com/questions/8751185/fortran-unformatted-file-format ? See more on the right in **Linked** and **Related** from there. — Vladimir F Героям слава, Oct 03 '17 at 06:17
@Guido , I am not a gfortran expert but I do know that at some time in the past gfortran added support for very large records in a manner compatible with Intel Fortran. If each of the arrays is over 1GB, you are into the territory where this support matters. You still have not told us how the record was written - in what language and with which compiler (including version). — Steve Lionel, Oct 03 '17 at 19:30
Hello, the binary file was written using a C code( The name of the code is Gadget 2,a code for running N-body simulations). The compiler used was: GCC: (GNU) 4.6.1 20110908 (Red Hat 4.6.1-9) — Guido, Oct 04 '17 at 00:48
@SteveLionel, I found the information about the gfortran default record length: https://gcc.gnu.org/onlinedocs/gcc-5.1.0/gfortran/GFORTRAN_005fDEFAULT_005fRECL.html#GFORTRAN_005fDEFAULT_005fRECL acoording to this, it is 1GB. So, do you think that was the problem? — Guido, Oct 04 '17 at 16:01
@Guido, to be honest I am not sure. That is for creating files, not reading them, though. You still haven't told us how this file was created. — Steve Lionel, Oct 06 '17 at 16:39
@SteveLionel, I put on the comment above that the C compiler used was: GCC: (GNU) 4.6.1 20110908 (Red Hat 4.6.1-9) — Guido, Oct 07 '17 at 03:55
Oh, you used a C program to write the file read by Fortran with unformatted I/O? I can well believe that would fail. I assume that you wrote the record lengths before and after the data as 32-bit integers, but when the size gets over 1GB, gfortran (like Intel/Compaq Fortran), uses "segments" to split the data among multiple writes. — Steve Lionel, Oct 09 '17 at 01:04
Hello @SteveLionel, thank you for your answer. I think that I am starting to understand the problem. So, when the written file have a size bigger than a certain amount(defined by the compiler version), it is written in segments? and as a consequence, the record lengths are also stored on segments? So that should be the reason why this code works fine with output files obtained with the same code but with a small size, so with data not written in segments?. — Guido, Oct 11 '17 at 22:05
@SteveLionel Is what you are saying is related to this post: https://stackoverflow.com/questions/15608421/inconsistent-record-marker-while-reading-fortran-unformatted-file — Guido, Oct 11 '17 at 22:08
@Guido there are multiple ways to solve it. 1) You can write the file in the manner gfortran expects, with the segments (keeping in mind that if the length is shorter, not to use segments. 2) You can restructure the writing and reading programs to not write a single long record 3) Not sure this will work for you, but you can try writing the data without the record lengths and then in the Fortran code add ACCESS='STREAM' to the OPEN. This will read the data raw without a record structure. — Steve Lionel, Oct 12 '17 at 23:19
@SteveLionel maybe I have a similar issue, I can't read "with" `gfortran` on an IBM machine, an unformatted file written "by" `ifort` on an Intel machine: https://stackoverflow.com/questions/62662469/fortran-runtime-error-i-o-past-end-of-record-on-unformatted-file-reading-bina is there something I could do? — andreagalle, Jun 30 '20 at 16:59
@andreagalle, I just now saw your question. I see you got some help in the other thread. ifort supports a CONVERT= option in OPEN with one of the possible values being 'BIG_ENDIAN'. This is likely what you need here. It not only converts the data but also treats the unformatted record lengths as big-endian. A caveat is that ifort will do the conversion only for I/O statements that have individual variables in the I/O list; if you use a derived type, it will not convert the components. I think gfortran will do that. ifort also supports 'IBM', but that's the 360/370 series format. — Steve Lionel, Jul 18 '20 at 13:35
thank you @SteveLionel. I'm already compiling the whole program with the `-fconvert=big-endian` or `-convert big_endian` depending on which compiler I'm using. I'll try out this I/O `CONVERT=` option in the OPEN statements. Fortunately I don't have any derived type in the restart file! — andreagalle, Jul 21 '20 at 10:12

Fortran runtime error: I/O past end of record on unformatted file

0 Answers0