Writing a file larger than 2Gb

Question

I write a file storing all the raw data I read from two external sensors and I realized that program stops writing data when the file gets 2Gb, the program continuous working but it does not write anything more in the file.

I am using the next code (it is one threads that waits the signal and writes in the file):

while(1)
{
    //Lock, Wait the signal and unlock
    pthread_mutex_lock(&mutex_WriteOFIMUandGPS);
    pthread_cond_wait(&cond_WriteOFIMUandGPS, &mutex_WriteOFIMUandGPS);
    X_CopySnapshot = X_Snapshot;
    Y_CopySnapshot = Y_Snapshot;
    Z_CopySnapshot = Z_Snapshot;
    Vx_CopySnapshot = Vx_Snapshot;
    Vy_CopySnapshot = Vy_Snapshot;
    Vz_CopySnapshot = Vz_Snapshot;
    HPL_CopySnapshot = HPL_Snapshot;
    HDOP_CopySnapshot = HDOP_Snapshot;
    StdPosX_CopySnapshot = StdPosX_Snapshot;
    StdPosY_CopySnapshot = StdPosY_Snapshot;
    StdPosZ_CopySnapshot = StdPosZ_Snapshot;
    pthread_mutex_unlock(&mutex_WriteOFIMUandGPS);


    //Get Time and Date
    now = time(0);
    localtm = localtime(&now);

    //Get millis
    gettimeofday(&tp, NULL);
    ms = round(tp.tv_usec / 1000);
    sprintf(buf,"%03d",(int)ms);

    //Empty strings
    DateTime = "";
    text.clear();
    text.str(string());

    //Store Data and Time information in a string
    text << localtm->tm_year+1900 << "/";
    text << (((localtm->tm_mon+1)<10) ? "0" : "") << localtm->tm_mon+1 << "/";
    text << ((localtm->tm_mday<10)? "0" : "") << localtm->tm_mday << ",";
    text << ((localtm->tm_hour<10)? "0" : "") << localtm->tm_hour << ":";
    text << ((localtm->tm_min<10)? "0" : "") << localtm->tm_min << ":";
    text << ((localtm->tm_sec<10)? "0" : "") << localtm->tm_sec << "."<< buf;
    DateTime = text.str();

    //Save data
    fprintf(fid,"%s,"
            "2,6,0,"                            // Alg_ID,SolStatus,EGNOSStatus,
            "%12.3f,%12.3f,%12.3f,"             // XyzUKF[0],XyzUKF[1],XyzUKF[2],
            "%8.8f,%8.8f,%8.8f,%7.18f,"         // V_UKF[0],V_UKF[1],V_UKF[2],HPL,
            "0,0,%5.2f,nan,nan,nan,nan,"        // NumSat,NumSatEx,HDOP,PDC1,PDC2,PDC3,PDC4,
            "%7.2f,%7.2f,%7.2f,"                // StdPos[0],StdPos[1],StdPos[2]
            "0,0,0,"                            // StdVel[0],StdVel[1],StdVel[2],
            "1,0,"                              // TypePositioning[0],TechUsedPos,
            "0,0,0,0,0,"                        // Observables
            "0,0,0,0,0,"                        // Observables
            "0,0,"                              // VPL,VDOP
            "0,"                                // TechRec
            "0,0,0,0,0,"                        // Observables
            "0,0,0,0,0\n",                      // Observables
            DateTime.c_str(),
            X_CopySnapshot, Y_CopySnapshot, Z_CopySnapshot,
            Vx_CopySnapshot, Vy_CopySnapshot, Vz_CopySnapshot,HPL_CopySnapshot,
            HDOP_CopySnapshot,
            StdPosX_CopySnapshot, StdPosY_CopySnapshot, StdPosZ_CopySnapshot);
}

I do not know if there is any way to write files larger than 2Gb and how I should achieve it.

Many thanks for your help.

Sounds like a 32bit int issue... Did you took a look at [2GB limit on file size when using fwrite in C?](http://stackoverflow.com/questions/730709/2gb-limit-on-file-size-when-using-fwrite-in-c) — Thomas Ayoub, Jun 30 '16 at 07:46
Maybe you are using a file system with a 2G limit on file size? FAT? — Thomas Padron-McCarthy, Jun 30 '16 at 07:49
You are right, it is not a c code, but the function to write the file is C, that why I wrote it. — Julen Uranga, Jun 30 '16 at 07:51
As they said : 32bit and /or filesystem issue, if you have to keep these settings, track your file sile size, and spawn a new one whenever size reaches a threshold near 2GB, that way you'll split the data across as many files as it is needed. — Gar, Jun 30 '16 at 07:55
Okay! I will check the size of the file and create a new one when it's close to 2Gb. — Julen Uranga, Jun 30 '16 at 08:24
I think your answer may be here: http://stackoverflow.com/questions/730709/2gb-limit-on-file-size-when-using-fwrite-in-c — LordWilmore, Jun 30 '16 at 08:37
Try `df -T` to check if the filesystem you are writing on is a limited Microsoft-invented `FAT` type or a proper Unix filesystem like `ext2/3/4`. — Mark Setchell, Jun 30 '16 at 10:01

score 0 · Answer 1 · answered Oct 11 '16 at 07:28

As you state that you use linux: man 2 open:

       O_LARGEFILE
              (LFS) Allow files whose sizes cannot be represented in an off_t (but can be represented
              in  an  off64_t)  to  be opened.  The _LARGEFILE64_SOURCE macro must be defined (before
              including any header files) in order to obtain this definition.  Setting the _FILE_OFF-
              SET_BITS  feature  test  macro  to  64 (rather than using O_LARGEFILE) is the preferred
              method of accessing large files on 32-bit systems (see feature_test_macros(7)).

Otherwise, file operations are limited to 2GiB.

Writing a file larger than 2Gb

1 Answers1