0

fstream.write() gets stuck during perform to remove directory. It seems that my HDD is so busy while write and remove at the same time. So how can I avoid this stuck, any timeout parameter for the write() function?. Below is my procedure code:

#include <fstream>
#include <iostream>
int main()
{
    std::ios_base::sync_with_stdio(false);
    fstream myfile = std::fstream("sample.txt", std::ios::out | std::ios::binary);
    //Start thread remove old data
    INT64* paramsInput = new INT64[2];
    char* dir = "D:\";
    paramsInput[0] = (INT64)dir;
    paramsInput[1] = 50; //GB
    _beginthreadex(Null, 0, &remove_old_data, (VOID*)paramsInput);

    int size = 0;
    char* data = NULL;
    while (true)
    {
        data = NULL;
        size = getData(data); //data is available every 10 ms
        if(size > 0 && data != NULL) //size ~= 30 KB
        {
            myfile.write(data, size); //write data to file
        }       
    }   
}

UINT32 __stdcall remove_old_data(VOID* _pArguments)
{
    char* dir = (char*)_pArguments[0];
    int   freeSpaceThreshold = _pArguments[1];
    delete[] _pArguments;
    while(true)
    {
        int curFreeSpace = GetFreeSpace(dir);
        if(curFreeSpace < freeSpaceThreshold )
        {
            //remove old files and directory here
            ClearData(dir);//File size is about 10 MB, 40,000 files in dir
        }
        Sleep(10000);
    }   
}
vominhtien961476
  • 317
  • 4
  • 17

1 Answers1

1

It's a bit tough to say exactly what the bottleneck is in your case, although there could be several. Deleting files can be time consuming, particularly when doing several in a directory with many files. Writes to a mostly full drive are also slower, as it takes the OS longer to find empty space to store new data, and those chunks are smaller.

Here are some suggestions to improve the performance, in no particular order:

1) Use an SSD. This eliminates almost all the latency for hard disk access.

2) Use OS API functions for file access, and use unbuffered writes. This will avoid filling the disk cache with data that is only written and not read again, allowing the directory information to stay cached.

3) Use multiple subdirectories to store your data. File access in a directory can slow down if the directory size gets to be too large.

4) Cache the 30K data chunks locally, keeping multiple chunks queued up for writing, and only write them out when remove_old_data is not cleaning up the directory.

1201ProgramAlarm
  • 32,384
  • 7
  • 42
  • 56
  • 1. I'm writing data about 30 cameras to hard disk. So do think SSD is the best option in case we write/read/delete many times. – vominhtien961476 Nov 22 '17 at 07:07
  • 2. I will try with a OS API function to see the result. 3. The data is stored with multiple sub-directories like: date -> camera_index -> hour -> fileperminutes. 4. what is the best size data for each writing? – vominhtien961476 Nov 22 '17 at 07:15
  • @vominhtien961476 The SSD eliminates the latency from seeking, which is where most of the delay comes from. The chunk size wouldn't be an issue, either. They cost more and have a smaller capacity, and will eventually wear out. With a non-SSD, a fewer number of large blocks is generally better than more frequent, smaller chunks, but the best size depends on a lot of factors. You'd have to experiment. – 1201ProgramAlarm Nov 22 '17 at 15:28
  • Finally, there will be no timeout to prevent a blocking for fstream.write()?. I expected that my app still works in worse case at least no hang no stuck. I've searched for similar post and see that there was not much difference between File* and fstream https://stackoverflow.com/a/11564931/1226316. what is your idea ? – vominhtien961476 Nov 23 '17 at 00:51