4

I'm trying to open a file for writing using std::ofstream and I want to set it in write-through mode (i.e. like using the "FILE_FLAG_WRITE_THROUGH " provided by CreateFile Win API).

Is there some STL way to achieve it? I don't want to write code based on WinAPI. My target is to disable OS caching and perform writes using different block sizes in order to obtain data related to storage performances. I can't use a standard benchmark tool because the target is to understand how to optimize my write-layer settings for the particular storage I have to rely on.

@Update This is an MWE where I would like to see different save times when changing the value of blk_size:

#include <cstdlib>
#include <fstream>
#include <iostream>
#include <vector>
#include <ctime>

std::vector<unsigned char>
GenerateRandomData(long numBytes) 
{
    std::vector<unsigned char> res(numBytes);
    std::srand(std::time(0));

    for (int i = 0; i < numBytes; ++i)
        res[i] = static_cast<unsigned char>(std::rand() % 256);

    return res;
}

int main(int, char)
{
    // generate random data
    const long dataLength = 1 * 1024 * 1024 * 1024; // 3 GB
    std::vector<unsigned char> outbuf = GenerateRandomData(dataLength);

    // define I/O block size (
    const auto blk_size = 128 * 1024; // 128K
    char blk[blk_size];

    // configure output stream
    std::ofstream ofs;
    ofs.rdbuf()->pubsetbuf(blk, blk_size);
    ofs.setf(std::ios_base::unitbuf);

    // open file to write
    ofs.open("output.dat", std::ofstream::binary | std::ofstream::trunc);

    // write all data performing 512K I/O Operations
    auto ptr_idx = 0;
    auto ptr = reinterpret_cast<char*>(outbuf.data());
    const auto outbuf_size = outbuf.size();

    std::clock_t sw = clock();    

    ofs.write((const char *)&ptr[ptr_idx], outbuf_size);

    ofs.flush();
    ofs.close();

    sw = ( clock() - sw );

    double writtenBytes = static_cast<double>(outbuf.size());
    double writtenMBytes = writtenBytes / (1024 * 1024);
    double testSeconds = static_cast<double>(sw) / static_cast<double>(CLOCKS_PER_SEC);
    double avgSpeed = writtenMBytes / testSeconds;

    std::cout << "Benchmark: written " << writtenMBytes << " MB in " << testSeconds << " sec. => " << avgSpeed << "MB/s" << std::endl;  

    std::getchar();

    return 0;
}

Thank you in advance

Alfatau
  • 233
  • 2
  • 8
  • Related: https://stackoverflow.com/questions/11563963/writing-a-binary-file-in-c-very-fast – jotik Apr 21 '16 at 11:19
  • Almost a perfect question ... then you added a "thanks" signature. D'oh! – Lightness Races in Orbit Apr 21 '16 at 12:40
  • You want to use block-mode IO, but using streams instead? The purpose behind std:: streams is to provide minimal support for all types of IO, not for block-mode efficiency. Is there a reason why you aren't using the block-mode C-compatible fopen()/fwrite()/fclose() instead, since it is at least made for block-IO? – Matt Jordan Apr 21 '16 at 14:25
  • Yes, the reason is that I've an higher level 3rd party c++ library that performs writes calling it's own routines. However there's the possibility to pass an output stream to the library and then it will write on the stream. So what I want to do is to change the I/O block size in order to best fit the actual storage performances. This is probably my only option to fix the performance problem I'm experiencing, without changing the hardware and without make big modifications to the working-and-tested application. – Alfatau Apr 21 '16 at 14:36

1 Answers1

1

Use: std::unitbuf

std::ofstream outfile ("file.txt");
outfile << std::unitbuf <<  "Hello\n";  // immediately flushed
shrike
  • 4,449
  • 2
  • 22
  • 38
  • Ok, thanks. I applied your suggestion this way http://pastebin.com/sTpQUK8g because I'm not using the "<<" operator. The strange thing is that even I change the value of blk_size (e.g. 1K) the save time remains the same. Please, see @Update – Alfatau Apr 21 '16 at 12:36