1

I'm trying to write out an image from a camera to disk. To keep it simple for post processing down stream, we chose to use a CSV format (Other systems we have use this format as opposed to binary). The issue is to write a 1MB image it takes ~1 Second.

I'm looking to speed this up by at-least 10x. Ideally we maintain the CSV format, but we can move to a binary file if needed. This all is running on a WIN 10 machine, and compiled with VS 2017.

The code below is inefficient but prints prints out a rectangle image with ',' delineation between pixels. I'm sure 99% of the inefficiency comes from the Million repetitive IO requests for every loop, but I'm not sure how to append ',' without something like this...

int toCSV(uns16 * bufferCopy, uns32 bufsize, uns32 imgWidth, string directory, string fileout)
{
    std::ofstream outputFile(directory + fileout);
    for (uns32 i = 0; i < bufsize; i++)
    {
        outputFile << (bufferCopy[i]);
        if (i % (imgWidth+1) < imgWidth)
            outputFile << ",";
        else
            outputFile << "\n";
    }
    outputFile.close();
    return 0;
}

An example of the output looks like this (Just with ~400x~600 rows):

32.323,23.456,54.323,45.332
45.343,73.846,35.328,15.842
32.323,23.456,54.323,45.332
45.343,73.846,35.328,15.842

Update:

If I move to a binary format, I'll be using the question below as reference code: Writing a binary file in C++ very fast

MadHatter
  • 133
  • 9
  • Writing numerical data as text is [inherently very slow](https://stackoverflow.com/a/24396176/463827), on top of resulting in very large files. You're much better off moving to any of a number of standard binary formats. – Jonathan Dursi Feb 05 '18 at 17:11
  • @JonathanDursi do you have a simple binary format to recommend? It will be read back by a Python program, so I’m trying to keep it simple to reduce the amount of development needed in both programs. – MadHatter Feb 05 '18 at 17:17
  • I had in mind any of a number of simple bitmap image formats but I notice now that the use of floating point values rules them out (I thought the source code suggests that the data would be unsigned ints?). Are there formats used by others doing similar work as you? If not, you could follow the binary answer you point to in your question, with a header describing the size, writing a Python reader would be easy - use eg numpy.memmap (as in an old answer of mine [here](https://stackoverflow.com/a/5858061/463827) or elsewhere on SO) using the `offset` option to skip over the header. – Jonathan Dursi Feb 06 '18 at 23:00

0 Answers0