How to make C++ code faster: Creating .csv file from camera data using CFile

Question

A high level overview is that 'CFile file's 'file.write()' method gets called for every individual integer data value (line 9) along with line 12, where I write a comma to file.

That means that for 327,680 input data integers, there are 2*327,680 = 655,360 file.write() calls. The code is very slow because of this and as a result, the code takes 3 seconds to create one csv file. How could I improve the efficiency of my code?

Note: I cannot change any declarations of the code. I must use CFile. Also, pSrc is of type uint_16_t and is containing the data that I want to store in the .csv file. The data ranges from 0 - 3000 integer values.

1           CFile file;
2           int mWidth = 512;
3           int mHeight = 640;
4           UINT i = 0;
5           char buf[80];
6           UINT sz = mHeight * mWidth; //sz = 327,680
7           while (i < sz) {
8                  sprintf_s(buf, sizeof(buf), "%d", pSrc[i]); 
9                  file.Write(buf, strlen(buf));
10                 i++;
11                 if (i < sz)  
12                        file.Write(",", 1);
13                 if (i % mWidth == 0) 
14                        file.Write("\r\n", 2);
15  }

All values are outputted in the 640x512 .csv file containing integers representing degrees Celcius.

*That means that for 327,680 input data integers, there are 2*327,680 = 655,360 file.write() calls.* -- Create a string consisting of all of that data and write the string once. Memory is faster than disk (even though disk caching may be involved). — PaulMcKenzie, Jan 30 '19 at 23:44
*Also, I don't know what line 8 and line 9 are doing* -- What do you think those lines are doing? What research have you done to figure out what those lines do? — PaulMcKenzie, Jan 30 '19 at 23:47
*How could I improve the efficiency of my code?* Write less data. Consider creating a binary format because raw binary data is often much smaller than when represented in text. Plus you don't have to convert text <-> number all the time. Note byte order is a concern with binary data as is the variance in the size of integer data types. It is often quicker to read compressed data into memory, decompress it, and then use it than to read uncompressed raw data. — user4581301, Jan 30 '19 at 23:58
In short, since `CFile` seems to be unbuffered IO, write bigger chunks at a time (basically implement some sort of buffering). It doesn't need to be whole file at a time, but even writing chunks of say 16 values should net some visible improvement. Or one write per line... you get the idea. | You're using C++, so even though you have to write to `CFile`, nothing stops you from using c++ streams for the buffering... — Dan Mašek, Jan 30 '19 at 23:59
@user4581301 Raw binary data is *sometimes* smaller. Good note on the compression! — Captain Giraffe, Jan 31 '19 at 00:51
Do you all think I could increase the buffer and add data along with commas? Ex. "345,3456,76,23,54,56" and they write this whole thing to the file and then flush the buffer for next line? Is it possible? — Daniel Ramsey, Jan 31 '19 at 17:25

score 1 · Accepted Answer · answered Feb 20 '19 at 17:31

Just Figured it out! Below is the implementation that seemed to get the job done.

int sz = mHeight * mWidth;

std::string uniqueCSV = "Frame " + to_string(miCurrentCSVImage + 1) + ".csv";
std::string file = capFile + "/" + uniqueCSV;
std::ofstream out;
out.open(file);

std::string data = "";

int i = 0;
while (i < sz) {
    data += to_string(pSrc[i]);
    i++;
    if (i < sz)
        data += ",";
    if (i % mWidth == 0)
        data += "\n";
}

out << data;
out.close();
miCurrentCSVImage++;

Spinkoo · Answer 2 · 2019-01-31T17:51:46.057

0

how about trying this use a string of a whole line size

then at every iteration add your data to the buf and a comma(by concatenating the whole line to the buf) & when you get to

 if (i % mWidth == 0)

write the whole line to the CFile and clear you buf using

something like this

UINT sz = mHeight * mWidth; std::string line = "";
while (int i < sz) { line += std::to_string(pSrc[i])) + ','; i++;
if (i % mWidth == 0) { 
file.Write(line.c_str(), line.size()); 
file.Write("\r\n", 2); 
line = ""; } }

edited Jan 31 '19 at 17:51

answered Jan 30 '19 at 23:49

Spinkoo

2,080
1
7
23

I tired implementing your solution however, I just get 00000... on the first column for 512 rows. Here is the code implemented. char buf[80]; char buf1[1024]; UINT sz = mHeight * mWidth; std::string line = ""; while (int i < sz) { line += sprintf_s(buf, sizeof(buf), "%d", pSrc[i]) + ','; i++; if (i % mWidth == 0) { const char * charLine = line.c_str(); strcpy(buf1, charLine); file.Write(buf1, strlen(charLine)); file.Write("\r\n", 2); strcpy(buf1, ""); line = ""; } } – Daniel Ramsey Jan 31 '19 at 17:26

How to make C++ code faster: Creating .csv file from camera data using CFile

2 Answers2