Writing Floats to a CSV File Using C++ Efficiently

Question

I am currently using Microsoft Kinect C++ to get skeletal coordinate data (x,y,z) along with a time_stamp. So this translates to 61 columns of data per write.

What I am currently doing is setting up the csv file w/ the column headers at the beginning of the main() method. Then I run an infinite while loop to process color/depth frame skeleton data and save the skeleton data into this csv file: pseudo (cuz my code is too long): main method() { ostream myfile; myfile.open(cvsfile); setup myfile stuff;

...
while(1) {
    get color frame ready;
    get depth frame ready;
    get skelframe() ready;

    ... other stuff that isn't related to the problem
}
end of main method} 
... 
skelframe() {
    setsup skeleton frame and image;

    ...
    get the 60 data points (x,y,z) of 20 joints;
    ostream myfile;
    myfile.open(csvfile)
    myfile<<all 60 data points + time_stamp<<endl;
}

So in the skelframe() method, I am setting up the skeleton and then obtaining the x,y,z coordinates. I have a couple of questions:

Should I pass in myfile to the skelframe(), so that I don't have to do the setup for the csv file every time the method loops through the while(1)? If so, do I just pass it as skelframe(..., myfile), or is there some other fancy way of doing this?
When I am writing all those 60 datapoints into the csv file, is there an efficient way to do this using a buffer of some sorts? I thought of using a fixed array size of 60 floats+1 string with ","s and then writing that entire buffer to the csv file; looked into stringstream, but no idea how to implement. Need some help w/ example code cuz pretty lost and noob at this lower level coding concepts.

Thanks!

Are you sure that writing to `myfile` is a bottleneck? If it isn't, you're doing premature optimization. — Jashaszun, Aug 06 '14 at 22:10
Also note `std::ofstream` already makes use of buffered output, unless you flush explicitly. — πάντα ῥεῖ, Aug 06 '14 at 22:12
well right now, I am doing this in the skelframe() method: myfile<<1st thing<<","<<2nd thing<<","....60 times< — ajl123, Aug 06 '14 at 22:20
Question 1: is it really too slow, or are you just assuming it will be without having tried it? Question 2: have you profiled it to see what the slow part is? — Mark Ransom, Aug 06 '14 at 22:23
Replace endl with '\n'. http://stackoverflow.com/questions/213907/c-stdendl-vs-n — drescherjm, Aug 06 '14 at 22:31
You can improve efficiency by outputting to `std::ostringstream`, and when the 60 are written, output the string to the file. This reduces the file operations from 60 to 1 for each row. — Thomas Matthews, Aug 06 '14 at 22:37
Will this work if the 60 coordinate data are floats and then the time_stamp is a string? If not, what would be a good workaround? Also have tried running program, and the program is too slow for what I need. :/ — ajl123, Aug 06 '14 at 23:27
`stringstream` works like a file stream, you just use `cout << whatever` to put stuff in it, and then use `myfile << yourstringstream.str() << '\n';` to write the whole string to the file at once. But as mentioned before, are you sure this is the problem? If you comment out the line that you use to write into file, does it get faster? How much? — triple_r, Aug 06 '14 at 23:47

score 0 · Accepted Answer · answered Aug 27 '14 at 05:12

Writing to myfile was not the bottleneck, but I switched the code around, so that the file remained open at all times until my program was done running.

The real bottleneck was the fact that I was saving images. I changed the image types from ".png" to ".bmp", which increased the size of the images, but increased the speed of saving. I also changed the data collection stream of skeleton from cv_8UC3 to CV_8UC1 which decreased the amount of sensors in the Microsoft Kinect my skeletal data was gathering from (it really only needs 1 data stream).

Thanks for the help.

Writing Floats to a CSV File Using C++ Efficiently

1 Answers1