Faster File Operations C++

Question

So I am making a renderer in c++ and opengl for a class of mine. I am making an animation program for extra credit that will change values in a text file right before my renderer reads them in each frame. My problem is that this section of code isn't writing fast enough

while (clock() < time_end)
    {
        timeStep = clock() + fps * CLOCKS_PER_SEC;
        for(int k=0; k < currOps.size(); k++)
        {
            // increase/decrease each set once for the current timestep
            // a case for each operation
            int pos = currAxis[k];
            if(currOps[k] == "loc")
            {
                opsFile[8+pos] = patch::to_string(atof(opsFile[8+pos].c_str()) + locScale[pos-1]*timeAdjust);
                //edit this value by loc adjust
            }
            else if(currOps[k] == "rot")
            {
                opsFile[4+pos] = patch::to_string(atof(opsFile[4+pos].c_str()) + rotScale[pos-1]*timeAdjust);
                //edit this value by rot adjust
            }
            else if(currOps[k] == "scl")
            {
                opsFile[pos] = patch::to_string(atof(opsFile[pos].c_str()) + sclScale[pos-1]*timeAdjust);
                //edit this value by scl adjust
            }
        }
        currFile.close(); //save file and restart so we don't append multiple times
        currFile.open(files[location[0]].c_str(), ofstream::out); // so we can write to the file after closing
        for(int j=0; j <opsFile.size(); j++)
        {
            // update the file
            currFile << opsFile.at(j);
            currFile << "\n";
        }

        while(clock() < timeStep)
        {
            //wait for the next time steps
        }
    }

Specifically currFile operations at the end. If I take the currFile operations out it will run at the desired fps. FPS is set to .033 so that it does 30 fps. Also it will run fast enough when fps = 0.1. Any optimizations would be great. If need to see any other part of my code let me know and I will upload. The whole thing is around 170 lines. currOps, files, and opsFile are vectors of strings sclScale, rotScale, and locScale are vectors of doubles currAxis is vectors of ints

Interesting. Why does the file need to be written in real-time? — LogicStuff, Nov 19 '15 at 14:23
The `currFile << endl;` statement not only writes a newline but also flushes the data to disk. Chances are that you need just the former, but not the latter. Try using `currFile << '\n';` instead. — Frerich Raabe, Nov 19 '15 at 14:23
Have you profiled your app yet? Is it really the file writing that's the bottleneck, or is it all the ascii->float->ascii conversions you're doing in the loop over "currOps.size()" ? — Matt Godbolt, Nov 19 '15 at 14:25
Other suspect bits: `at( )` - you don't show the variable type but this may be ` std::vector::at` which is slower than `std::vector::operator [ ]` - and `to_string(....c_str())` — MSalters, Nov 19 '15 at 14:25
I will make a few changes using your suggestions with the at, and to_string isn't supported in our compiler so I used a suggestion and added a patch struct and defined to_string. Also, yes when i take out the currFile statements it will run with fps up to 0.006 and make it in time. It has to be written in real time cause every frame my renderer reads in the values from the text files again. — DragonTorchSlash, Nov 19 '15 at 14:27
[That's not how you do it](http://stackoverflow.com/questions/372198/best-way-for-interprocess-communication-in-c). — LogicStuff, Nov 19 '15 at 14:34
We aren't allowed to use boost. Just c++ that is supported in his compiler, that we type ourselves, and opengl. — DragonTorchSlash, Nov 19 '15 at 14:36
Thanks mSalters, that gained me a few frames. Originally it only performed 77% of the transformation with the vector update it performed 81-87% in a few different tests. — DragonTorchSlash, Nov 19 '15 at 14:37
Another optimization is to not use 'formatted' output (which is what `<<`) does. Try replacing writing to `currFile` with `std::string s = opsFile.at(j); currFile.rdbuf()->sputn(s.c_str(), s.size()); currFile.rdbuf()->sputc('\n');`. — Frerich Raabe, Nov 19 '15 at 14:46
That helped. Consistently running 87% now. Gonna try to take out the atof stuff. — DragonTorchSlash, Nov 19 '15 at 14:54
why is opsFile holding `string` rather than `float`? That makes no sense to me. I would have a `vector` for `opsFile`. — Nim, Nov 19 '15 at 15:01
It makes writing out to the file easier because the file is like this, scale 1.0 1.0 1.0 rotation (x, y, z) 45.0 10.0 0.0 translation 0.0 0.0 0.0 — DragonTorchSlash, Nov 19 '15 at 15:03
Also changing the atof meant I had to use another vector to keep track of the string and the double otherwise my write would be even slower. It slowed it to 70% — DragonTorchSlash, Nov 19 '15 at 15:03
But you do know that you can *read and write floats from a text file* right? — Nim, Nov 19 '15 at 15:04

score 3 · Accepted Answer · edited May 23 '17 at 11:51

Here are some general changes which may help:

I would convert the curOps to an enum rather than a string (save you the string comparisons.) Looks like you should pre-process that container and build a sequence of enums (then your code in the loop becomes a switch)
Don't use vector<string> for curOps, simply read the floats from the file and write the floats out - this will save you all those pointless conversions to and from string. If you wanted to take it further, convert the file to binary (if you are allowed by the exercise), and store a simple structure which contains the floats you need and use memory mapped files (you don't need boost for that, it's straight forward just using mmap!)

Before going down the mmap route - try the float read/write from file. For example, let's say that each "row" in your file corresponds to something like the following:

struct transform {
  double x, y, z;
};

struct op {
  transform scale, rot, loc;
};

Declare a bunch of stream in/out operators for these (for example:

std::ostream& operator<<(std::ostream& os, const transform& tx) {
  return os << tx.x << ' ' << tx.y << ' ' << tx.z;
}
std::istream& operator>>(std::istream& is, transform& tx) {
  return is >> tx.x >> tx.y >> tx.z;
}

(a similiar set will be required for op)

Now your vector is std::vector<op>, which you can easily stream in and out from your file, for example, to read:

op i;
while(file >> i) { curOps.push_back(i); }

To write:

for (auto o : curOps) file << o << '\n';

If this is still not enough, then mmap (btw - it's possible on windows too: Is there a memory mapping api on windows platform, just like mmap() on linux?)

The switch case and enum change made it consistently 87% now. mmap sounds like a great idea, but do you have an example? From what Im reading it's linux only. — DragonTorchSlash, Nov 19 '15 at 15:35
@DragonTorchSlash: It's called [`CreateFileMapping`](https://msdn.microsoft.com/en-us/library/windows/desktop/aa366537(v=vs.85).aspx) — MSalters, Nov 19 '15 at 15:50
Thanks to everyone who helped. I am a lot closer now, and I think after I get the filemapping figured out and implemented I will have it running fast enough. — DragonTorchSlash, Nov 19 '15 at 17:49

score 0 · Answer 2 · edited May 23 '17 at 11:51

0

Try using the functions in stdio.h instead. iostreams are terribly inefficient.

In your case all you will need is fseek() ... fputs(). Avoiding reopening the file each time should actually help quite a lot.

edited May 23 '17 at 11:51

Community

1
1

answered Nov 19 '15 at 14:55

Ben Voigt

277,958
43
419
720

Faster File Operations C++

2 Answers2