How can I switch between fstream files without closing them (Simultaneous output files) - C++

Question

I have a little C++ issue that I couldn't solve by browsing online. Here is my code (extracted):

if(File.is_open()) {
    while(!File.eof())  {
        i++;
        getline(File,Line);
        if(i>=2) {                //Skip Headers
            int CharCount=0;
            for(int CharPosition=0; CharPosition<Line.size(); CharPosition++)                       {
                if(Line[CharPosition]==',') {
                    Length=CharPosition;
                    break;
                }
            }
            NameText=Line.substr(0,Length);
            Path= Path_Folder + "\\" + NameText + ".csv";
            if(!CheckExistance(Path.c_str())) {
                fstream Text_File;
            }
            Text_File.open(Path, fstream::in | fstream::out | fstream::app);
            Text_File<<Line<<"\n";
            Text_File.close();
        }
    }
}

This code is working fine, but I would like to change the fact that it closes the Text_File every time it goes in the while loop.

Basically, this program split a big input file in a lot of smaller files. As my smaller files get bigger and bigger, the execution gets slower and slower (normal). My goal is then to let all the smaller files (Text_File) opened in this while loop and just switch the fstream pointer (pointer?) from one to another.

I tried to change as:

...

NameText=Line.substr(0,Length);
Path= Path_Folder + "\\" + NameText + ".csv";

if(!CheckExistance(Path.c_str())) {
    fstream Text_File;
}

if(!Text_File.open()) {
    Text_File.open(Path, fstream::in |fstream::out | fstream::app);
}

Text_File<<Line<<"\n";
\\Text_File.close();

...

But it is working on the same Text_File no matter what NameText is. So I am guessing that the pointer of the fstream Text_File doesn't change. What do I need to be then? Rest the pointer? How?

Thank you, all!

Not sure it is relevant but I am working with Microsoft Visual C++ 2010 Express. In addition, I am not a programmer neither by education nor by living, so if you can explain it without too advanced words, I'll appreciate.

I feel like a pointer to one (or each) of the already declared `Text_File`s would work — James Hurley, Jul 26 '13 at 19:38

sehe · Accepted Answer · 2013-08-06T23:32:38.953

It looks like you would like to juggle the filebufs on an ostream object.

Now, the only obstacle is that ostream or basic_filebuf<char> aren't copyable types, so you can't put them into a map (by filename) directly. This is easily worked around by creating a little Holder type:

struct Holder {
    Holder(std::string const& path) 
        : buf(std::make_shared<std::filebuf>())
    { 
        buf->open(path.c_str(), std::ios::out | std::ios::app);
    }
    std::shared_ptr<std::filebuf> buf;
};

std::map<std::string, Holder> buffers;

Now the complete program (tested) would look like this:

#include <fstream>
#include <sstream>
#include <iostream>
#include <map>
#include <memory>

const std::string Path_Folder = ".";

int main()
{
    std::istream& File     = std::cin; // just for example
    std::filebuf  dummy;
    std::ostream  TextFile(&dummy);

    struct Holder {
        Holder(std::string const& path) 
            : buf(std::make_shared<std::filebuf>())
        { 
            buf->open(path.c_str(), std::ios::out | std::ios::app);
        }
        std::shared_ptr<std::filebuf> buf;
    };

    std::map<std::string, Holder> buffers;
    int i = 0;

    std::string   Line;
    while(getline(File, Line))
    {
        if (i++<2)
            continue; //Skip Headers

        auto NameText = Line.substr(0, Line.find(','));
        auto Path = Path_Folder + '/' + NameText + ".csv";

        // open, only if not allready opened
        auto found = buffers.find(NameText);
        if (end(buffers) == found)
            found = buffers.insert({ NameText, Path }).first;

        TextFile.rdbuf(found->second.buf.get());

        TextFile << Line << std::endl; // notice implicit std::flush in std::endl
    }

    // all files are automatically closed here
}

Three more notes:

files get automatically closed when the buffers map goes out of scope.
you might need to add explicit flushes when switching rdbuf() like this, if you don't end your lines with an implicit std::flush (like with std::endl).
dummy only exists to have an ostream object that we can switch the buffer of

I tested this with the following input:

Header Row #1
Header Row #2
Jack,1,some data
Jill,2,some more data
Jack,3,not reopening :)
Jill,4,jill still receiving output
Romeo,5,someone else reporting

Now, I got the following output: see it live at Coliru

/tmp$ rm *.csv
/tmp$ make && ./test < input.txt && tail *.csv

g++ -std=c++11 -Wall -g test.cpp -o test
==> Jack.csv <==
Jack,1,some data
Jack,3,not reopening :)

==> Jill.csv <==
Jill,2,some more data
Jill,4,jill still receiving output

==> Romeo.csv <==
Romeo,5,someone else reporting

Fixed and further simplified after realizing you wanted to have files created if they didn't exist yet. **And** removed the use of Boost Filesystem (as `CheckExistance` wasn't needed after all). **[See it all live at Coliru](http://coliru.stacked-crooked.com/view?id=1af27de002d18cf2ded1e05f58942835-9c316e88ae784971c383263e9035353b)** — sehe, Jul 28 '13 at 23:34
Improved code to be less confused about filebufs. Also fixed potential multi-open issue; (I wanted `map::emplace` but GCC doesn't have it yet...) [Live on Coliru](http://coliru.stacked-crooked.com/view?id=9c388874ad76178c7bcdd49beeae01a9-9c316e88ae784971c383263e9035353b) — sehe, Aug 06 '13 at 23:33

score 2 · Answer 2 · edited Jul 29 '13 at 08:15

2

Note: it looks like your Text_File is out of scope. I guess you declared it somwhere else in the code. So, this line is useless:

if(!CheckExistance(Path.c_str())){fstream Text_File;}

To access multiple file streams you can use this simple class which utilizes the std::map data structure:

#include <iostream>
#include <map>
#include <string>
#include <fstream>

class StreamWriter
{
    typedef std::map<std::string, std::fstream> StreamMap;
    static StreamMap Files;

public:
    static std::fstream& GetFile(const std::string& filename)
    {
        std::fstream& stream = Files[filename];
        if (!stream.is_open())
        {
            stream.open(filename, std::fstream::in
                   | std::fstream::out | std::fstream::app);
        }
        return stream;
    }
};

StreamWriter::StreamMap StreamWriter::Files = StreamWriter::StreamMap();

Then, access to files is as simple as:

StreamWriter::GetFile("C:/sample1.txt") << "test";

That's it.

edited Jul 29 '13 at 08:15

sehe

374,641
47
450
633

answered Jul 26 '13 at 19:11

podkova

1,019
7
16

-1 You cannot put `fstream` objects into a `map` like that (because they're not copyable/movable). See my answer for a fully tested solution – sehe Jul 28 '13 at 23:40
Ouch, I've just tried to compile this code in gcc, and you are right... However, Visual C++ version of `std::map` compiles fine, no errors, no warnings. – podkova Jul 29 '13 at 07:23
Interesting :) It should not compile. Although since C++11 `fstream` is supposed to be movable. I'm revoking my -1 out though, because you had tested it and it might work for someone. – sehe Jul 29 '13 at 08:01

score 1 · Answer 3 · edited May 23 '17 at 12:24

What I would do is use std::map or std::unordered_map to map names to fstream objects.

map<string, fstream> files;

...

while(getline(File,Line)) // don't use while(File.eof())
{
    ...

    if( files.count(NameText) == 0 ) // checks for the existence of the fstream object
    {            
        files[NameText].open(Path, fstream::in | fstream::out);
    }

    files[NameText] << Line << "\n";
}

See here for why I changed the condition for the while loop.

Your OS may have trouble having that many open files at once. Perhaps you could try something like this.

Alongside your map, keep a list of the names of the files that are open. Each time you need to write to a file, first search for it in your list, remove it and add it to the front of the list. If it's not there, just add it to the front of the list. Check to make sure the file is open. If it's not, then try to open it. If opening it fails, then one by one, remove items from the back of the list, close the corresponding file to that item, and try to open the current file again. Repeat until opening the file succeeds.

Doing this will ensure that the most frequently written to files stay at the front of the list and remain open. The less frequently written files will move to the back, and eventually be closed. The search for the file in the list is not optimal (O(n)), but since we're dealing with writing to files here, which is a much more expensive operation, you shouldn't notice any kind of perf hit.

Hi Benjamin, thank you. I tried your solution. The program goes into the if condition and adds the NameText to the fstream map, but it doesn't create physically the NameText file. My folder is empty. Any idea? — Vince, Jul 27 '13 at 00:27
Hi Benjamin, I tried your code and it does work now (I had ios:app in the open line). But it seems to skip some files and I don't understand why. I got 508 files instead of 545. When I add the line "files[NameText].close();" at this end of the loop while, it works and I do get 545 files, but then I am at the same point than before: the files are opened and closed every time. — Vince, Jul 31 '13 at 14:33

score 0 · Answer 4 · edited May 23 '17 at 11:53

0

You are trying to reuse the Text_File fstream. To do this, you have to do a close() to flush the stream, after you are done writing to a csv file. Please see this question: C++ can I reuse fstream to open and write multiple files?

Also: Here's my Google search for this question: http://goo.gl/Oy5KKM

edited May 23 '17 at 11:53

Community

1
1

answered Jul 26 '13 at 18:45

Sabuncu

5,095
5
55
89

score 0 · Answer 5 · answered Jul 26 '13 at 18:58

Note that Text_File is a variable and like all variables you can have more than one with the same type. If you need to manage several different files, you can even use std::fstream in any of the standard containers such as std::vector or std::map. Also, you should consider breaking your code down into smaller more manageable parts. For example, you can create a function which takes an std::fstream& as a parameter. This allows the rest of the program to control which std::fstream& is used at any given time. I strongly suggest that you look at different design options to help organize your code.

score 0 · Answer 6 · answered Jul 26 '13 at 19:35

The existence check statement has no effect - as mentioned already. Perhaps your intention was to do something like this:

if(!CheckExistance(Path.c_str())) {
    fstream Text_File;

    Text_File.open(Path, fstream::in | fstream::out | fstream::app);
    Text_File<<Line<<"\n";
    Text_File.close();
}

The fstream within the scope of if statement will hide the one you must have in the outer scope. Also, close is optional - stream will be closed when it goes out of scope.

How can I switch between fstream files without closing them (Simultaneous output files) - C++

6 Answers6

Linked