Trying to tokenize a string

Question

Possible Duplicate:
How do I tokenize a string in C++?

I have this text file

q1 t q2

q2 e q3

q3 x q4

q4 t q5 q6 q11

q5 | q6 q11

I want to extract each element that is separated by a space. For example, in line one I want to be able to extract "q1" "t" and "q2" as separate tokens.

I'm thinking there's two ways to go about this

read from the file, token by token, using ifstream>>. The problem I'm having with this approach is that I don't know how to tell when the end of a line has been reached so that I can move onto the next line.
The other approach, is to get the whole line at once with getline(); The problem with this approach is that I have to tokenize the string myself, and each line is different so I'm not sure that's the best idea. I'm pretty blown away there's no built in way to do this, besides strtok() which looks to be not at all what I want. Thanks guys any help is appreciated.

So do you want one giant linear array containing every token in the file? If so, the first one works fine, and can be augmented with `std::vector`'s constructor taking two iterators. — chris, Oct 15 '12 at 09:27
If you're tokens are as nicely separated by whitespace (including newlines) as this, `while (fs >> str)` will totally work for you. — WhozCraig, Oct 15 '12 at 09:29
Either your newlines have meaning. Then go for `getline()` and feed the line through stringstream to get tokens. Or your newlines *don't* have meaning, then reading from `ifstream` will ignore them like any other whitespace. — DevSolar, Oct 15 '12 at 09:29

score 3 · Accepted Answer · answered Oct 15 '12 at 09:30

3

Use getline, and to tokenize the resulting string, put it into a std::stringstream and extract the tokens from that.

std::string line_string;
while ( getline( file, line_string ) ) {
    std::istringstream line( line_string );
    std::string token;
    while ( line >> token ) {
        do something with token
    }
}

answered Oct 15 '12 at 09:30

Potatoswatter

134,909
25
265
421

That's what I would do also... It's a great way to chuck together a simple configuration file when you're in a hurry =) – paddy Oct 15 '12 at 09:31
Awesome, trying that right now, looks easy! – Tyler Pfaff Oct 15 '12 at 09:31

score 0 · Answer 2 · answered Oct 15 '12 at 09:32

If you only want to use standard library, you can go with getline and then use a stringstream to ifstream>> it word by word.

Otherwise C++ is indeed quite limited in terms of text processing and you could do things more easily with Qt's strings and streams, or string algorithms from boost (including split), which integrate well with STL strings.

Trying to tokenize a string

2 Answers2