7

I want to use string Tokenizer for CPP string but all I could find was for Char*. Is there anything similar for CPP string?

djot
  • 2,952
  • 4
  • 19
  • 28
Scarlet
  • 271
  • 2
  • 12
  • 17
    How about some of the examples from the following: http://www.codeproject.com/KB/recipes/Tokenizer.aspx They are very efficient and somewhat elegant. The String Toolkit Library makes complex string processing in C++ simple and easy. –  Dec 08 '10 at 05:31
  • possible duplicate of [Using strtok with a std::string](http://stackoverflow.com/questions/289347/using-strtok-with-a-stdstring) – bobobobo Mar 05 '13 at 19:00

5 Answers5

7

What do you mean by "token"? If it's something separated by any whitespace, the string streams is what you want: 

std::istringstream iss("blah wrxgl bxrcy") 
for(;;) {
  std::string token;
  if(!(iss>>token)) break;
  process(token);
}
if(!iss.eof()) report_error();

Alternatively, if your looking for a a certain single separating character, you can replace iss>>token with std::getline(iss,token,sep_char).

If it's more than one character that can act as a separator (and if it's not whitespaces), a combinations of std::string::find_first() and std::string::substr() should do.

sbi
  • 219,715
  • 46
  • 258
  • 445
  • Is there any good reason for preferring `for(;;)` over `while(iss>>token)` (assuming `token` was declared before the loop)? In this example it would be one line shorter, and, in my opinion, at least not less readable. – Björn Pollex Aug 26 '10 at 10:55
  • @Space_C0wb0y: (I only now understood your comment. Well, at least I think I do...) I just happen to prefer a more local variable over the terseness of the `while` loop. – sbi Sep 03 '10 at 11:29
  • 1
    @sbi You could use a for-loop like this: `for (string token; iss >> token; ){...}` to get rid of the if-statement and still have a localized variable. – erlc Oct 07 '13 at 11:42
  • @erlc That seems indeed better than my solution. Good idea! (I would amend my answer, but that's a pain to do on the phone. Feel free to do it yourself, if you want.) – sbi Oct 08 '13 at 09:05
4

You can do as said by chubsdad or use boost tokenizer : http://www.boost.org/doc/libs/1_44_0/libs/tokenizer/tokenizer.htm

Doing it by yourself is not so complicated if you're affraid by Boost.

Guillaume Lebourgeois
  • 3,796
  • 1
  • 20
  • 23
1

You should take a look at Boost Tokenizer

reko_t
  • 55,302
  • 10
  • 87
  • 77
0

Check out STL algos like find_first_of, find_first_not_of and so on to create a custom one.

Chubsdad
  • 24,777
  • 4
  • 73
  • 129
0

Try this snippet I found somewhere (maybe even around here?):

#include <string>
#include <vector>
#include <sstream>

std::vector<std::string> &split(const std::string &s, char delim, std::vector<std::string> &elems) {
    std::stringstream ss(s);
    std::string item;
    while(std::getline(ss, item, delim)) {
        elems.push_back(item);
    }
    return elems;
}


std::vector<std::string> split(const std::string &s, char delim) {
    std::vector<std::string> elems;
    return split(s, delim, elems);
}
Gunnar
  • 213
  • 3
  • 13