How do I tokenise a string in C++ into substrings

Question

I have:

string content;

which is inputed by the user.

For example:

content = "I am new to this site".

I want to loop through this string and for each word in the sentence I want to run a function which processes the word.

something like this:

for(int i = 0; i < stringSize; i++){
//remove a word from the string
process(word);
}

Any help would be much appreciated. Thank you

also: [How to split a string in C++](http://stackoverflow.com/questions/236129/how-to-split-a-string-in-c?rq=1) — MatthiasB, May 09 '14 at 07:19

score 1 · Accepted Answer · answered May 09 '14 at 07:26

To split a std::string on whitespace (space, tab, newline etc.) when you can use the following:

std::string content = "I am new to this site";
std::vector<std::string> words;  // Will contain each word of the `content` string

std::istringstream iss(content);
std::copy(std::istream_iterator<std::string>(iss),
          std::istream_iterator<std::string>(),
          std::back_inserter(words));

for (const std::string& w : words)
    std::cout << w << '\n';

The above code snippet will print out each word in content, one word on each line.

References:

Thank you, I believe this is what I need. – fadim May 09 '14 at 07:40 — fadim, May 09 '14 at 07:40

score 0 · Answer 2 · answered May 09 '14 at 07:24

Here's a function that takes a string and returns a list of tokens.

std::vector<std::string> tokenize(std::string const& content)
{
   std::vector<std::string> tokens;
   size_t iter = 0;
   size_t next = 0;
   while ( ( next = content.find(" ", iter)) != std::string::npos )
   {
      tokens.push_back(content.substr(iter, next-iter));
      iter = ++next;
   }
   tokens.push_back(content.substr(iter));
   return tokens;
}

Here's the main function I used to test it.

int main()
{
   std::string content = "I am new to this site";
   std::vector<std::string> tokens = tokenize(content);
   std::vector<std::string>::iterator iter = tokens.begin();
   for ( ; iter != tokens.end(); ++iter )
   {
      std::cout << *iter << std::endl;
   }
}

Here's the output:

I
am
new
to
this
site

crogg01 · Answer 3 · 2014-05-09T17:02:14.403

Here is a boost solution just for the fun of it. It will remove the string you don't want and treat successive delimiters as one. if you do not have C++11, replace the lambda with a pointer to your own function :

#include <boost/algorithm/string/find_iterator.hpp>
#include <boost/algorithm/string/split.hpp>
#include <boost/algorithm/string/classification.hpp>
#include <boost/algorithm/string.hpp>
using namespace std;

int main(int argc, char** argv) 
{
    const string delim{' '};
    vector<string> splitVec;
    string inStr = "keep all this but remove THAT if you see it";
    boost::split( splitVec, inStr, boost::algorithm::is_any_of(&delim[0]), boost::token_compress_on );
    copy_if(splitVec.begin(),splitVec.end(), ostream_iterator<string>(cout," "), [](const string& argStr) -> bool { return(argStr != "THAT"); } );
}

Gives:

keep all this but remove if you see it

How do I tokenise a string in C++ into substrings

3 Answers3