Split string into tokens - without OS-specific functions

Question

Possible Duplicate:
How do I tokenize a string in C++?

strtok function is not thread safe. Microsoft has Windows-specific strtok_s and CString::Tokenize safe functions. Is there cross-platform CRT/C++ library way to do this without manual coding?

http://stackoverflow.com/questions/53849/how-do-i-tokenize-a-string-in-c — Andriy, Aug 22 '12 at 06:30
Are you asking for portable, *thread-safe* tokenization? If so then maybe you can mention that in the title. — juanchopanza, Aug 22 '12 at 06:34

score 5 · Accepted Answer · answered Aug 22 '12 at 06:34

boost::split. http://www.boost.org/doc/libs/1_51_0/doc/html/string_algo/reference.html#header.boost.algorithm.string.split_hpp

example of usage

#include <vector>
#include <string>
#include <boost/algorithm/string.hpp>

int main()
{
   const std::string s = "hello and what";
   std::vector<std::string> v;
   boost::split(v, s, [](char c) { return c == ' '; }, boost::token_compress_on);
   for (const auto& str : v)
   {
      std::cout << str << std::endl;
   }
}

http://liveworkspace.org/code/3dfc9ee9c8497741f9976ac41a14a390

Or use boost::tokenizer

score 2 · Answer 2 · answered Aug 22 '12 at 06:34

2

The thread safe version of strtok() is strtok_r(). And it is POSIX compliant too.

answered Aug 22 '12 at 06:34

Pavan Manjunath

27,404
12
99
125

score 2 · Answer 3 · answered Aug 22 '12 at 06:46

Actually, strtok() is generally thread safe (virtually any runtime supporting a multithreaded OS will have a thread-safe strtok()). strtok() can't be used to tokenize different strings in an 'alternating' fashion, but that's a pretty rare occurrence and is under control of your code.

However, that said, strtok_r() is a common (though not part of the C standard - it's POSIX) variant that lets you maintain control over the context so you can juggle as many strtok_r() contexts at the same time as you like. It's not available with MSVC, but you can find a public domain implementation here: http://snipplr.com/view/16918/

Moreover strtok_r (on POSIX side) and strtok_s (on Visual Studio side) have exactly the same prototype and behaviour. With a simple macro that maps strtok_r to strtok_s on Windows targets you should have a portable solution. — greydet, Aug 22 '12 at 07:12

Split string into tokens - without OS-specific functions

3 Answers3