Stack Overflow
Stack Overflow

Questions tagged [boost-tokenizer]

The Boost.Tokenizer C++ library provides a flexible and easy to use way to break a string or other character sequence into a series of tokens.

The Boost.Tokenizer C++ library provides a flexible and easy to use way to break a string or other character sequence into a series of tokens.

37 questions
10
votes
2 answers

Boost::tokenizer comma separated (c++)

Should be an easy one for you guys..... I'm playing around with tokenizers using Boost and I want create a token that is comma separated. here is my code: string s = "this is, , , a test"; boost::char_delimiters_separator…
Lexicon
  • 2,467
  • 7
  • 33
  • 41
4
votes
1 answer

Removing duplicates from Boost::Tokenizer?

I am trying to split a comma-separated string and then perform some action on each token, but ignoring duplicates, so sth. along the following lines: int main(int, char**) { string text = "token, test string"; char_separator sep(",…
tt293
  • 500
  • 4
  • 14
3
votes
1 answer

Using BOOST Tokenizer to display delimiter and to not tokenize a string in quotes

I am using BOOST Tokenizer to break a string into toekn. Basically the tokens will be used to create a compiler for VSL based on c/c++. What i wanted to ask that is it possible that the delimiter defined created using char_separator sep("; <<…
Hassan Jalil
  • 1,114
  • 4
  • 14
  • 34
2
votes
1 answer

Boost.Tokenizer for quotation marks and parentheses

I'd like to split a string into tokens using Boost.Tokenize. It is required that text in quotes or parentheses is a single whole token. More specifically, I need split a line like "one (two),three" four (five "six".seven ) eight(nine, ten) into…
Loom
  • 9,768
  • 22
  • 60
  • 112
2
votes
2 answers

boost lexical cast check

This should be an easy one. I have a function that traverses a csv and tokenizes based on commas and does things with the tokens. One of these things is convert it into an int. Unfortunately, the first token may not always be an int, so when it is…
Lexicon
  • 2,467
  • 7
  • 33
  • 41
2
votes
1 answer

tokenizing string with boost fails when casting tokens to char* const*

I'm using boost::tokenizer to tokenize a string in C++, then I want to pass it to execv. Consider the following code snippet (compilable): #include #include #include #include // I will put every…
Daniel
  • 2,318
  • 2
  • 22
  • 53
2
votes
1 answer

Tokenize a "Braced Initializer List"-Style String in C++ (With Boost?)

I have a string (nested strings even) that are formatted like a C++ braced initializer list. I want to tokenize them one level at a time into a vector of strings. So when I input "{one, two, three}" to the function should output a three element…
DiB
  • 554
  • 5
  • 19
2
votes
2 answers

Boost Tokenizer: Extra Space?

I am using Boost Tokenizer to remove formatted coordinates, eg (x,y). However, it is adding an additional space after the removal. There are no spaces, but I can't figure out how to get rid of this. while (std::getline(input, line)) { …
badfilms
  • 4,317
  • 1
  • 18
  • 31
2
votes
1 answer

boost::tokenizer to consider absence of tokens between separators

I am using boost::tokenizer to get ';' separated fields from a string. I am able to retrieve the fields as shown in the code below but i have 2 questions: Is there any function which tokenizer provides to know the count of tokens in a string based…
anurag86
  • 1,635
  • 1
  • 16
  • 31
2
votes
2 answers

Vector of comma separated token to const char**

I am trying to convert a comma separated string to vector of const char*. With the following code, by expected output is ABC_ DEF HIJ but I get HIJ DEF HIJ Where am I going wrong? Code: #include #include #include…
armundle
  • 1,149
  • 2
  • 15
  • 28
2
votes
1 answer

Erroneous tokenizing

I have this code: #include typedef boost::tokenizer > tokenizer; int main() { using namespace std; boost::char_separator sep(","); string s1 = "hello, world"; tokenizer…
r.v
  • 4,697
  • 6
  • 35
  • 57
2
votes
2 answers

Use boost::tokenizer with boost::iterator_range

I'm using boost::tokenizer to read a CSV-like file. I'm storing the the tokens in a std::vector. It works well, but I want to store only a boost::iterator for each token. I tried: #include #include #include…
Baptiste Wicht
  • 7,472
  • 7
  • 45
  • 110
1
vote
1 answer

How can I tokenize CSV file with TokenizeBoost library?

I have a problem with converting any string from CSV into string (but not string of char) and then tokenize it. There is my code here: #include #include #include "NumCpp.hpp" #include #include…
SerTitan
  • 11
  • 2
1
vote
1 answer

Prevent escaped_list_separator from consuming quotes in quoted token

Is it possible to prevent boost's escaped_list_separator from consuming quotes in a quoted token? Or are there any other ready-to-use constructs to archive this behavior? The inner quotes cannot be escaped as the grammar doesn't support that and is…
sigy
  • 2,408
  • 1
  • 24
  • 55
1
vote
1 answer

Splitting string with multiple delimiters, allowing quoted values

The docs for boost::escaped_list_separator provide the following explanation for the second parameter c: Any character in the string c, is considered to be a separator. So, I need to split the string with multiple separators, allowing the quoted…
Oleg Shirokikh
  • 3,447
  • 4
  • 33
  • 61
1
2 3