Tokenize string in C++

Asked Dec 05 '15 at 21:42

Active Dec 05 '15 at 21:42

Viewed 30 times

If I have a sentence like

"Hello, I am going for a walk. I will be back home in an hour!"

How do I tokenize the string in C++ so that I get this output:

Hello
,
I
am
going
for
a
walk
.
I
will
be
back
home
in
an
hour
!

The tokenizers I've looked at removes special signs like period, comma etc. But in this case I want to treat these special signs as "words" as shown above.

asked Dec 05 '15 at 21:42

user1884325

2,530
1
30
49

Hint: check isspace: http://www.cplusplus.com/reference/cctype/isspace/ – Alexandre Cartaxo Dec 05 '15 at 21:44
All teokenizers I've ever met accept a "token parameter". Read up some more. – StoryTeller - Unslander Monica Dec 05 '15 at 21:45
That seems a reasonable thing to implement. You need to try coding it yourself. Read one character at a time, output a linefeed when you switch from alpha to non-alpha and to spaces. How are you supposed to treat sequences of special signs such as `...`? – chqrlie Dec 05 '15 at 21:46
@AlexandreCartaxo This isn't a whitespace tokenizer. It's a whitespace and punctuation tokenizer. `std::regex` is a good solution here. – erip Dec 05 '15 at 21:48
Then ispunct (http://www.cplusplus.com/reference/cctype/ispunct/) will solve the issue. – Alexandre Cartaxo Dec 05 '15 at 21:50
2

This isn't "tokenizing". This is "parsing". – Lightness Races in Orbit Dec 05 '15 at 21:54

Tokenize string in C++

0 Answers0