1

I still have some problems and I think if I manage to figure this one out that I will finally get the grip on it.

I have this line that I strtok it by the delimiter space. Now I want to store all tokens in a pointer on the array char* tokens[50]. How would I store all the tokens in this pointer and how would I access all tokens once they are stored. I think I'd also need a counter int token_count.

Robert
  • 10,403
  • 14
  • 67
  • 117
jabk
  • 1,388
  • 4
  • 25
  • 43

2 Answers2

4

This is straight-forward. For example:

char * tokens[50];
size_t n = 0;

for (char * p = strtok(line, " "); p; p = strtok(NULL, " "))
{
    if (n >= 50)
    {
        // maximum number of storable tokens exceeded
        break;
    }
    tokens[n++] = p;
}

for (size_t i = 0; i != n; ++i)
{
     printf("Token %zu is '%s'.\n", i, tokens[i]);
}

Note that line must point to a mutable character string, since strtok mangles the string.

Kerrek SB
  • 464,522
  • 92
  • 875
  • 1,084
  • In C++, you would have `std::vector tokens` and `tokens.push_back(p)`, without any size limitations. – Kerrek SB May 31 '14 at 13:08
  • Much obliged, but I don't really get why do we increment n in `tokens[n++] = p` – jabk May 31 '14 at 13:11
  • @user2202368: `n` keeps track of the next slot in `tokens` that's to be filled. – Kerrek SB May 31 '14 at 13:17
  • so is n a counter for chars of a token or for tokens themselves – jabk May 31 '14 at 13:20
  • 1
    @user2202368: It's a counter for the tokens themselves - you see how it's used as the array index on the `tokens` array. The final value of `n` is the number of array elements that have been assigned. – Kerrek SB May 31 '14 at 13:21
  • Is `char *tokens` overrun and can be used further more without any worries? – jabk May 31 '14 at 13:34
  • @user2202368: The `if` check inside the loop makes sure that you don't exceed the capacity of the array. Also, it's `char * tokens[50]`, not `char * tokens` -- you have an array of 50 pointers. – Kerrek SB May 31 '14 at 13:35
  • Sorry, typo. So if I set size_t n = 0 every time I call function tokenize, there will be no problems? – jabk May 31 '14 at 18:43
  • strangely enough, when I use `tokens[i++]` and print out the token I just stored, I get `(null)`, but when I increment `i` after storing it stores the token properly. – Ungeheuer Apr 07 '17 at 05:31
  • Hold on, doesn't this tie `tokens[i]` to the same memory address as where the token from `strtok` came from? – Ungeheuer Apr 07 '17 at 07:14
-2

Why don't you use regular expressions in C++11? You can represent a space (one or more) as a simple regular expression "\s+" and use a regex token iterator to iterate through the tokens, you can store the tokens in a vector from there.. Here is an example that just prints out the tokens. 

#include <regex>
#include <string>
#include <iostream>
using namespace std;


int main()
{

    regex re("\\s+");
    string s = "Token1 token2   token3"; //example string

    sregex_token_iterator reg_end;
    for(sregex_token_iterator it(s.begin(), s.end(), re, -1); it != reg_end; ++it) {
         cout << it->str() << endl;
    }
}
vladimirm
  • 261
  • 1
  • 2
  • 8
  • 3
    I'm using C, never used C++ :) – jabk May 31 '14 at 13:15
  • 1
    Using regular expressions for a simple substring search is an outright crime. Think about the high-level algorithm first before throwing arbitrarily large cannons at the problem. – Kerrek SB May 31 '14 at 13:18