Get only one token from strtok at a time

Question

Consider the following scenario with two different strings:

Row1: MyID-MyName-MyAddress-MyNumber-MyNumber2-MyAlias
Row2: MyID-MyName-MyAddress-MyNumber--MyAlias

In the second example, the value for MyNumber2 is missing. I need to extract each attribute using strtok(). So far, this is my code:

MyID      = strtok (str,  "-"); //where 'str' is the full string
MyName    = strtok (NULL, "-");
MyAddress = strtok (NULL, "-");
MyNumber  = strtok (NULL, "-");
MyNumber2 = strtok (NULL, "-");
MyAlias   = strtok (NULL, "-");

The first example works good, and I am able to store each attribute inside the variables. However, in the second example, I am having troubles:

When getting into the variable MyNumber2, strtok() does not return an empty string (as I would like). Instead, it reads through the string until the first character that does not match the delimiter "-", thus ignoring the existence of the empty value in the string.

Is there any possibility to split the string just once per delimiter?

"_the value for `MyNumber` is missing_" -- Did you mean _the value for `MyNumber2` is missing_ — Spikatrix, Jul 17 '15 at 08:54
That's expected behaviour. If you want to process empty tokens, you have to write your own strtok function. Find soume strtok source and modify it to your needs. — Jabberwocky, Jul 17 '15 at 09:06

score 4 · Answer 1 · answered Jul 17 '15 at 09:19

I think you should use standard function strchr instead. For example

#include <stdio.h>
#include <string.h>

int main( void )
{
    char s[] = "MyID-MyName-MyAddress-MyNumber--MyAlias";

    for ( char *p = s, *q = s; p != NULL; p = q )
    {
        q = strchr( p, '-' );
        if ( q )
        {            
            printf( "\"%*.*s\"\n", ( int )(q - p ), ( int )( q - p ), p );
            ++q;
        }
        else
        {
            printf( "\"%s\"\n", p );
        }
    }
}

The program output is

"MyID"
"MyName"
"MyAddress"
"MyNumber"
""
"MyAlias"

Suggest `if (q) { printf("1) \"%*.*s\"\n", ... } else { printf("2) \"%s\"\n", ...` to add clarity to outputs origin — chux - Reinstate Monica, Jul 17 '15 at 18:30

score 0 · Answer 2 · edited May 23 '17 at 12:23

0

Strtok cant tokenize the empty values between the delimiters. To get the reason, Please check the first answer here

I was under the same situation as you. I used strstr to do the same thing as shown below:

#include<stdio.h>
#include<string.h>
#include<stdlib.h>

void main(void)
{
    char str[] = "MyID-MyName-MyAddress-MyNumber--MyAlias";
    char * str1, * token;
    int i = 0;
    int parsed_buffer_length = 0;
    char * previous_delimiter_index = str;
    char * delimiter_index = str;
    for(;;)
    {
            str1 = str+parsed_buffer_length;
            delimiter_index = strstr(str1,"-");
            if(delimiter_index==NULL)
            {
                    printf("%s",str1);  //prints last token
                    break;
            }
            token = malloc(delimiter_index-previous_delimiter_index+1);
            memset(token,'\0',delimiter_index-previous_delimiter_index+1));
            strncpy(token,str1,(delimiter_index-previous_delimiter_index));

            printf("%s\n",token);

            parsed_buffer_length = (int)(parsed_buffer_length+delimiter_index-previous_delimiter_index+1);
            previous_delimiter_index = delimiter_index+1;
            free(token);
    }
}

Which gives the output:

MyID
MyName
MyAddress
MyNumbers

MyAlias

edited May 23 '17 at 12:23

Community

1
1

answered Jul 17 '15 at 09:22

Gajendra Bagali

177
1
2
12

1

`memset(token,'\0',sizeof(token));` is bad. `sizeof(token)` is the sizeof of the pointer, not the size of the memory allocated. Rather than `memset(token,'\0',sizeof(token)); strncpy(...);`, suggest `*token = '\0'; strncat(token,str1,(delimiter_index-previous_delimiter_index)+1);` – chux - Reinstate Monica Jul 17 '15 at 18:36
Yeah that's right. Thanks for the correction. The way you is suggested works totally fine but I changed it in very simpler way: replaced `memset(token,'\0',sizeof(token));` with `memset(token,'\0',sizeof(delimiter_index-previous_delimiter_index+1));` – Gajendra Bagali Jul 18 '15 at 19:26
The `sizeof(delimiter_index-previous_delimiter_index+1)` is also not the size of the buffer that needs zero-ing. Try various input and notice `sizeof(delimiter_index-previous_delimiter_index+1)` is always the same. – chux - Reinstate Monica Jul 18 '15 at 19:31
Oops.. there should not be `sizeof`. It should be only `delimiter_index-previous_delimiter_index+1`. I hope there is no confusion now – Gajendra Bagali Jul 18 '15 at 19:34

Get only one token from strtok at a time

2 Answers2