Issues with Pointer Arithmetic - Trying to tokenize input String

Question

Currently I am working on a program that allows a user to enter a string that is then tokenized, then the tokens are printed to the screen by using an array of pointers. It is "supposed" to do this by calling my tokenize function which reads the input string until the first separator ( ' ', ',', '.', '?', '!'). It then changes that separator in my string to a NULL char. It then should return a pointer to the next character in my string. In main after the string has been input, it should keep calling the tokenize function which returns pointers which are then stored in a array of pointers to later print my tokens. Once the tokenize() returns a pointer to a NULL character which is at the end of my string it breaks from that loop. Then I print the tokens out using my array of pointers. //trying to be detailed

#include <stdio.h>
#include <string.h>

char *tokenize ( char *text, const char *separators );

int main ( void )
{
        char text[30];
        char separators[6] = { ' ','.',',','?','!','\0'};
        char *pch = NULL;
        int tokens[15];
        int i = 0;
        int j = 0;

        printf("Enter a string: \n");
        fgets( text, 30, stdin );

        printf("%s", text );

        pch = tokenize ( text, separators );

        do
        {
                pch = tokenize ( pch, separators );
                //printf("%c", *pch);
                tokens[i] = pch;
                i++;
        }
        while( *pch != NULL );


        i--;
        while( j != i )
        {
                printf("%s", tokens[i] );
                j++;
        }

        return 0;
}

char *tokenize ( char *text, const char *separators )
{
        while( text != NULL )
        {

                if( text != NULL )
                {
                        while( separators != NULL )
                        {
                                if( text == separators )
                                {
                                        text = '\0';
                                }
                                separators++;
                        }
                }
                text++;
        }
        return text;


}

3 big known problems currently. 1.When I compile, it reads the string then prints it, then gets stuck in a endless loop with nothing printing, still trying to get input. 2. Im pretty sure I am using the " * " for my pointers in the wrong place. 3. My function passes in a reference to my arrays, so I assumed i could just increment them as is.

I appreciate any feedback! I will be watching this post constantly. If i left something unclear, I can respecify. Thanks.

Answer 1

Maybe you will want to take a look at strsep and this post Split string with delimiters in C If you need more reference points try searching "split string" it is what you want to do if I understood correctly.

Answer 2

You had right idea for approaching the problem, but you had numerous pointer/int errors throughout your code. Make sure you compile your code with Warnings enabled, this will tell you where you have problems in your code. (don't expect your code to run correctly until you address and eliminate all warnings). At a minimum, compile with -Wall -Wextra options in your build command.

There are a lot easier ways to do this, but for the learning experience, this is a great exercise. Below is your code with the errors corrected. Where possible, I have left your original code commented so you can see where the issues were. I also include a bit of code to remove the newline included by fgets at the end of text. While this isn't required, it is good practice not to have stray newlines filter through your code.

Let me know if you have questions:

#include <stdio.h>
#include <string.h>

char *tokenize ( char *text, const char *separators );

int main ( void )
{
        char text[30];
        char separators[6] = { ' ','.',',','?','!','\0'};
        char *pch = NULL;
        char *tokens[15] = {0};             /* declare array of pointers    */
        int i = 0;
        int j = 0;

        printf("Enter a string: \n");
        fgets( text, 30, stdin );
        size_t len = strlen (text);

        if (text[len-1] == '\n')            /* strip newline from text      */
            text[--len] = 0;

        pch = text;                         /* pch pointer to next string   */
        char *str = text;                   /* str pointer to current       */

        do
        {
                pch = tokenize ( str, separators ); /* pch points to next   */
                tokens[i++] = str;                  /* save ptr to token    */
                str = pch;                          /* new start of str     */
        }
        while (pch != NULL && *pch != 0);           /* test both pch & *pch */

        printf ("\nTokens collected:\n\n");
        while (tokens[j])                   /* print each token             */
        {
                printf("  token[%d]: %s\n", j, tokens[j] );
                j++;
        }
        printf ("\n");

        return 0;
}

char *tokenize ( char *text, const char *separators )
{
        const char *s = separators;     /* must use pointer to allow reset  */

        //while( text != NULL )
        while( *text != '\0' )
        {
                s = separators;         /* reset s                          */
                while( *s != 0 )        /* 0 is the same as '\0'            */
                {
                        //if( text == separators )
                        if( *text == *s )
                        {
                                //text = '\0';
                                *text = '\0';
                                return ++text;
                        }
                        s++;
                }
                text++;
        }
        return text;
}

Example output:

$ ./bin/tokenizestr
Enter a string:
This is a test string

Tokens collected:

  token[0]: This
  token[1]: is
  token[2]: a
  token[3]: test
  token[4]: string