strtok seems to delete string portion - Am I using it right?

Question

I want to use strtok() to divide a line into tokens, but the function seems to just make the first token and "destroying" the rest. I used printf() to check where did that happen and found out it was just after token= strtok(line,delimit). Here's my main, read_line and parse_args functions:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define PROMPT "$"
#define MAX_LINE 512
void main(){
    char line[MAX_LINE]; 
    while(read_line(line)){
        execute_line(line);
    }
}

char *read_line(char *line){ 
    printf("%s ",PROMPT);
    fflush(stdout);
    line=fgets(line,MAX_LINE,stdin);
    return line;
}

int parse_args(char **args, char *line){
    printf("%s", line); //If it reads "Hello how are you?", here it prints properly
    int n=0;
    char* token;
    char delimit[]=" \t\r\n\v\f";
    token=strtok(line,delimit);
    printf("%s", line); //however here it just prints "Hello" 
    while(token!=NULL){
        printf("token%i: %s\n",n,token);
        *args=token;
        n++;
        *args++;
        token=strtok(NULL,delimit);
    }
    printf("token%i: %s\n",n,token);
    *args=token;
    return n;
}

int execute_line(char *line){
    char **args;
    parse_args(args,line);
    check_internal(args);
    return 0;
}

See: http://en.cppreference.com/w/c/string/byte/strtok. Specifically the 6th bullet for #1. — Michael Albers, Nov 19 '16 at 17:37

pat · Accepted Answer · 2016-11-19T17:39:50.823

1

strtok replaces delimiters with NUL characters so that it can return pointers to the tokens still in the original buffer.

The *args++ is misleading. It does increment args, but the * is unnecessary.

edited Nov 19 '16 at 17:39

answered Nov 19 '16 at 17:38

pat

12,587
1
23
52

Ohh yes. The `*` is unnecessary. – pat Nov 19 '16 at 17:40
@KeineLust *`parse_args()` is never called in the OPs code.* Most likely, the OP just failed to post a full MCVE. – Andrew Henle Nov 19 '16 at 17:42
thanks @pat , it was that *, however now it reports a segmentation fault... – Januan Nov 19 '16 at 18:05
Now that I can see `execute_line`, I can see that you aren't allocating any memory for the `args` array. You should declare `args` as `char *args[MAX_ARGS+1]` in `execute_line` (and pick some value for `MAX_ARGS`; the `+1` is for the `NULL` you are storing at the end). Be aware that, with the current code, if there are too many args, it will overflow the array; you should perform a bounds check. – pat Nov 19 '16 at 18:15
Oh, and declaring `delimit` as an array will cause the function to copy the string onto the stack every time you call `parse_args`. It's better to declare `delimit` as `const char *delimit = "..."` so you can use the copy in the constant strings section. – pat Nov 19 '16 at 18:17
Also, when you drop out of the `while` loop, `token` will necessarily be `NULL` (that was the bound on the loop), so it's probably not a good idea to pass `NULL` to `printf` with a `%s` format char (see [here](http://stackoverflow.com/questions/11589342/what-is-the-behavior-of-printing-null-with-printfs-s-specifier)). – pat Nov 19 '16 at 18:19

score 0 · Answer 2 · answered Nov 19 '16 at 17:40

0

Per the standard declaration:

#include <string.h>

char *strtok(char *restrict s, const char *restrict sep);

Note well that char *restrict s contains no const. strtok() modifies the string it's splitting into tokens.

answered Nov 19 '16 at 17:40

Andrew Henle

32,625
3
24
56

score -1 · Answer 3 · answered Nov 19 '16 at 18:00

token=strtok(line,delimit);
printf("%s", line); //however here it just prints "Hello"

that's exactly what strtok does :)

It actually takes your line, and puts a '\0' instead of the next delimiter. You should try looking at your code with a debugger, it will make more sense to use strtok after that.

Good luck!

strtok seems to delete string portion - Am I using it right?

3 Answers3