How do I make a token parser in C?

Question

I have seen "config" files(yes, text files) for various console applications that look like the following

<token> <value>

How would I go about parsing such a thing in C, where < value > could be a string, a letter or even an integer/float/double?

I read this question " How do I parse a token from a string in C? " where it is mostly recommended to use strtok, but also that it's not thread-safe, and I am planning on spawning multiple threads(provided I am able to finish my application)

P.S Here is an example of a token and a value

user username123
pass 123456

Ah, I forgot the tricky part. I must also be able to parse a token, which has multiple values either separated by a comma.

On the very thread you linked to there is a comment that says "You can use strtok_r() which is thread safe. You can see them both in the same manpage." — dee-see, Mar 28 '12 at 23:26
Define what is `` and `` and it will be easy to provide concrete answers. — Khaled Alshaya, Mar 28 '12 at 23:27
@Arak I have even provided an example. [at]Vache, I must have missed that, I will take a look, in the meantime I am waiting for more suggestions :) — farmdve, Mar 28 '12 at 23:28
Buy the Dragon Book and find out :) http://www.amazon.com/Compilers-Principles-Techniques-Alfred-Aho/dp/0201100886 — paulsm4, Mar 28 '12 at 23:29
Could possibly look into Lex and Yacc (http://en.wikipedia.org/wiki/Lex_(software)) — Ed Heal, Mar 28 '12 at 23:39

score 3 · Accepted Answer · answered Mar 29 '12 at 00:04

I think that fgets() and sscanf() are your friend:

int parseTokens(FILE *filePtr, char **tokens, char **values)
{
    int i = 0;

    char line[128];

    while (fgets(line, 127, filePtr)) {
        tokens[i] = malloc(64);
        values[i] = malloc(64);

        sscanf(line, "%s %s", tokens[i], values[i]);

        i++;
    }

    return i;
}

int main(void)
{
    char *tokens[20];
    char *values[20];

    FILE *filePtr = fopen("~/test.txt", "r");

    if (!filePtr)
    {
        fprintf(stderr, "Error opening file: %s", strerror(errno));
    }

    int count = parseTokens(filePtr, tokens, values);

    for (int i = 0; i < count; i++) {
        printf("%s %s\n", tokens[i], values[i]);

        free(tokens[i]);
        free(values[i]);
    }

    fclose(filePtr);
}

score 2 · Answer 2 · edited May 23 '17 at 11:59

Using getc(), read characters from an input stream into a per-line buffer. Once you hit a token delimiter, you strncpy() or strdup() the per-line buffer into a token char*. If needed, parse the token again on a within-token delimiter (such as a comma), grabbing a character at a time and storing it in a per-token buffer, until you hit a within-token delimiter. Once you hit a line delimiter, copy the per-line buffer into a value char*. If you know the value is an int, float, etc., use C functions for converting a char* into those primitives (e.g., strtol(), etc.). If you have multiple token-value pairs, either keep an array of or pointers to token and value char* variables. Repeat until EOF (end-of-file).

I would suggest `strtol` over `atoi` as you cannot judge if it failed or if the number was 0 using `atoi`. — Ed S., Mar 28 '12 at 23:50

score 1 · Answer 3 · answered Mar 29 '12 at 01:26

Try this:

    FILE* fp;
    fp = fopen("in.txt","r");

    if(fp == NULL)
    {
        printf("Can't open/read file.\n");
        exit(1);
    }

    char* buf = NULL;
    char* key = malloc(64);
    char* val = malloc(64);
    size_t read;
    size_t len = 0;

    if(key == NULL || val == NULL)
    {
        printf("malloc failed.\n");
        exit(1);
    }


    while((read = getline(&buf, &len, fp)) != -1)
    {
        sscanf(buf,"%s %s", key, val);
        printf("<%s> <%s>\n", key, val);
    }

    if(buf != NULL)
    {
      free(buf);
    }

    free(key);
    free(val);

    fclose(fp);

in.txt file:

key value
key1 value1

C application output:

<key> <value>
<key1> <value1>

I hope this help you.

score 0 · Answer 4 · answered Mar 29 '12 at 03:19

0

how about use regexp? if you are in linux, you can just #include <regexp.h> to use it. andman regexp.hwill get how to use it. store them by string. and, it they are number, use sprintf to trans them to unmber.

answered Mar 29 '12 at 03:19

madper

806
1
10
25

How do I make a token parser in C?

4 Answers4