Efficient String Parsing in C

Question

I'm implementing some C getter/setter for an embedded system which takes a C string as an input.

I need to parse out some commands, ex: command, option, option, option. The options themselves will need to be further parsed. As a simple example

set_speed M1=10 M2=20, set_speed needs to be parsed, then each token M1=10 and M2=20 need to be further parsed.

strtok can not be repeatedly called unfortunately, if it could be the problem would be simple.

The old-school way would be to write a tokenizer (eg: using lex or flex) and maybe a parser (by hand, or using something like yacc), but depending on the specific grammar of your commands it may be easier or harder. — Paul Hankin, Jan 23 '22 at 09:22
I think in my case I can do the following, strtok and extract command, then for each following token do strcmp to see if the first N bytes match the option, then finally send the substring off to atof to get the value. Would like to know if there is a better way. I wish strtok was re-entrant but I guess I could use malloc to save each token and then run it again. — FourierFlux, Jan 23 '22 at 09:32
You don't need to copy the tokens, just store the pointers that `strtok` returns in an array. I'm assuming that there's some reasonable upper limit on the number of options that follow a command. Once you have the array of token pointers, you can use `strtok` on each one to break it down further. That way you don't need `strtok` to be re-entrant. — user3386109, Jan 23 '22 at 09:48
`strtok can not be repeatedly called unfortunately` it can. (?) `, strtok and extract command, then for each following token do strcmp to see if the first N bytes match the option, then finally send the substring off to atof to get the value` exactly do that. Do not think of "better way", just do it any way and be done with it. `I wish strtok was re-entrant` Why do you need re-entrancy? Do you have threads? Will you be calling strtok multiple times _at the same time_? — KamilCuk, Jan 23 '22 at 10:03
The problem _is_ simple. Make a custom function. Search for next space. Check that each character on the way is legit. Convert everything between to integer with `strtol`. Then check character by character that they match til you hit =, then again search for space and convert. Repeat again then done. — Lundin, Jan 24 '22 at 07:39

Shawn · Answer 1 · 2022-01-23T11:15:03.197

Doing it with strtok() to split both words and the options up is possible, but a bit of a pain. There's POSIX strtok_r(), which would work better, but that isn't in standard C (But is easy to write yourself...). In most languages, regular expressions would be a good choice, but again they're not in standard C, just POSIX or third-party libraries like PCRE2. A few other routes come to mind (Like sscanf(), or a parser generator created routine (Maybe ragel or re2c are worth exploring since they compile to C code embedded in a larger source file and don't need a support framework, but I'm not very familiar with using them)), but aren't really efficient or suitable for an embedded environment.

However, it's easy enough to parse strings like this in a single pass with just standard string search functions and a bit of pointer manipulation:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// Note: Destructively modifies its argument
void parse(char *str) {
  static const char *ws = " \t"; // Whitespace characters that separate tokens
  char *command = str;

  // Find the end of the first word
  str += strcspn(str, ws);
  if (*str) {
    *str++ = 0; // Terminate command
  }
  printf("Command: %s\n", command);

  while (*str) {
    // Skip leading whitespace
    str += strspn(str, ws);
    if (!*str) {
      // That was actually trailing whitespace at the end of the string
      break;
    }

    // Split at = sign
    char *option = str;
    str = strchr(str, '=');
    if (!str) {
      fputs("Missing = after option!\n", stderr);
      exit(EXIT_FAILURE);
    }
    *str++ = 0; // Terminate option

    // Parse the numeric argument
    char *valstr = str;
    double val = strtod(valstr, &str);
    if (valstr == str || !strchr(ws, *str)) {
      fprintf(stderr, "Non-numeric argument to %s!\n", option);
      exit(EXIT_FAILURE);
    }

    printf(" Option %s, value %f\n", option, val);
  }
}

int main(void) {
  char command_string[] = "set_speed M1=10 M2=20";
  parse(command_string);
  return 0;
}

Example:

$ gcc -g -O -Wall -Wextra -o demo demo.c
$ ./demo
Command: set_speed
 Option M1, value 10.000000
 Option M2, value 20.000000

score 1 · Answer 2 · answered Jan 23 '22 at 09:18

1

getopt is a standard C function/library for parsing parameters. There's a good example of it here...

https://www.gnu.org/software/libc/manual/html_node/Example-of-Getopt.html

answered Jan 23 '22 at 09:18

Mashed Spud

452
4
12

Don't have that, I only have C and C++ stdlib supported on my device. – FourierFlux Jan 23 '22 at 09:26
There's a SO hit for that. I didn't appreciate that getopt was a GNU function. Take a look at... https://stackoverflow.com/questions/10404448/getopt-h-compiling-linux-c-code-in-windows I think it offers a couple of solutions. – Mashed Spud Jan 23 '22 at 09:31
@FourierFlux `Don't have that, I only have C and C++ stdlib` what compiler and compiler library are you using? – KamilCuk Jan 23 '22 at 10:05
2

How is `getopt(3)` supposed to handle OP's commands? – Shawn Jan 23 '22 at 10:33
This is completely irrelevant both to the question and to embedded systems. – Lundin Jan 24 '22 at 07:33

Efficient String Parsing in C

2 Answers2