Best way to parse out incoming HTTP post request on Arduino?

Question

I'm writing a simple HTTP webserver on my Arduino Uno Wifi Rev2 to handle an incoming HTTP POST Request in JSON format.

This is how I'm sending the HTTP request (with JSON) from my client:

curl \
--request POST \
--header "Content-Type: application/json" \
--data '{
    "A": "B",
    "C": "D"    
}' \
"http://192.168.4.1/myEndpoint"

This is the string the Arduino web-server receives:

POST /myEndpoint HTTP/1.1\r\nHost: 192.168.4.1\r\nUser-Agent: curl/7.54.0\r\nAccept: */*\r\nContent-Type: application/json\r\nContent-Length: 34\r\n\r\n{\n    "A": "B",\n    "C": "D"    \n}

I'm using Nick Gammon's Arduino Regexp library to parse this request, validate it and extract the JSON data.

This works, but parsing the HTTP request in this way is extremely brittle and feels hacky. It could easily break if a different client re-orders/omits a header or skips the carriage return characters. This is the god-awful regexp I'm using for validation:

    httpRegexp = "POST /myEndpoint HTTP/[%d%.]+%\r%\nHost: 192%.168%.4%.1%\r%\nUser%-Agent: curl/[%d%.]+%\r%\nAccept: %*/%*%\r%\nContent%-Type: application/json%\r%\nContent%-Length: %d+%\r%\n%\r%\n{%s*\"[A-Za-z]+\"%s*:%s*\".+\"%s*,%s*\"[A-Za-z]+\"%s*:%s*\".+\"%s*}";

Is there a better/recommended way for me to validate and parse the HTTP request? This must be a problem that others have already encountered and solved. Please post a code fragment solving this issue if possible.

Sorry but your "JSON" is in no way a valid JSON (space, line breaks) - I do JSON parsing / validation manually whith char arrays but it works only on valid JSONs. And for clarification what parts of the incoming you really need and are you able to provide valid header data (e.g. no space after ':') Thanks — Codebreaker007, Mar 26 '20 at 08:19
I need to pick out the two data fields. So I need to know that A=B and C=D. The rest is just validation. — Saqib Ali, Mar 26 '20 at 15:30
"The rest is just validation" what do you really need to process, because this defines the complexity of a solution- coming to the question - why not take an existing proven library and adapt the code to your "special" need = Open source instead of reinventing the wheel. Parsing a JSON needs a valid JSON beforehand else you willl fail - I could possibly provide working examples if I know what you intend. — Codebreaker007, Mar 26 '20 at 15:39
@Codebreaker007 I'm asking you what proven library I should use. I need to process the values of B and D. OK, the incoming request isn't valid JSON. I get that. I still need to parse requests that look like this anyway. — Saqib Ali, Mar 26 '20 at 18:25
Most commonly used library for handling json objects is ArduinoJson, specifically to your case, you can read this [JsonParser](https://arduinojson.org/v6/example/parser/) example. — hcheung, Mar 27 '20 at 04:03

score 1 · Answer 1 · answered Mar 26 '20 at 19:34

As a starter:
First send a correct (syntax!) test request

curl \
 request POST \
header "Content-Type:application/json" \
data '{"A":"B","C":"D"}' \
"http://192.168.4.1/myEndpoint"

there are tons of excellent examples if you do a search for :

arduino webserver ethernet library

on your favorite search enginne.
One would be: https://startingelectronics.org/tutorials/arduino/ethernet-shield-web-server-tutorial/
or you use the webserver library from esp8266 and adapt it (not really hard imho)
you would do on the Arduino something like

webServer.on("/myroute/lighton", HTTP_POST, readJson);

char jsonField[64] = '\0'; //This is a predefined buffer for data handilng

the function would look like (partly working code, partly pseudo code )

bool readJson(){
 if (webserver.args() == 0) return false;  // we could do in the caller an error handling on that
 strcpy (jsonField, webserver.arg(1).c_str());  // here we copy the json to a buffer

 /** Get rid of starting and finishing bracket and copy to */
   strncpy(jsonField , jsonField + 1, strlen(jsonField) - 2);
  jsonField[strlen(jsonField) - 2] = '\0';
   uint16_t maxIndex = strlen(jsonField); // number of characters received - without the brackets
     uint16_t index = 0;
  int16_t nextIndex = 0;
  uint8_t  i = 0;
  // In this routine we get the value pairs e.g. "A":"B"
  while ((nextIndex != -1) && (nextIndex < maxIndex)) {
    nextIndex = indexOf(jsonField, ',', index);

    ... the next step would be to process the value pairs by stripping the " and split on the ':' delimiter --
    if you need just the values = content in your example B and D its easy, 
    you could do 
    if (strcmp (firstValofPair ,'A')==0) valueB = atoi(B); // given B is a number and we have to convert from char to int

    .... some more logic and you have a simple reliable JSON Parser for all kind of web server usage

  }
 return true; // success parsing
}

I have implemented this kind of logic in some real live scenarios and all work reliable and stable for some years now. A final tip:
Never use the Arduino String class in Arduino webserver scenarios. The String class fractures your heap and crashes your Arduino. In my example I use fixed chars which are compiled to the stack and keeps your memory happy.

score 0 · Answer 2 · answered Mar 26 '20 at 15:51

The linked Regexp library is based on Lua, which has an interesting pattern match operator

%b() balanced nested pair ( ... ( ... ) ... )

Then to capture a balanced curly brace expression at the end of the HTTP request, use the expression

"(%b{})%s*$"

As pointed out in the comments, this doesn't mean the capture is valid JSON. The next step would be to pass the captured substring to a JSON parser for validation.

Best way to parse out incoming HTTP post request on Arduino?

2 Answers2

Linked