42

I'm developing an API that can accept the main three data types as values to query string parameters: boolean, string & numeric (integer or float). By default, everything is retrieved a string (I don't think there's a way around that), but the parameters are configured with a type so that I convert them to the proper type.

I now need to start accepting NULL as a value, for all three data types, but I'm not sure what is the best way to do so.

Some people seem to accept no value as NULL (?param, without =), but that is no quite working for me:

  1. I'm already accepting that as true for booleans, although that could be changed.
  2. It is received as an empty string, which is a valid value for the string data type.

So, here are the only two options I can see, none of which are working for me:

  1. Use a constant, like "NULL" or "~". That works for booleans & numbers, but not for strings, as that might be the desired string value.
  2. Use an empty value. Once again, that can work for booleans & numbers, but not for strings, as that might be the desired string value.

Seems like using a constant might be my only choice, and to avoid issues that constant can be forbidden from strings. But is that really the only viable option? I wish there were a better way. :(

Edit: I just had another idea that might work, but I'm not sure how it would be perceived. I think it definitely breaks standards, and a lot of people would most likely not like it, but would fix my issue better than any other solution so far. Basically, the idea would be to use JSON, which supports native data types. I'm not having this issue with POST requests because they're using JSON.

So the idea would be to take JSON in the query string, basically allowing an alternative query string that would override the standard HTTP one. Something like: ?_json={"foo":"bar","param":null} instead of ?foo=bar&param=~. Or maybe even take over the query string entirely: ?{"foo":"bar","param":null}.

Marco Roy
  • 4,004
  • 7
  • 34
  • 50
  • 3
    Can't you just send a URLEncoded NUL value? (%00)? Since it's a query parameter, it's in the URL and should be decoded. – Kylar Jul 06 '15 at 19:58
  • 2
    Kylar's suggestion seems to be the accepted method. See here for reference: http://www.w3schools.com/tags/ref_urlencode.asp – John Chrysostom Jul 06 '15 at 19:59
  • @Kylar: Wow, thanks, I never thought of that & was not aware that it was the accepted method. Mind posting it as an answer? I'll accept it as the correct solution. – Marco Roy Jul 06 '15 at 20:32

4 Answers4

33

Send a URLEncoded NULL value (%00) for any thing that you're specifically setting to null. It should be correctly URL Decoded.

Naman
  • 27,789
  • 26
  • 218
  • 353
Kylar
  • 8,876
  • 8
  • 41
  • 75
  • 1
    In PHP, it's not being decoded properly. Instead, it gets the character `�`. But it's still possible to catch it by comparing it with `"\x00"`. So I had to add a loop that checks each query parameter and replaces `"\x00"` with `null`. – Marco Roy Jul 07 '15 at 00:23
  • 1
    Actually, here's a better implementation than a manual loop: `array_walk_recursive($queryParameters, function (&$value, $key) { if ($value === "\x00") { $value = null; } });` – Marco Roy Jul 07 '15 at 05:50
  • 2
    � is the default 'Unknown/Unmappable' unicode character. You always need to look at the bytes. ;) – Kylar Jul 07 '15 at 15:06
  • 1
    Oh yeah, don't worry, I knew about that. :P Just wanted to provide the information in case it could be helpful for someone else. – Marco Roy Jul 07 '15 at 22:09
  • 1
    `%A0` worked for me. %00 didn't work. like: http://localhost/GetUsers/%A0/filter_value – Alexander Trofimov Apr 25 '20 at 05:59
  • %A0 is *not* the same as a NUL value, it's a non-breaking space (NBSP). – Kylar Apr 27 '20 at 13:00
  • Im confused. `The ASCII control characters %00-%1F were originally designed to control hardware devices.` and `Control characters have nothing to do inside a URL.` https://www.w3schools.com/TAGS/ref_urlencode.asp So how am i supposed to send a NULL via GET ? Should i post a new question? – cottton May 08 '20 at 12:54
  • If you're referring to https://www.w3schools.com/tags/ref_urlencode.asp - where it looks like you copied that info from - you'll find that *immediately* after it, it specifies that %00 is the encoding for NUL. Whether or not they are *supposed* to be in URLs, it's widely accepted and used. – Kylar May 08 '20 at 13:01
  • 1
    @Kylar But how can we accept and use this if its not recommended. Looked into https://tools.ietf.org/html/rfc3986#section-7.3 and found `... the "%00" ... (NUL) ... should be rejected if the application is not expecting to receive raw data`. I read about endpoints rejecting `%00` (by default config). So this would be an internal solution only, which may cause security problems (null byte attacks), even if the further sanitizing could fix it. I really cannot believe that there is no "real" accepted solution for this. Any idea how to dig deeper? – cottton May 08 '20 at 13:36
9

or just don't send any value. leave query param value empty in your get request url.. like -

API URL/?foo=&some_other_param=value

in the API foo will be received as null

JON
  • 965
  • 2
  • 10
  • 28
enthusiast
  • 961
  • 7
  • 10
  • 22
    This is wrong. `foo` will be received as an empty string and cause issues with other data types, as I've outlined in the question. – Marco Roy Jul 09 '18 at 22:17
  • 2
    no ..depends on your configurations and how you handle in API .. try with ASP.Net core default behavior or WebAPI .. – enthusiast Jul 13 '18 at 01:43
  • 1
    I don't use .Net, and the answer shouldn't be specific to .Net either. I was looking for a language & framework-agnostic answer; some kind of Web Standard. – Marco Roy Jul 24 '18 at 22:05
  • 1
    What if we dont send the key at all.. will it be NULL then ? – Rahul Jangra Jan 10 '19 at 12:36
  • hi @RahulJangra , apologies for delayed response. If you are expecting a string in your get method as querystring parameter and you don't pass it at all then you will get null in your API. – enthusiast Jan 21 '19 at 09:12
  • I'd argue that usually null should be represented by simply not sending a value, i.e. ```?some_other_param=value``` then e.g. foo should be read as null. If you really want to do it explicitly, I like the `?foo&some_other_param=value` and if that doesn't work, then `?foo=&some_other_params` seems like a fine compromise. =p. – tugend Nov 14 '19 at 12:35
  • 1
    @tugend agreed.. null should be/can be represented by not sending the param. It just depends on how you build your API. In case you read the values by index or order then of-course you have to pass an explicit parameter. But again there also the preferred approach should be reading values by parameter name at API :) – enthusiast Nov 15 '19 at 17:47
4

In your use case that is more clear after the edit I would go surely for the ?_json= parameter, there is nothing wrong with it and since everything works for POST as you write there is no doubt that is better than inventing a generic binding protocol through request parameters (application/x-www-form-urlencoded). I personally have been using two parameters to support also other content types like XML, one $rqc carries the content like your _json and one $rqct carries the content-type, eg:application/json

Said that, I think JSON works well because it allows 4 possibilities for a string property:

  1. filled {"param":"foo"}
  2. empty {"param":""}
  3. absent {}
  4. null {"param":null}

A request parameter instead can have only 3:

  1. filled ?param=foo
  2. empty ?param= or ?param (Server API usually does not differentiate the 2 cases)
  3. absent ?

The 4th possibility, an explicit null, can be handy for partial updates: if one updates only the fields present in the request JSON object; you cannot easily express that with a request parameter.

However if you don't do partial updates you may consider an absent parameter as a null value, but you need to know all parameters allowed for that endpoint, that is not always possible without a schema or other specification (a class to bind to); since you say that you associate parameters to types you probably have it.

The fact that ASP.NET (I learn it now from the answers) does not differentiate between empty and absent parameter reminds me that is often dangerous to allow an empty string and/or treat it differently from an absent/null string, for example because for Oracle an empty string is converted to null, so is not valid for a NOTNULL column (in mysql is ok), see null vs empty string in Oracle.

So if you don't need to support the difference between empty and null/absent string you may also consider the empty parameter as null.

If instead you want to recreate point 4. of JSON {"param":null} with request parameters you must use a convention, like passing %00 as suggested in the answers or using the null marker parameter I propose later.

If the purpose is to create something generic I wouldn't use %00 because some servers may have some parameters sanitization mechanism enabled: if it refuses %00 (as it probably should) then the convention can not work. I checked now and JSOUP removes all in the range %00-%0F. Maybe also ASP.net does by default, see Special character causes "A potentially dangerous Request.Form value"

A more HTML-ish approach could be using an extra null marker parameter whose name is obtained by the original parameter with a suffix, eg: from param it could be param$null: if the parameter is present then the bound value is null, whatever the value of param, it is like putting a unset checkbox before a text input and un-setting the bound value when is checked. Furthermore if you have some way to annotate a parameter/property to support this syntax you might limit the check to those parameters (probably less than 1% of total) and for all the others treat empty parameter and absent parameter in the same way: binding it to a null value.

Testo Testini
  • 2,200
  • 18
  • 29
  • `%00` Should only be considered `null` if it's by itself. If it's part of a larger string, that string should be decoded normally. That way, there's no risk of rejection or truncation. – Marco Roy Dec 21 '20 at 22:22
  • 1
    Yes if you control the environment, I expanded the answer to clarify what I mean. – Testo Testini Dec 22 '20 at 19:22
-9

With IIS and C#, I send it this way: 

?param=#
4b0
  • 21,981
  • 30
  • 95
  • 142
Ariel
  • 1
  • 2