HTTP GET request with many items in query string

Question

We make an HTTP GET request with a lot of data in the query string, representing all of the ids of a collection to retrieve.

Regarding the limit on the length of the query string in the url, a quick google search says:

RFC 3986 also states there is no limit, but indicates the hostname is limited to 255 characters because of DNS limitations (section 2.3.3). Microsoft states that the maximum length of a URL in Internet Explorer is 2,083 characters, with no more than 2,048 characters in the path portion of the URL.May 1, 2009

If we don't use IE, should I be concerned about potentially exceeding the limit on the query string length? I am certain I have seen the limit exceeded on my Node.js Express server, specifically when I included a base64 string representing an image in the query string of a GET request.

What's a good way to get around this problem? Should we just use an HTTP POST request instead? Certainly we do not want to break apart 1 GET request into 1000's just to avoid this problem.

Turns out with Chrome Canary, we hit a limit in the URL length pretty quick (definitely not anything close to unlimited length). — Alexander Mills, Jul 24 '17 at 17:56
Seems like the best thing to do is find a way to query the collection on a field that does not change for each document / row. So instead of an id, which is unique for each row, search by some shared id that can identify the subset of the collection you wish to retrieve. — Alexander Mills, Jul 24 '17 at 18:25
https://stackoverflow.com/questions/14202257/design-restful-query-api-with-a-long-list-of-query-parameters, https://stackoverflow.com/questions/4203686/how-can-i-deal-with-http-get-query-string-length-limitations-and-still-want-to-b, https://stackoverflow.com/questions/5020704/how-to-design-restful-search-filtering, https://stackoverflow.com/questions/207477/restful-url-design-for-search — CodeCaster, Jul 25 '17 at 08:40
@CássioMazzochiMolin I didn't mean to cause you to delete your answer, I rather wanted to discuss it. My comment may have sounded too strongly-worded. — CodeCaster, Jul 25 '17 at 08:41
@CodeCaster That definitely was a bad design: Persistence would be an issue and for sure it would violate the statelessness constraint. I deleted to avoid encouraging other users to follow that approach :) — cassiomolin, Jul 25 '17 at 09:12
@CássioMazzochiMolin glad you agree. Sad to see the same approach mentioned various times in the questions linked in the above comment. It's really hard to do it _well_, and then still, you're building an extra dependency just to keep your search "restful", whatever that may mean. Of course you can ask yourself why you need a search with a parameter weight of 2 KB in the first place... — CodeCaster, Jul 25 '17 at 09:16
possible duplicate of https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string — Marcel, Jul 31 '17 at 14:04
@Marcel I am looking for a good solution to the problem, not really looking to find out the maximum query length, because we have already hit it. — Alexander Mills, Jul 31 '17 at 16:11
Web infrastructure often allows caching of GET requests but HTTP query string have limited length. It can be limited by the client (Firefox, IE, ...), the server (Apache, IIS, ...) or the network equipment (applicative firewall, ...). I thought, In this case we should use POST. I read and got reference link [here](https://stackoverflow.com/questions/4203686/how-can-i-deal-with-http-get-query-string-length-limitations-and-still-want-to-b) — Von, Jul 31 '17 at 17:44

score 5 · Answer 1 · edited Jan 04 '20 at 18:55

If you intend to keep your API clean and RESTful, you could split your request into two parts. The first one will send your configuration to the server for calculations (POST); the second one will collect the results (GET).

This approach has multiple advantages besides overcoming the limit of HTTP GET on browser, firewall, routers, etc. For example, you can later introduce partial result polling or a possibility to cancel a long running calculation. Furthermore, you keep the URL short and user-readable.

The implementation details were already described here.

score 2 · Answer 2 · answered Jul 14 '17 at 23:47

In a RESTful web service, GET retrieves data. POST requests create data on the server and do not have size limits, but servers often have a default limit that can be changed.

If maintaining a RESTful application does not apply to your situation, then you can use POST.

It is important to note that GET requests can be cached but POST requests cannot, so if you are getting a lot of (the same) data in return, you should use GET.

score 0 · Answer 3 · answered Jul 31 '17 at 10:30

I would say that if you have GET request with a length over 2000 than it smells like something is wrong in the design (I can't find a good reason to pass an image via GET in base64 instead of POST as you mentioned). I would rather go with POSTs in that scenarios; also searching around about the 2048 characters limit seems it does not really apply, see firefox and Chrome and i won't rely on this dated info about microsoft ie, here a more detailed analysis.

You can also consider to implement a string shortener for your parameters names and values but in my opinion readability is more important. Have a look at this question on how to shorten parameter in javascript.

score 0 · Answer 4 · answered Jul 31 '17 at 14:02

The limit from IE applies to both GET and POST requests, so first things first, a POST request won't totally solve your problem here.

For browsers, there's no universal limit, each browser implements different limits with different consequences.

In addition, apart from limits placed by the web browsers, servers could also have their own limits, either default or set (e.g. Apache's LimitRequestLine Directive).

Also, memory problems could be an obstacle when processing data (PHP for example can set memory limits for each script), and PHP on Apache also has a configurable post_max_size directive.

Generally, consensus on this subject lays around limiting your GET request length to a maximum of 2000 characters, or to use POST requests (albeit with limits as well).

My suggestion: Truncating your request and sending several requests which get stored by the server and processed when a request comes with a final flag might be a possible solution. Also, string compression libraries exist to help you compress and decompress large data, see LZW compression for example which works with several dozen programming languages.

HTTP GET request with many items in query string

4 Answers4