3

I am debugging a test case. I use Python's OptionParser (from optparse) to do some testing and one of the options is a HTTP request.

The input in this specific case for the http request was 269KB in size. So my python program fails with "Argument list too long" (I verified that there was no other arguments passed, just the request and one more argument as expected by the option parser. When I throw away some of the request and reduce its size, things work fine. So I have a strong reason to believe the size of the request is causing my problems here.)

Is it possible for a HTTP request to be that big ? If so how do I fix the OptionParser to handle this input?

Alex Riley
  • 169,130
  • 45
  • 262
  • 238
ap1234
  • 489
  • 2
  • 6
  • 15
  • 1
    Are you passing a very long string from the command line? – Padraic Cunningham Jul 25 '15 at 20:11
  • 'Argument list too long' makes it sound like the shell/operating system objected to how long your command line was before the HTTP request even ran (but yes, others are correct that GETs in practice top out well before 269KB). – twotwotwo Jul 25 '15 at 20:13
  • Don't use options to pass in DATA for a request. You should take it as a stream. – Lucas Holt Jul 25 '15 at 20:14
  • This might be relevant as far as your string length goes http://stackoverflow.com/questions/29801975/why-is-the-subprocess-popen-argument-length-limit-smaller-than-what-the-os-repor/29802900#29802900 – Padraic Cunningham Jul 25 '15 at 20:19

3 Answers3

3

Is it possible for a HTTP request to be that big ?

Yes it's possible but it's not recommended and you could have compatibility issues depending on your web server configuration. If you need to pass large amounts of data you shouldn't use GET.

If so how do I fix the OptionParser to handle this input?

It appears that OptionParser has set its own limit well above what is considered a practical implementation. I think the only way to 'fix' this is to get the Python source code and modify it to meet your requirements. Alternatively write your own parser.

UPDATE: I possibly mis-interpreted the question and the comment from Padraic below may well be correct. If you have hit an OS limit for command line argument size then it is not an OptionParser issue but something much more fundamental to your system design that means you may have to rethink your solution. This also possibly explains why you are attempting to use GET in your application (so you can pass it on the command line?)

MattP
  • 2,798
  • 2
  • 31
  • 42
2

A GET request, unlike a POST request, contains all its information in the url itself. This means you have an URL of 269KB, which is extremely long.

Although there is no theoretical limit on the size allowed, many servers don't allow urls of over a couple of KB long and should return a 414 response code in that case. A safe limit is 2KB, although most modern software will allow a bit more than that.

But still, for 269KB, use POST (or PUT if that is semantically more correct), which can contain larger chunks of data as the content of a request rather than the url.

GolezTrol
  • 114,394
  • 18
  • 182
  • 210
0

Typical limit is 8KB, but it can vary (like, be even less).

Borsunho
  • 1,127
  • 7
  • 21