24

Is there a limit to the size when making HTTP GET requests in Node.js? And if so, how can I change this?

var url = "..." // very long, ~50'000 chars
http.get(url, function (res) {
    res.pipe(fs.createWriteStream("file.txt"));
});

Gives me this:

<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>

Same thing using wget in PowerShell or Bash works perfectly fine.

$url = "..."
wget -outf file.txt $url
Bruno Schäpper
  • 1,262
  • 1
  • 12
  • 23
  • what URL could possibly be >50K chars, is this a base64 url or something, if so see: http://stackoverflow.com/questions/20267939/nodejs-write-base64-image-file – RobertPitt Sep 24 '15 at 14:44
  • 1
    some sort of an api call.. it _is_ quite large i know ;-) and possibly borderline insane – Bruno Schäpper Sep 25 '15 at 06:25

2 Answers2

31

There is a built-in request size limit enforced by Node. Requested headers + URI should not be more than 80 kb.

As it is defined in http_parser.h#L55: 

/* Maximium header size allowed */
#define HTTP_MAX_HEADER_SIZE (80*1024)

Asuming that UTF-8 character can be between 1 and 4 bytes, the size of a string with 50 000 characters would be from 50 000 to 200 000 bytes, or from ~48kb to 195kb.

Community
  • 1
  • 1
Alexandr Lazarev
  • 12,554
  • 4
  • 38
  • 47
  • 3
    can we override this max limit? I want more to be used in my application. – anubhs Jul 19 '18 at 08:20
  • 3
    It is for the size of the header. Is there any limit for the GET URL length – Newton Joshua Feb 28 '19 at 14:15
  • 1
    @NewtonJoshua, not as far as I can tell. I just booted up a local node.js server and created a very long URL. Node returned a 431 "Request Header Fields Too Large" when the URL + Request Header size went over 8.052 KB (so 8,052 Bytes). – Mark Kurkowski Dec 20 '19 at 22:44
21

UPDATE

In newer versions of NodeJS (v12.6.0+), the header request size limit is 8kb by default, and this also impacts the response header size. To use header sizes above 8KB, you can use the switch --max-http-header-size 65535 (or whatever size you need). This is described in-depth here.

Andrew Knackstedt
  • 533
  • 4
  • 8
  • Oh my, that's very relevant for us. Thank you for the heads up! – Bruno Schäpper Jul 10 '19 at 15:06
  • Is there any way this can be done programmatically at runtime? I am now able to add command line switches to node... – Spock Jul 18 '19 at 16:15
  • @Spock I'm not sure, I have mine set in the package.json file. I don't think there is built in functionality for NodeJS to change that after startup. You could write something that updates the start command or package.json before Node starts though. – Andrew Knackstedt Jul 18 '19 at 17:43
  • how do you enable/define the 'switch' `--max-http-header-size`? just in the `start` script in `package.json`, eg `nodemon app.js --max-http-header-size 65535`? – user1063287 Jul 22 '19 at 08:57
  • 3
    @user1063287 I put it right after the command in my package.json `node --max-http-header-size 1048576 ./bin/www` – Andrew Knackstedt Jul 22 '19 at 12:07
  • do u know if including `--max-http-header-size 80000` only applies the setting until `package.json` is saved again? eg if i included it, and later wanted to 'undo' it, would removing the flag from the command revert the setting back to the default 8kb limit? – user1063287 Jul 22 '19 at 12:13
  • Adding it to the package.json file will keep the setting applied for all startups that occur with that configuration. If you wanted to 'undo' it, removing the flag and restarting the server would work. – Andrew Knackstedt Jul 22 '19 at 12:41