1

I have this HTTP header that came to my C server include include request and headers.

"GET / HTTP/1.1\r\n" /
"Host: localhost:5000\r\n" /
"User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0\r\n" /
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\r\n" /
"Accept-Language: en-US,en;q=0.5\r\n" /
"Accept-Encoding: gzip, deflate\r\n" /
"Connection: keep-alive\r\n" /
"Upgrade-Insecure-Requests: 1\r\n" /
"Sec-Fetch-Dest: document\r\n" /
"Sec-Fetch-Mode: navigate\r\n" /
"Sec-Fetch-Site: none\r\n" /
"Sec-Fetch-User: ?1\r\n" /

I like to know what the maximum size is of each line until \n according to HTTP protocol which should be valid so I can try parsing it easily.

"GET / HTTP/1.1\r\n" /
"Host: localhost:5000\r\n" /
"User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0\r\n" /
"Sec-Fetch-User: ?1"
Yun
  • 3,056
  • 6
  • 9
  • 28
user786
  • 3,902
  • 4
  • 40
  • 72
  • 1
    related https://stackoverflow.com/questions/686217/maximum-on-http-header-values – Agent_L Oct 12 '21 at 08:38
  • @Agent_L that link says `For example in Apache default limit is 8KB,` that means the socket buffer size has to hold and can be able to hold 8000 characters as header and request combined in a request. is that means all the lines in my question can come to apache and allowed as headers and request and more than that size is not allowed or will not be read? please confirm – user786 Oct 12 '21 at 08:44
  • 1
    Yeah, it seems that they try to not put a limit on a single line. But the limit is only on headers, the body is unlimited. – Agent_L Oct 12 '21 at 08:51
  • @Agent_L `the body is unlimited` if body is allowed unlimited in some server then how they don't run out of space on computer. what this mean there no data type storage I know of in C that is unlimited in size how they handle this with series of `realloc` calls. Can u clarify please? – user786 Oct 12 '21 at 09:13
  • Yes, the physical limits of the machine always apply. – Agent_L Oct 12 '21 at 10:04

2 Answers2

3

The short answer is that there is no such restriction. The web server may have a limit on the total size of the HTTP request headers it will accept.

  • Apache : 8K
  • IIS : 8K - 16K
  • nginx : 4K - 8K
  • Tomcat : 8K - 16K

You could test the limit of your server by sending requests with junk headers and checking the response status code. A 400, 413 or 431 response would be an indication that you reached the limit.

Further investigation:

I checked out the response codes from big name domains and there is no consistency in the response code received.

For example:

Here's the Python script I wrote for the task:

#!/usr/bin/env python3 
import os
import random
import requests
import string
import sys

def header_size_test(url, limit=64):
    print(f'Checking "{url}".')
    # Generate a large enough string of ASCII characters
    size_limit = limit * 1000
    junk_string = ''.join(random.choice(string.ascii_letters + string.digits) for _ in range(size_limit))
    for size in range(4, (limit+1)):
        N = size * 1000
        # Set the junk header of the required size
        headers = {'junk': junk_string[:N]}
        # Send the request
        response = requests.get(url, headers)
        # Display the response
        print(f'Header size: {size}K, Response Code: {response.status_code}')
        # No point in continuing if the server response is not OK (200)
        if (response.status_code != 200):
            break

if __name__ == '__main__':
    if len(sys.argv) > 1:
        header_size_test(sys.argv[1])
    else:
        print(f'Please provide a url string.\n\tExample: {os.path.basename(__file__)} "http://www.google.com"')

Implementing in your server

HTTP GET request headers end with \r\n\r\n so you can search your buffer for this substring. My knowledge of C is a bit rusty so please forgive any syntax errors.

#define MAX_HEADER_LENGTH 4000
char *marker = strstr(buffer, "\r\n\r\n");
// end of header found?
if (marker != NULL) {
  // find the length of the headers in buffer
  headers_length = &marker - &buffer;
  if (MAX_HEADER_LENGTH < headers_length) {
    // send a nice response
  }
}

If you need to parse the content of the headers then refer to the RFC 2616 document.

Dan Nagle
  • 4,384
  • 1
  • 16
  • 28
  • can u please talk about little bit `413 Payload Too Large` how I can find the size of received http request and headers. Is this what returned from `data.read=read(...)`function? I have to match with my server limit and if return of `read()` is greater than the limit I reply with `413` is this how its done? – user786 Oct 12 '21 at 09:06
  • 1
    I've updated the answer to outline a solution to calculating the header length. – Dan Nagle Oct 12 '21 at 10:31
  • I don't think it's good idea to model your limits after other services. OP should model their limits according to the needs of their service. – Agent_L Oct 12 '21 at 15:31
  • 1
    @Agent_L It wasn't my intention to suggest such. I was highlighting the inconsistency of the response codes when a request breaches their limits. – Dan Nagle Oct 12 '21 at 18:30
1

There seems to be no protocol-level limit. Existing limits are implementation specific. Which means that you parse as much as you can and when you can't anymore you respond 431 Request Header Fields Too Large

Quote:

431 can be used when the total size of request headers is too large, or when a single header field is too large
Agent_L
  • 4,960
  • 28
  • 30