How can a web server know when an HTTP request is fully received?

Question

I'm currently writing a very simple web server to learn more about low level socket programming. More specifically, I'm using C++ as my main language and I am trying to encapsulate the low level C system calls inside C++ classes with a more high level API.

I have written a Socket class that manages a socket file descriptor and handles opening and closing using RAII. This class also exposes the standard socket operations for a connection oriented socket (TCP) such as bind, listen, accept, connect etc.

After reading the man pages for the send and recv system calls I realized that I needed to call these functions inside some form of loop in order to guarantee that all bytes are successfully sent/received.

My API for sending and receiving looks similar to this

void SendBytes(const std::vector<std::uint8_t>& bytes) const;
void SendStr(const std::string& str) const;
std::vector<std::uint8_t> ReceiveBytes() const;
std::string ReceiveStr() const;

For the send functionality I decided to use a blocking send call inside a loop such as this (it is an internal helper function that works for both std::string and std::vector).

template<typename T>
void Send(const int fd, const T& bytes)
{
   using ValueType = typename T::value_type;
   using SizeType = typename T::size_type;

   const ValueType *const data{bytes.data()};
   SizeType bytesToSend{bytes.size()};
   SizeType bytesSent{0};
   while (bytesToSend > 0)
   {
      const ValueType *const buf{data + bytesSent};
      const ssize_t retVal{send(fd, buf, bytesToSend, 0)};
      if (retVal < 0)
      {
          throw ch::NetworkError{"Failed to send."};
      }
      const SizeType sent{static_cast<SizeType>(retVal)};
      bytesSent += sent;
      bytesToSend -= sent;
   }
}

This seems to work fine and guarantees that all bytes are sent once the member function returns without throwing an exception.

However, I started running into problems when I began implementing the receive functionality. For my first attempt I used a blocking recv call inside a loop and exited the loop if recv returned 0 indicating that the underlying TCP connection was closed.

template<typename T>
T Receive(const int fd)
{
   using SizeType = typename T::size_type;
   using ValueType = typename T::value_type;

   T result;

   const SizeType bufSize{1024};
   ValueType buf[bufSize];
   while (true)
   {
      const ssize_t retVal{recv(fd, buf, bufSize, 0)};
      if (retVal < 0)
      {
          throw ch::NetworkError{"Failed to receive."};
      }

      if (retVal == 0)
      {
          break; /* Connection is closed. */
      }

      const SizeType offset{static_cast<SizeType>(retVal)};
      result.insert(std::end(result), buf, buf + offset);
   }

   return result;
}

This works fine as long as the connection is closed by the sender after all bytes have been sent. However, this is not the case when using e.g. Chrome to request a webpage. The connection is kept open and my receive member function is stuck blocked on the recv system call after receiving all bytes in the request. I managed to get around this problem by setting a timeout on the recv call using setsockopt. Basically, I return all bytes received so far once the timeout expires. This feels like a very inelegant solution and I do not think that this is the way web servers handles this issue in reality.

So, on to my question.

How does a web server know when an HTTP request have been fully received?

A GET request in HTTP 1.1 does not seem to include a Content-Length header. See e.g. this link.

Answer 1

HTTP/1.1 is a text-based protocol, with binary POST data added in a somewhat hacky way. When writing a "receive loop" for HTTP, you cannot completely separate the data receiving part from the HTTP parsing part. This is because in HTTP, certain characters have special meaning. In particular, the CRLF (0x0D 0x0A) token is used to separate headers, but also to end the request using two CRLF tokens one after the other.

So to stop receiving, you need to keep receiving data until one of the following happens:

• Timeout – follow by sending a timeout response
• Two CRLF in the request – follow by parsing the request, then respond as needed (parsed correctly? request makes sense? send data?)
• Too much data – certain HTTP exploits aim to exhaust server resources like memory or processes (see e.g. slow loris)

And perhaps other edge cases. Also note that this only applies to requests without a body. For POST requests, you first wait for two CRLF tokens, then read Content-Length bytes in addition. And this is even more complicated when the client is using multipart encoding.

Answer 2

A request header is terminated by an empty line (two CRLFs with nothing between them).

So, when the server has received a request header, and then receives an empty line, and if the request was a GET (which has no payload), it knows the request is complete and can move on to dealing with forming a response. In other cases, it can move on to reading Content-Length worth of payload and act accordingly.

This is a reliable, well-defined property of the syntax.

No Content-Length is required or useful for a GET: the content is always zero-length. A hypothetical Header-Length is more like what you're asking about, but you'd have to parse the header first in order to find it, so it does not exist and we use this property of the syntax instead. As a result of this, though, you may consider adding an artificial timeout and maximum buffer size, on top of your normal parsing, to protect yourself from the occasional maliciously slow or long request.

Answer 3

The solution is within your link

A GET request in HTTP 1.1 does not seem to include a Content-Length header. See e.g. this link.

There it says:

It must use CRLF line endings, and it must end in \r\n\r\n

Answer 4

The answer is formally defined in the HTTP protocol specifications ¹:

• in W3C's spec for HTTP 0.9.

• in RFC 1945 for HTTP 1.0, specifically in Section 4: HTTP Message, Section 5: Request, and Section 7: Entity.

• in RFC 2616 for HTTP 1.1, specifically in Section 4: HTTP Message, particular in 4.3: Message Body and 4.4: Message Length.

• in RFC 7230 (and 7231...7235) for HTTP 1.1, specifically in Section 3: Message Format, in particular 3.3: Message Body.

So, to summarize, the server first reads the message's initial start-line to determine the request type. If the HTTP version is 0.9, the request is done, as the only supported request is GET without any headers. Otherwise, the server then reads the message's message-headers until a terminating CRLF is reached. Then, only if the request type has a defined message body then the server reads the body according to the transfer format outlined by the request headers (requests and responses are not restricted to using a Content-Length header in HTTP 1.1).

In the case of a GET request, there is no message body defined, so the message ends after the start-line in HTTP 0.9, and after the terminating CRLF of the message-headers in HTTP 1.0 and 1.1.

^{1: I'm not going to get into HTTP 2.0, which is a whole different ballgame.}