4

I've been messing around with HTTP over the past few days by building a simple CL download manager in Rust using the reqwest crate to handle the HTTP stuff. I've got a basic understanding of how the protocol works - what headers to look for when downloading a file, how to verify the request has worked etc. - but I'm having trouble finding an answer as to where the actual bytes in the body of a HTTP response are stored.

For example, sending a request and getting a response with reqwest takes very little time, so I figure that the downloading can't happen at this stage. What actually takes time is reading the bytes from the body. But surely this body can't be stored in RAM, as downloading a large file would make memory use skyrocket. I realise that where this data is stored may be different across different HTTP frameworks, but I suppose that what I'm after is a more general explanation of where a large HTTP response body is stored when not using a browser to download a file.

James Mclaughlin
  • 570
  • 1
  • 8
  • 16

1 Answers1

9

In the case of reqwest, the response body is not stored fully in memory unless you ask it to be (by calling .bytes() or .json() for example) — the network connection is still active at that point (the headers have been fully received, but not the body), and so the server is responsible for storing or otherwise being ready to provide the rest of the response. It may be that the HTTP server has the response in its memory, or it may be reading directly from its own disk; and parts of the response will be momentarily stored in various network buffers or moving along cables from their network to yours.

This is why Response doesn't implement Clone, and why the methods for retrieving the body take self; a Response is (besides a way to read response headers) a handle to a network connection that's not finished. You use it to instruct reqwest how to deliver the rest of the file to you — reading it into memory, parsing it into some JSON or other data type, or even processing the bytes as they come in using your own code.

Every good HTTP client will have functionality like this, simply because it is not efficient to store a large response entirely into memory before doing whatever the next step is.

Kevin Reid
  • 37,492
  • 13
  • 80
  • 108
  • Somewhat separate from the original topic, this still doesn't explain why reading body has to consume the whole response `self` (including headers and status code). There is no fundamental reason why it should not be possible to get request headers/status code after having read body. Reading body can only be done once, of course, but it could just borrow headers it needs to read the body, instead of consuming them entirely. – nirvana-msu Feb 16 '22 at 11:48