1

The Situation

My team is creating an API that receives a large structured text file (100MB - 1TB, 1GB expected) and modifies each row and returns the resulting file. We can process the file as fast as it is transmitted, so would like to avoid caching the file on our servers. We favor ease-of-use for our clients over our own resource use, so this is not a hard requirement.

Some Options

HTTP/1.1 implicitly requires that the full request be processed before the response is sent (except in the case of errors) and bad things can happen, especially with proxies, if you try to get around this. So we were going to bite the bullet and store the request or response and use another resource in our organization for uploading large files for processing.

HTTP/2 explicitly allows you to send before the request has finished and requires that the client read what you send and HTTP/2 is already supported in all major browsers.

So, I see a few potential apis (all POST):

HTTP1.x: upload/download - there is already some infrastructure for this

/transformed_file_id/ --> returns id for the uploaded file 
/transformed_file/{id} --> returns the transformed data

HTTP1.x: single request

/transformed_file/ --> returns the transformed version of the file - stores stuff under-the-hood

HTTP2: single request

/transformed_file/ --> returns the transformed version of the file - starts sending response as soon as it receives the first couple of K.

The Question(s)

Though I wouldn't shy away from it for browser content, is it wise to use HTTP/2 for a service in order to access this feature?

Or is all this a bad idea and clients should be forced to upload smaller parts of the file at a time (and we'll need to write a front-end to allow this on a browser-interface - which could be quite tough).

Community
  • 1
  • 1
Eponymous
  • 6,143
  • 4
  • 43
  • 43

2 Answers2

3

My experience with various clients, servers and proxies is that it is not true that HTTP/1.1 requires that the full request to be sent before an application can start responding. It happens all the times.

On the other hand, if your clients have to upload 100 MiB - 1 TiB of data (!) in a single request, I would setup some mechanism to recover upload failures, similar to the range headers for downloads. See also: Standard method for HTTP partial upload, resume upload

Having said that, with HTTP/2 and big uploads you have to pay special attention to the client's flow control send window. This is by default 64 KiB, which means that the client can only send at most 64 KiB before waiting for the server to acknowledge that content. The acknowledgement must travel from server to client, so the network latency plays an important role here: the client may be really fast at writing the 64 KiB, but then wait most of the time for the server acknowledgement. This could cause terrible slowdowns of uploads.

Just to give you an idea, browsers (Firefox) modify their receive window to be able to perform fast downloads from servers from 64 KiB to 12 MiB (almost 200x). Unfortunately they don't do the same for uploads.

You don't specify if your clients are browsers or not; if not, make sure you can have control over the configuration of the flow control window, both send and receive, and enlarge them enough to not be slowed down by flow control acknowledgements.

Community
  • 1
  • 1
sbordet
  • 16,856
  • 1
  • 50
  • 45
0

As sbordet mentioned, HTTP/1.1 supports streaming downloads. This happens every time you download a large file.

However, I don't think your use case is good for simultaneously uploading downloading the new file. How long will it take for the client to upload 1TB? What if the connection is dropped half-way through?

It might be simpler and safer to allow the client to upload the whole file and then process it in the background. When it is finished, the client can download the whole file using their browser.

I think you will need a good way to manage the upload process. Although you could write Javascript or an extension, there should be many good upload manager extensions already available. File transfer over HTTP and through the browser is robust.

B Seven
  • 44,484
  • 66
  • 240
  • 385