2

I am proxying a web API that needs auth behind a Cloudflare Worker so some functions can be accessed anonymously. I have got simple POST messages working fine, but when it comes to POSTing file uploads it is unclear to me how to stream the file data from the client to the remote server.

It certainly appears that this feature is supported with Streams, per this example:

export default {
  async fetch(request, env, ctx) {
    // Fetch from origin server.
    let response = await fetch(request);

    // Create an identity TransformStream (a.k.a. a pipe).
    // The readable side will become our new response body.
    let { readable, writable } = new TransformStream();

    // Start pumping the body. NOTE: No await!
    response.body.pipeTo(writable);

    // ... and deliver our Response while that’s running.
    return new Response(readable, response);
  }
}

But I can't really make sense of how to have this example take the incoming file upload POST and pipe it to api.example.com/fileupload?

S.Richmond
  • 11,412
  • 6
  • 39
  • 57
  • If you are just passing the content through verbatim, you don't need to use streams. Simply doing `fetch(request)` will forward the whole request including the body. Workers will automatically use a streaming upload behind the scenes. If that isn't working for you, please post your code. – Kenton Varda Apr 14 '23 at 21:17
  • Doesn't fetch(request) simple pull in the file from the client? How do I push it to the remote server as the client uploads it? – S.Richmond Apr 16 '23 at 01:18
  • The Workers Runtime automatically uses streaming if you do `fetch(request)`. However, separately, Cloudflare will buffer uploads and only send the whole request to Workers once the whole file is uploaded to the client. This is something Cloudflare does even when not using Workers. Unfortunately at present there's no way to turn that off unless you have an enterprise contract. – Kenton Varda Apr 17 '23 at 15:01
  • Maybe I don't understand the code and/or your comment, but what I'm looking for it client -> worker -> remote server. Isn't `fetch(request)` simply doing client -> worker -> client? How do I fetch the entire file upload to the remote server? – S.Richmond Apr 18 '23 at 01:44
  • Are you just asking how to upload a file with `fetch`? If so: https://stackoverflow.com/q/36067767/12914833 You just need two fetch calls: one from client to worker and one from worker to server. – Chris Hamilton Apr 19 '23 at 00:51
  • I've since solved my problem with `fetch` yes, but AFAIK this isn't very time efficient - The worker will wait until the client (with a slow upload speed) has finished uploading to the worker, and then the worker will upload to the remote server. The reason I asked here about streams is that it could drastically reduce total transaction time if it could stream the upload straight away. – S.Richmond Apr 19 '23 at 06:17
  • Yes, this is because Cloudflare buffers request bodies as I mentioned -- something it does whether or not you're using Workers. This is meant to protect you against "slow loris" attacks but sometimes it gets in the way. Unfortunately there's no way to disable this feature right now, unless you are on an enterprise contract. – Kenton Varda Apr 19 '23 at 16:47

0 Answers0