1

I'm using a stream passthrough to pipe a large file to an S3 bucket and I believe the response is being sent too soon - before the stream and be initialized. Something like that. Smaller requests seem to work just fine.

Here's an example:

const express = require("express");
const router = express.Router();

const stream = require("stream");
const AWS = require("aws-sdk");

const s3 = new AWS.S3(awsCreds);

const uploadFromStream = id => {
    let pass = new stream.PassThrough();
    let params = {
        Bucket: "testbucket",
        Key: `test/${id}.json`,
        Body: pass
    };
    s3.upload(params, (err, data) => {
        if (err) {console.error(err)};
        console.log(data);
    });
    return pass;
}

router.post("/filestream", (req, res) => {
    let id = Math.floor(Math.random() * 100000);
    req.pipe(uploadFromStream(id));
    res.status(200).send(`API: Stream initiated. Check S3 path for file: '${id}.json'`);
});

module.exports = router;

The idea is; don't lock the thread to allow multiple requests for streams, which is why I'm sending the 200 back immediately.

If I put a 1 sec delay in there, I can get much larger POSTs through, consistently. 

router.post("/filestream", (req, res) => {
    let id = Math.floor(Math.random() * 100000);
    req.pipe(uploadFromStream(id));
    setTimeout(() => {
        res.status(200).send(`API: Stream initiated. Check S3 path for file: '${id}.json'`);
    }, 1000);
});

However, that feels hacky. I would prefer not to intentionally lock the thread and slow all requests for all users. Streaming was supposed to solve that issue.

Is there a better way to do this? Is throttling necessary? What am I missing?

Tsar Bomba
  • 1,047
  • 6
  • 29
  • 52
  • Do you really need to set the 200 status? Maybe you can use a promise like in [this answer](https://stackoverflow.com/a/50291380/4761254) It won't block the main thread, when the request finishes it executes the code of the success function. – Albondi Aug 21 '19 at 14:09
  • @Albondi No, but I do need to send a response or the request hangs. When am I sending the response in this case? After the promise is executed (inside of then() or after awaiting it?) Will awaiting the promise before sending the response cause the thread to hang? Thanks! – Tsar Bomba Aug 21 '19 at 15:47
  • 1
    Sorry I edited this answer...both await or then will not cause the main thread to hang. – Albondi Aug 21 '19 at 16:33
  • [The fundamental difference between await and vanilla promises](https://mathiasbynens.be/notes/async-stack-traces#targetText=The%20fundamental%20difference%20between%20await,this%20difference%20is%20pretty%20significant.) is that await X() suspends execution of the current function, while promise.then(X) continues execution of the current function after adding the X call to the callback chain. – Albondi Aug 21 '19 at 16:42
  • 1
    @Albondi Seems to have worked wonderfully. Thanks! If you'd like to post it as an answer, I'd be happy to give you credit for it. – Tsar Bomba Aug 29 '19 at 11:56
  • 1
    Great! Don't worry, my comment was just a comment for guidance, you came up with the solution. Cheers! – Albondi Aug 29 '19 at 13:42

0 Answers0