Why is fs.createReadStream ... pipe(res) locking the read file?

Question

I'm using express to stream audio & video files according to this answer. Relevant code looks like this:

function streamMedia(filePath, req, res) {
  // code here to determine which bytes to send, compute response headers, etc.

  res.writeHead(status, headers);
  var stream = fs.createReadStream(filePath, { start, end })
    .on('open', function() {
      stream.pipe(res);
    })
    .on('error', function(err) {
      res.end(err);
    })
  ;
}

This works just fine to stream bytes to <audio> and <video> elements on the client. However after these requests are served, another express request can delete the file being streamed from the filesystem. This second request is failing, sort of.

What happens is that as long as the file is streamed at least once (meaning a createReadStream was invoked for the file's path while running the code above), then a different express request comes in to delete the file, the file remains on the filesystem until express is stopped. As soon as express is stopped, the files are deleted from the filesystem.

What exactly is going on here? Is it fs or express that is locking the file, why, and how can I get the process to release the file so that it can be deleted (after its contents have been read and piped to a response, if any is pending)?

Update 1:

I've modified the above code to set autoClose: true for the second function arg, and added both 'end' and 'close' event handlers, like so:

res.writeHead(status, headers);
var streamReadOpts = { start: start, end: end, autoClose: true };
var stream = fs.createReadStream(filePath, streamReadOpts)
    // previous 'open' & 'error' event handlers are still here
    .on('end', function () {
      console.log('stream end');
    })
    .on('close', function () {
      console.log('stream close');
    })

What I have discovered is that when a page initially loads with a <video> or <audio> element, only the 'open' even is fired. Then when the user clicks to play the video/audio, a second request is made, and this second time, both the 'end' and 'close' events fire, and subsequently deleting the file succeeds.

So it appears that the file is being locked when a user loads the page that has the <video> or <audio> element that gets its source from the request that calls this function. It isn't until that media file is played that a second request is made, and the file is unlocked.

I've also discovered that closing the browser also causes the 'end' and 'close' events to fire, and the file to be unlocked. My guess is that I'm doing something wrong with the express res to make it not close properly, but I'm still not sure what that could be.

score 4 · Accepted Answer · edited May 23 '17 at 11:53

It turned out the solution to this was to read and pipe smaller blocks of data from the file during each request. In my test cases for this, I was streaming a 6MB MP4 video file. Though I was able to reproduce the issue using either firefox or chrome, I debugged using the latter, and found that the client was blocking the stream.

When the page initially loads, there is an element that looks something like this:

<video> <!-- or <audio> -->
    <source src="/path/to/express/request" type="video/mpeg" /> <!-- or audio/mpeg -->
</video> <!-- or </audio> -->

As is documented in the other answer referenced in the OP, chrome will send a request with a range header like so:

Range:bytes=0-

For this request, my function was sending the whole file, and my response looked like this:

Accept-Ranges:bytes
Connection:keep-alive
Content-Length:6070289
Content-Range:bytes 0-6070288/6070289
Content-Type:video/mp4

However, chrome was not reading the whole stream. It was only reading the first 3-4MB, then blocking the connection until a user action caused it to need the rest of the file. This explains why closing either the browser or stopping express caused the files to be unlocked, because it closed the connection from either the browser or the server's end.

My current solution is to only send a maximum of 1MB (the old school 1MB, 1024 * 1024) chunk at a time. The relevant code can be found in an additional answer to the question referenced in the OP.

score 1 · Answer 2 · answered Aug 20 '16 at 02:51

1

Set autoClose = true in options. If autoClose = false you have to close it manually in 'end' event.

Refer node doc :- https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options

answered Aug 20 '16 at 02:51

Abhilash Km

116
1
5

I was optimistic about this answer because I had read those docs, and overlooked the fact that I was passing `start` and `end` in the options. However even after adding `autoClose: true` to the 2nd options argument, the issue persists. Also noteworthy is that the `'end'` even is not always firing. – danludwig Aug 20 '16 at 13:05

Why is fs.createReadStream ... pipe(res) locking the read file?

Update 1:

2 Answers2

Linked