How to chop up response to REST-call into small chunks?

Question

I am working on a web-application in node.js. Some parts of the application allow users to upload/retrieve files. I use the 'SoftLayer Object Storage'-service to host these files. When serving a video, however, it plays fine in the HTML5-video element on Chrome/Firefox/Android but it does not on Safari/iOS.

To get to the bottom of this problem, I ran into this explanation on another Stackoverflow-post:

iOS devices expect the videos to arrive in small chunks. So for instance a streaming server is able to do this. However, a blob server just hands the video as a blob which is not what the iOS device expects. Some browsers are smart enough to handle this but others not.

In order to check whether this explanation made any sense, I tried to stream the same mp4-file. In order to do so, I based myself on the accepted answer on this stackoverflow-post and this actually worked fine on all browsers and platforms which leads me to believe that the explanation I quoted above is actually true.

Unfortunately, this streaming-code will not work in my case because the file is hosted on SoftLayer whereas the accepted assumes that the file exists on the filesystem of the server. I communicate with the "Softlayer Object Store" through this API. The code I have now looks like this:

router.get('/testFile', function (req, res) {
  res.writeHead(200, {'Content-Type' : 'video/mp4','Accept-Ranges': 'bytes'});
  request.get({url: authEndpoint, headers: {"X-Auth-Key": apiKey, "X-Auth-User": user}}, function (err, res1) {
    var data = JSON.parse(res1.body);
    var objectPath = data.storage.public + '/' + container + '/' + filename;
    request.get({
      url: objectPath,
      headers: {"X-Auth-Token": res1.headers['x-auth-token']}
    }, function (err) {
      if (err) {
        console.log('error', err);
      }
    }).pipe(res);
  });
});

Obviously, this code doesn't "stream" the video and just sends the entire video to the client at once. Using this code, the video does not work in Safari.

I would like to somehow send the video chunk by chunk (as is the case in the stackoverflow-answer discussed above) rather than entirely at once but I have no idea how to do this when the video is retrieved through a REST-request. I did notice that the API allows one to provide an optional "Range"-parameter when doing a GET-request but I'm not sure how I can use this to my advantage.

Finally I would also like the mention that I don't believe that the encoding of the video is the problem. I am using this video http://www.w3schools.com/html/mov_bbb.mp4 and when I type in that link on iOS/safari it works just fine.

Answer 1

The problem is that you're trying to pipe the whole response from the API, with status and headers and so on.

Instead, you need to convert the response body to a readable stream and pipe it to the response. To convert a Buffer to a ReadableStream you could use a library like stream-buffers

So, your code should look somewhat like this:

router.get('/testFile', function (req, res) {
  res.writeHead(200, {'Content-Type' : 'video/mp4','Accept-Ranges': 'bytes'});
  request.get({url: authEndpoint, headers: {"X-Auth-Key": apiKey, "X-Auth-User": user}}, function (err, res1) {
    var data = JSON.parse(res1.body);
    var objectPath = data.storage.public + '/' + container + '/' + filename;
    request.get({
      url: objectPath,
      headers: {
        "X-Auth-Token": res1.headers['x-auth-token']
      }
    }, function (err, data) {
      if (err) {
        console.log('error', err);
      } else {
        var bodyStream = new streamBuffers.ReadableStreamBuffer({
          frequency: 10,   // in milliseconds. 
          chunkSize: 2048  // in bytes. 
        });
        bodyStream.pipe(res);
        bodyStream.put(data.body);
      }
    });
  });
});