Method for streaming data from browser to server via HTTP

Question

Are there any XHR-like browser APIs available for streaming binary to a server over HTTP?

I want to make an HTTP PUT request and create data programmatically, over time. I don't want to create all this data at once, since there could be gigs of it sitting in memory. Some psueudo-code to illustrate what I'm getting at:

var dataGenerator = new DataGenerator(); // Generates 8KB UInt8Array every second
var streamToWriteTo;
http.put('/example', function (requestStream) {
  streamToWriteTo = requestStream;
});

dataGenerator.on('data', function (chunk) {
  if (!streamToWriteTo) {
    return;
  }
  streamToWriteTo.write(chunk);
});

I currently have a web socket solution in place instead, but would prefer regular HTTP for better interop with some existing server-side code.

EDIT: I can use bleeding edge browser APIs. I was looking at the Fetch API, as it supports ArrayBuffers, DataViews, Files, and such for request bodies. If I could somehow fake out one of these objects so that I could use the Fetch API with dynamic data, that would work for me. I tried creating a Proxy object to see if any methods were called that I could monkey patch. Unfortunately, it seems that the browser (at least in Chrome) is doing the reading in native code and not in JS land. But, please correct me if I'm wrong on that.

Answer 1

I don't know how to do this with pure HTML5 APIs, but one possible workaround is to use a Chrome App as a background service to provide additional features to a web page. If you're already willing to use development browsers and enable experimental features, this seems like just an incremental step further than that.

Chrome Apps can call the chrome.sockets.tcp API, on which you can implement any protocol you want, including HTTP and HTTPS. This would provide the flexibility to implement streaming.

A regular web page can exchange messages with an App using the chrome.runtime API, as long as the App declares this usage. This would allow your web page to make asynchronous calls to your App.

I wrote this simple App as a proof of concept:

manifest.json

{
  "manifest_version" : 2,

  "name" : "Streaming Upload Test",
  "version" : "0.1",

  "app": {
    "background": {
      "scripts": ["background.js"]
    }
  },

  "externally_connectable": {
    "matches": ["*://localhost/*"]
  },

  "sockets": {
    "tcp": {
      "connect": "*:*"
    }
  },

  "permissions": [
  ]
}

background.js

var mapSocketToPort = {};

chrome.sockets.tcp.onReceive.addListener(function(info) {
  var port = mapSocketToPort[info.socketId];
  port.postMessage(new TextDecoder('utf-8').decode(info.data));
});

chrome.sockets.tcp.onReceiveError.addListener(function(info) {
  chrome.sockets.tcp.close(info.socketId);
  var port = mapSocketToPort[info.socketId];
  port.postMessage();
  port.disconnect();
  delete mapSocketToPort[info.socketId];
});

// Promisify socket API for easier operation sequencing.
// TODO: Check for error and reject.
function socketCreate() {
  return new Promise(function(resolve, reject) {
    chrome.sockets.tcp.create({ persistent: true }, resolve);
  });
}

function socketConnect(s, host, port) {
  return new Promise(function(resolve, reject) {
    chrome.sockets.tcp.connect(s, host, port, resolve);
  });
}

function socketSend(s, data) {
  return new Promise(function(resolve, reject) {
    chrome.sockets.tcp.send(s, data, resolve);
  });
}

chrome.runtime.onConnectExternal.addListener(function(port) {
  port.onMessage.addListener(function(msg) {
    if (!port.state) {
      port.state = msg;

      port.chain = socketCreate().then(function(info) {
        port.socket = info.socketId;
        mapSocketToPort[port.socket] = port;
        return socketConnect(port.socket, 'httpbin.org', 80);
      }).then(function() {
        // TODO: Layer TLS if needed.
      }).then(function() {
        // TODO: Build headers from the request.
        // TODO: Use Transfer-Encoding: chunked.
        var headers =
            'PUT /put HTTP/1.0\r\n' +
            'Host: httpbin.org\r\n' +
            'Content-Length: 17\r\n' +
            '\r\n';
        return socketSend(port.socket, new TextEncoder('utf-8').encode(headers).buffer);
      });
    }
    else {
      if (msg) {
        port.chain = port.chain.then(function() {
          // TODO: Use chunked encoding.
          return socketSend(port.socket, new TextEncoder('utf-8').encode(msg).buffer);
        });
      }
    }
  });
});

This App does not have a user interface. It listens for connections and makes a hard-coded PUT request to http://httpbin.org/put (httpbin is a useful test site but note it does not support chunked encoding). The PUT data (currently hard-coded to exactly 17 octets) is streamed in from the client (using as few or as many messages as desired) and sent to the server. The response from the server is streamed back to the client.

This is just a proof of concept. A real App should probably:

• Connect to any host and port.
• Use Transfer-Encoding: chunked.
• Signal the end of streaming data.
• Handle socket errors.
• Support TLS (e.g. with Forge)

Here is a sample web page that performs a streaming upload (of 17 octets) using the App as a service (note that you will have to configure your own App id):

<pre id="result"></pre>
<script>
 var MY_CHROME_APP_ID = 'omlafihmmjpklmnlcfkghehxcomggohk';

 function streamingUpload(url, options) {
   // Open a connection to the Chrome App. The argument must be the 
   var port = chrome.runtime.connect(MY_CHROME_APP_ID);

   port.onMessage.addListener(function(msg) {
     if (msg)
       document.getElementById("result").textContent += msg;
     else
       port.disconnect();
   });

   // Send arguments (must be JSON-serializable).
   port.postMessage({
     url: url,
     options: options
   });

   // Return a function to call with body data.
   return function(data) {
     port.postMessage(data);
   };
 }

 // Start an upload.
 var f = streamingUpload('https://httpbin.org/put', { method: 'PUT' });

 // Stream data a character at a time.
 'how now brown cow'.split('').forEach(f);
</script>

When I load this web page in a Chrome browser with the App installed, httpbin returns:

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 19 Jun 2016 16:54:23 GMT
Content-Type: application/json
Content-Length: 240
Connection: close
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {}, 
  "data": "how now brown cow", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Content-Length": "17", 
    "Host": "httpbin.org"
  }, 
  "json": null, 
  "origin": "[redacted]", 
  "url": "http://httpbin.org/put"
}

Answer 2

I'm currently searching for exactly the same thing (upstreaming via Ajax). What I currently found, looks as if we are searching at the bleeding edge of browser's feature design ;-)

XMLHttpRequest definition tells in step 4 bodyinit that the content extraction of this is (or can be) a readablestream.

I'm still searching (as a non-webdeveloper) for information of how to create such a thing and to feed data into the "other end" of that "readablestream" (which namely should be a "writablestream", but I yet did not find that).

Maybe you are better in searching and can post here if you found a method to implement these design plans.

^5
sven

Answer 3

An approach utilizing ReadableStream to stream arbitrary data; RTCDataChannel to send and, or, receive arbitrary data in form of Uint8Array; TextEncoder to create 8000 bytes of random data stored in a Uint8Array, TextDecoder to decode Uint8Array returned by RTCDataChannel to string for presentation, note could alternatively use FileReader .readAsArrayBuffer and .readAsText here.

The markup and script code were modified from examples at MDN - WebRTC: Simple RTCDataChannel sample, including adapter.js which contains RTCPeerConnection helpers; Creating your own readable stream.

Note also, example stream is cancelled when total bytes transferred reaches 8000 * 8 : 64000

(function init() {
  var interval, reader, stream, curr, len = 0,
    totalBytes = 8000 * 8,
    data = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789",
    randomData = function randomData() {
      var encoder = new TextEncoder();
      var currentStream = "";
      for (var i = 0; i < 8000; i++) {
        currentStream += data[Math.floor(Math.random() * data.length)]
      }
      return encoder.encode(currentStream)
    },
    // optionally reconnect to stream if cancelled
    reconnect = function reconnect() {
      connectButton.disabled = false;
      startup()
    };

  // Define "global" variables

  var connectButton = null;
  var disconnectButton = null;
  var messageInputBox = null;
  var receiveBox = null;

  var localConnection = null; // RTCPeerConnection for our "local" connection
  // adjust this to remote address; or use `ServiceWorker` `onfetch`; other
  var remoteConnection = null; // RTCPeerConnection for the "remote"

  var sendChannel = null; // RTCDataChannel for the local (sender)
  var receiveChannel = null; // RTCDataChannel for the remote (receiver)

  // Functions

  // Set things up, connect event listeners, etc.

  function startup() {
    connectButton = document.getElementById("connectButton");
    disconnectButton = document.getElementById("disconnectButton");
    messageInputBox = document.getElementById("message");
    receiveBox = document.getElementById("receivebox");

    // Set event listeners for user interface widgets

    connectButton.addEventListener("click", connectPeers, false);
    disconnectButton.addEventListener("click", disconnectPeers, false);
  }

  // Connect the two peers. Normally you look for and connect to a remote
  // machine here, but we"re just connecting two local objects, so we can
  // bypass that step.

  function connectPeers() {
    // Create the local connection and its event listeners
    if (len < totalBytes) {
      localConnection = new RTCPeerConnection();

      // Create the data channel and establish its event listeners
      sendChannel = localConnection.createDataChannel("sendChannel");
      sendChannel.onopen = handleSendChannelStatusChange;
      sendChannel.onclose = handleSendChannelStatusChange;

      // Create the remote connection and its event listeners

      remoteConnection = new RTCPeerConnection();
      remoteConnection.ondatachannel = receiveChannelCallback;

      // Set up the ICE candidates for the two peers

      localConnection.onicecandidate = e => 
        !e.candidate || remoteConnection.addIceCandidate(e.candidate)
      .catch(handleAddCandidateError);

      remoteConnection.onicecandidate = e => 
        !e.candidate || localConnection.addIceCandidate(e.candidate)
      .catch(handleAddCandidateError);

      // Now create an offer to connect; this starts the process

      localConnection.createOffer()
      .then(offer => localConnection.setLocalDescription(offer))
      .then(() => remoteConnection
                 .setRemoteDescription(localConnection.localDescription)
       )
      .then(() => remoteConnection.createAnswer())
      .then(answer => remoteConnection
                      .setLocalDescription(answer)
       )
      .then(() => localConnection
                 .setRemoteDescription(remoteConnection.localDescription)
      )
      // start streaming connection
      .then(sendMessage)
      .catch(handleCreateDescriptionError);
    } else {

      alert("total bytes streamed:" + len)
    }

  }

  // Handle errors attempting to create a description;
  // this can happen both when creating an offer and when
  // creating an answer. In this simple example, we handle
  // both the same way.

  function handleCreateDescriptionError(error) {
    console.log("Unable to create an offer: " + error.toString());
  }

  // Handle successful addition of the ICE candidate
  // on the "local" end of the connection.

  function handleLocalAddCandidateSuccess() {
    connectButton.disabled = true;
  }

  // Handle successful addition of the ICE candidate
  // on the "remote" end of the connection.

  function handleRemoteAddCandidateSuccess() {
    disconnectButton.disabled = false;
  }

  // Handle an error that occurs during addition of ICE candidate.

  function handleAddCandidateError() {
    console.log("Oh noes! addICECandidate failed!");
  }

  // Handles clicks on the "Send" button by transmitting
  // a message to the remote peer.

  function sendMessage() {

    stream = new ReadableStream({
      start(controller) {
          interval = setInterval(() => {
            if (sendChannel) {
              curr = randomData();
              len += curr.byteLength;
              // queue current stream
              controller.enqueue([curr, len, sendChannel.send(curr)]);

              if (len >= totalBytes) {
                controller.close();
                clearInterval(interval);
              }
            }
          }, 1000);
        },
        pull(controller) {
          // do stuff during stream
          // call `releaseLock()` if `diconnect` button clicked
          if (!sendChannel) reader.releaseLock();
        },
        cancel(reason) {
          clearInterval(interval);
          console.log(reason);
        }
    });

    reader = stream.getReader({
      mode: "byob"
    });

    reader.read().then(function process(result) {
        if (result.done && len >= totalBytes) {
          console.log("Stream done!");
          connectButton.disabled = false;
          if (len < totalBytes) reconnect();
          return;
        }

        if (!result.done && result.value) {
          var [currentStream, totalStreamLength] = [...result.value];
        }

        if (result.done && len < totalBytes) {
          throw new Error("stream cancelled")
        }

        console.log("currentStream:", currentStream
                   , "totalStremalength:", totalStreamLength
                   , "result:", result);
        return reader.read().then(process);
      })
      .catch(function(err) {
        console.log("catch stream cancellation:", err);
        if (len < totalBytes) reconnect()
      });

    reader.closed.then(function() {
      console.log("stream closed")
    })

  }

  // Handle status changes on the local end of the data
  // channel; this is the end doing the sending of data
  // in this example.

  function handleSendChannelStatusChange(event) {
    if (sendChannel) {
      var state = sendChannel.readyState;

      if (state === "open") {
        disconnectButton.disabled = false;
        connectButton.disabled = true;
      } else {
        connectButton.disabled = false;
        disconnectButton.disabled = true;
      }
    }
  }

  // Called when the connection opens and the data
  // channel is ready to be connected to the remote.

  function receiveChannelCallback(event) {
    receiveChannel = event.channel;
    receiveChannel.onmessage = handleReceiveMessage;
    receiveChannel.onopen = handleReceiveChannelStatusChange;
    receiveChannel.onclose = handleReceiveChannelStatusChange;
  }

  // Handle onmessage events for the receiving channel.
  // These are the data messages sent by the sending channel.

  function handleReceiveMessage(event) {
    var decoder = new TextDecoder();
    var data = decoder.decode(event.data);
    var el = document.createElement("p");
    var txtNode = document.createTextNode(data);

    el.appendChild(txtNode);
    receiveBox.appendChild(el);
  }

  // Handle status changes on the receiver"s channel.

  function handleReceiveChannelStatusChange(event) {
    if (receiveChannel) {
      console.log("Receive channel's status has changed to " +
        receiveChannel.readyState);
    }

    // Here you would do stuff that needs to be done
    // when the channel"s status changes.
  }

  // Close the connection, including data channels if they"re open.
  // Also update the UI to reflect the disconnected status.

  function disconnectPeers() {

    // Close the RTCDataChannels if they"re open.

    sendChannel.close();
    receiveChannel.close();

    // Close the RTCPeerConnections

    localConnection.close();
    remoteConnection.close();

    sendChannel = null;
    receiveChannel = null;
    localConnection = null;
    remoteConnection = null;

    // Update user interface elements


    disconnectButton.disabled = true;
    // cancel stream on `click` of `disconnect` button, 
    // pass `reason` for cancellation as parameter
    reader.cancel("stream cancelled");
  }

  // Set up an event listener which will run the startup
  // function once the page is done loading.

  window.addEventListener("load", startup, false);
})();

plnkr http://plnkr.co/edit/cln6uxgMZwE2EQCfNXFO?p=preview

Answer 4

You could use Promise , setTimeout, recursion. See also PUT vs POST in REST

var count = 0, total = 0, timer = null, d = 500, stop = false, p = void 0
, request = function request () {
              return new XMLHttpRequest()
            };
function sendData() {
  p = Promise.resolve(generateSomeBinaryData()).then(function(data) { 
    var currentRequest = request();
    currentRequest.open("POST", "http://example.com");
    currentRequest.onload = function () {
      ++count; // increment `count`
      total += data.byteLength; // increment total bytes posted to server
    }

    currentRequest.onloadend = function () {
      if (stop) { // stop recursion
        throw new Error("aborted") // `throw` error to `.catch()`
      } else {
        timer = setTimeout(sendData, d); // recursively call `sendData`
      }
    }
    currentRequest.send(data); // `data`: `Uint8Array`; `TypedArray`
    return currentRequest; // return `currentRequest` 
  });
  return p // return `Promise` : `p`
}

var curr = sendData();

curr.then(function(current) {
  console.log(current) // current post request
})
.catch(function(err) {
  console.log(e) // handle aborted `request`; errors
});

Answer 5

I think the short answer is no. As of writing this response (November 2021) this is not available in any of the major browsers.

The long answer is:

I think you are looking in the right place with the Fetch API. ReadableStream is currently a valid type for the body property of the Request constructor:
https://developer.mozilla.org/en-US/docs/Web/API/Request/Request#parameters

However, sadly if you look at the browser support matrix:
https://developer.mozilla.org/en-US/docs/Web/API/Request/Request#browser_compatibility
you can see that "Send ReadableStream in request body" is still No for all the major browsers. Though it is currently available in experimental mode in some browsers (including Chrome).

There is a nice tutorial on how to do it in experimental mode here:
https://web.dev/fetch-upload-streaming/

Looking at the dates of the posts and the work done on this feature, I think it looks pretty clear that this technology is stagnating and we probably won't see it anytime soon. Consequently, WebSockets are probably still sadly one of our few good options (for unbounded stream transfers):
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API

Answer 6

Server-Sent Events and WebSockets are the preferred methods but in your case you are wanting to create a Representational state transfer, REST, API and use Long Polling. See How do I implement basic “Long Polling”?

Long polling process is handled both on the client side and server side. The server script and http server must be configured to support long polling.

Other than long polling, short polling (XHR/AJAX) requires the browser to poll the server.