Is it possible to stream an octet stream being generated in javascript?

Question

Lets suppose a case where a huge string is generated from a small string using some javascript logic, and then the text file is forced to be downloaded on the browser.

This is possible using an octet-stream download by putting it as an href , as mentioned in this answer :

Create a file in memory for user to download, not through server.

function download(filename, text) {
  var pom = document.createElement('a');
  pom.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(text));
  pom.setAttribute('download', filename);
  pom.click();
}

But this solution requires 'text' to be fully generated before being pushed for the download, hence it will have to be held in browser memory fully .

Is it possible to stream the text as it gets generated using CLIENT SIDE LOGIC ONLY ?

For example :

var inputString = "A";
var outStr = "";
for(var i = 0; i < 10000000 ; i++)
 {
   /* concatenate inputString to output on the go */
 }

Answer 1

Yes & no. No because there's not a way to write to files with just client-side javascript. Kinda. You can prompt a user to download & save a file, but as you mentioned, the code must generate the whole file before that download happens. Note: By "stream" I assume you mean stream to file (constantly write to a file) & by "CLIENT SIDE LOGIC ONLY" I assume you mean in the browser.

Looks like Mozilla has been working on a way to let client-side code interact with files. Here comes the yes. Kind of. They have their own file system api that lets you interact with (write to) the local machines file system. Specifically, there's a function that lets you write an input stream to a file. However, there's a few asterisks:

1) looks like that whole system is being deprecated; they encourage developers to use OS.file over File I/O

2) You have to use XPConnect, a system that lets you access Mozilla's XPCOM (component library) in javascript. If you want to do this in the browser, it looks like only firefox extensions have the proper permissions to interact with those components (). If you didn't want to do this in the browser, you obviously could just use node.

Assuredly, more complications are bound to show up during implementation. But this looks like the most sure path forward, seeing as how OS.File gives you access to functions like OS.File.writeAtomic() & basic write to file

That being said, it's not that great of a path, but hopefully this gives you a solid starting point. As @dandavis mentioned, browsers (i.e. "client side logic") are designed to not allow this sort of thing. It would be an incredibly huge oversight / security flaw if a website could interact with any user's local file system.

Additional resources:
Wikipedia on XPConnect
Guide on working with XPCOM in javascript - may not be that useful

Answer 2

There is a way to do this, but it relies on a Chrome only Filesystem API. We will create and write to a temporary file in a sandboxed file system and the copy it to the regular file system once we are done. This way you do not have to store the entire file in memory. The asynchronous version of the Chrome API is not currently being considered for standardization by W3C, but the synchronous verison (which uses web workers) is. If browser support is a concern, then this answer is not for you.

The API works like this: First, we get the requestFileSystem() function from the browser. Currently it is prefixed by "webkit":

window.requestFileSystem  = window.requestFileSystem || window.webkitRequestFileSystem;

Next, we request a temporary file system (this way we do not need to ask for user permission):

var fileSystem; //This will store the fileSystem for later access
var fileSize = 1024*1024 //Our maximum file system size.
function errorHandler(e) {
  console.log('Error: ' + e.name);
}
window.requestFileSystem(window.TEMPORARY, fileSize, function (fs) { fileSystem = fs; }, errorHandler);

Now that we have access to the file system it is time to create a file:

var fileOptions = {
    create: true, //If the file is not found, create it
    exclusive: false //Don't throw an error if the file doesn't exist
};

Here we call the getFile() function, which can create a file if it doesn't exist. Inside of the callback, we can create a new fileWriter for writing to the file. The fileWriter is then moved to the end of the file, and we create a new text blob to append to it.

fileSystem.root.getFile(fileName, fileOptions, function(fileEntry) {
    fileEntry.createWriter(function(fileWriter) {
      fileWriter.seek(fileWriter.length); 
      var blob = new Blob([STRING_TO_WRITE], {type: 'text/plain'});
      fileWriter.write(blob);
    }, errorHandler);
});

Note that this API does not save to the normal, user filesystem. Instead, it saves to a special sandboxed folder. If you want to save it to the user's file system, you can create a filesystem: link. When the user clicks on it, it will prompt them to save it. After they save it, you can then remove the temporary file.

This function generates the filesystem link using the fileEntry's toURL() function:

var save = function () {
  var download = document.querySelector("a[download]");
  if (!fileSystem) { return; }
   fileSystem.root.getFile(fileName, {create: false, exclusive: true}, function(fileEntry) {
    download.href = fileEntry.toURL();
  }, errorHandler);
}

Using a link with the download attribute will force the download of the file.

<a download></a>

Here is a plunker that demonstrates this: http://plnkr.co/edit/q6ihXWEXSOtutbEy1b5G?p=preview

Hopefully this accomplishes what you want. You can continuously append to the file, it won't be kept in memory, but it will be in the sandboxed filesystem until the user saves it to the regular filesystem.

For more information take a look at this HTML5rocks article or this one if you want to use the newer, synchronous Web Worker API.

Answer 3

I would have suggest it the way @quantumwannabe describes it, using temporary sandbox file to append chunks.

But there is a new way that can be used today (behind a flag) but will be enabled in the next version of chrome (52)

And here is where i will make @KeenanLidral-Porter answer incorrect. And @quantumwannabe answer a unnecessary step
Because there is now a way to write a stream to the filesystem directly: StreamSaver.js

It acts as if there was a server sending octet-stream header and tells the browser to download chunks of data with help of a service worker

const writeStream = streamSaver.createWriteStream('filename.txt')
const encoder = new TextEncoder
let data = 'a'.repeat(1024) // Writing some stuff triggers the save dialog to show
let uint8array = encoder.encode(data + "\n\n")

writeStream.write(uint8array) // Write some data when you got some
writeStream.close() // End the saving