6

I'm wondering if there's some proper way (internet protocol, browser API, etc.), for a web app in a modern browser, to get a stream of large data from a Web Server (in my case, nodejs, but the question is focused on client side), and to save it to one big file in the client's file system. The assumption is that the amount of data is too large to be entirely buffered in the App's assigned memory, so I guess somehow the process should append chunks of data from the stream directly to a file.

The data is basically tens of thousands of db records, that should be written to a format such as csv.

Hope I'm making any sense here.

Luke Girvin
  • 13,221
  • 9
  • 64
  • 84
Israel
  • 1,184
  • 2
  • 13
  • 26
  • You mean like a _normal_ file download? Aka, generating the file on the server and then sending that file to the client. – George Mar 22 '17 at 14:31
  • 1
    @George no, the server should stream data from the db and the client should process and build the file – Israel Mar 22 '17 at 14:34
  • You will want to use `pipe()`. https://nodejs.org/dist/latest-v6.x/docs/api/stream.html#stream_readable_pipe_destination_options – Diego ZoracKy Mar 22 '17 at 15:04
  • 1
    @DiegoZoracKy thanks, but this is an approach for the server side, while I'm asking specifically about the client side – Israel Mar 22 '17 at 15:43

1 Answers1

4

You asked if there was a proper way. No, I'm afraid the technology that does it is badly supported and brand new. However, if you are brave enough to try anyway this is what you have.

1. Getting a stream of data from Web Server

If you don't have to support IE then you can use the Fetch API. See this question. If you do have to support IE then you could use a websocket to push down little bits at a time.

2. Writing a stream of data to the disk

StreamSaver.js Is a project that tries to accomplish this using the ReadableStream API, however it currently only works in Opera and Chrome.

Community
  • 1
  • 1
JosiahDaniels
  • 2,411
  • 20
  • 37